zookeeper config

[alistaire47]

So despite having used sergeant for a while now, I never actually set up ZooKeeper properly till I just discovered this book. Following the instructions for macOS didn't quite get me there, though, so I thought I'd open an issue.

The issue: After calling drillbit.sh start, everything looked fine, but when I ran drill-conf, I would get an error (among other derivative SqlLine errors) that

No active Drillbit endpoint found from ZooKeeper. Check connection parameters?

In trying to fix this, I did/discovered:

1. homebrew has a services command that I'm pretty sure does the same thing as the ln line. When you install ZooKeeper, it tells you under Caveats, or you can see the same message with brew info:

$ brew info zookeeper
zookeeper: stable 3.4.10 (bottled), HEAD
Centralized server for distributed coordination of services
https://zookeeper.apache.org/
/usr/local/Cellar/zookeeper/3.4.10 (242 files, 31.4MB) *
  Poured from bottle on 2018-04-20 at 14:36:51
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/zookeeper.rb
==> Dependencies
Optional: python@2 ✘
==> Options
--with-perl
	Build Perl bindings
--with-python@2
	Build with python@2 support
--HEAD
	Install HEAD version
==> Caveats
To have launchd start zookeeper now and restart at login:
  brew services start zookeeper
Or, if you don't want/need a background service you can just run:
  zkServer start

I'm 99% sure that was unrelated to my issues, but it is a nice alternative way to manage services.

2. After looking at the ZooKeeper startup docs, it looked like I was supposed to have a conf/zoo.cfg file for ZooKeeper. conf didn't exist in its libexec directory, so I created it and copied the example exactly (maybe I should've changed the dataDir, but the default is ok for me). This made a bunch of the SqlLine errors disappear, but still didn't fix the connection issue, so I'm not 💯 this is necessary.
3. Looking at drill/conf/drill-override-example.conf, I saw a zk section, which seemed relevant to connecting the two. I copied it:

  zk: {
         connect: "localhost:2181",
         root: "drill",
         refresh: 500,
         timeout: 5000,
         retry: {
           count: 7200,
           delay: 500
         }
  }

into my drill-override.conf, and then everything started working beautifully.

I'm not sure if anybody else's installation process will mirror mine exactly, but in case they do, I thought the information should be on the internet somewhere.

Thanks again for putting all of this together! Drill is, indeed, awesome.

[hrbrmstr]

this. is. great! #ty!

I'll get this shunted into the book soon since I also want to add recipe for ODBC this weekend.

[alistaire47]

After I posted this, I noticed that homebrew Drill has been lagging the current version (1.10.0 vs. 1.13.0). I tried to use its PR infrastructure, but its tests are failing for reasons I don't understand. Hopefully a maintainer can point it in the right direction.

tl;dr: My above approach may or may not be necessary for the current version. I'll update if anything changes.

[alistaire47]

Woo, a maintainer rescued my PR!

After I updated, I relinked to usr/local/drill, updated drill-override.conf again, and did a full system reboot (drill_version(drill_connection()) was inexplicably still returning "1.10.0" even after the directory was gone and restarted R, though everything was functioning fine and the Drill shell was on 1.13.0). ZooKeeper didn't require any adjustments.

ZooKeeper is a minor version behind on Homebrew as well, but the current version apparently has a severe bug, so it doesn't seem worth a PR till the next version.

[LDalby]

I'm aware that it is a long time since this issue was raised, but I'm running into similar issues just now, so I thought it was better to post here rather than create a new issue.

I'm on Ubuntu 20.04 and have followed the steps outlined in the book. All good, but running drill-conf gives me this:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by javassist.util.proxy.SecurityActions (file:/home/au206907/bin/apache-drill-1.17.0/jars/3rdparty/javassist-3.26.0-GA.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of javassist.util.proxy.SecurityActions
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Error: Failure in creating DrillConnectionImpl: java.lang.IllegalStateException: No active Drillbit endpoint found from ZooKeeper. Check connection parameters? (state=,code=0)
java.sql.SQLException: Failure in creating DrillConnectionImpl: java.lang.IllegalStateException: No active Drillbit endpoint found from ZooKeeper. Check connection parameters?
	at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:182)
	at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:67)
	at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:67)
	at org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
	at org.apache.drill.jdbc.Driver.connect(Driver.java:75)
	at sqlline.DatabaseConnection.connect(DatabaseConnection.java:135)
	at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:192)
	at sqlline.Commands.connect(Commands.java:1364)
	at sqlline.Commands.connect(Commands.java:1244)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
	at sqlline.SqlLine.dispatch(SqlLine.java:730)
	at sqlline.SqlLine.initArgs(SqlLine.java:410)
	at sqlline.SqlLine.begin(SqlLine.java:515)
	at sqlline.SqlLine.start(SqlLine.java:267)
	at sqlline.SqlLine.main(SqlLine.java:206)
Caused by: java.lang.IllegalStateException: No active Drillbit endpoint found from ZooKeeper. Check connection parameters?
	at org.apache.drill.shaded.guava.com.google.common.base.Preconditions.checkState(Preconditions.java:459)
	at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:348)
	at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:169)
	... 18 more
Apache Drill 1.17.0
"JSON ain't no thang."

A bit of a mouthful, but I guess most of it relates to the problem that Zookeeper can't find an active Drillbit endpoint.
I've tried to do as alistaire47 suggested and created a zoo.cfg file in conf, but that didn't help and from what I understand from reading through this issue, it ended up not being necessary to do any adjustments after simply installing Zookeeper.

So here I'm stuck. Any help would be highly appreciated.

[juanfung]

I'm on Mac OS Catalina, not using Homebrew. Having the same issue with drill-conf as @LDalby. In fact, drillbit.sh start appears to start but is not working.

jff$ $DRILL_HOME/bin/drillbit.sh start
/Users/jff/bin/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
/Users/jff/bin/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
/Users/jff/bin/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
/Users/jff/bin/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
Starting drillbit, logging to /usr/local/drill/log/drillbit.out
jff$ $DRILL_HOME/bin/drillbit.sh status
/Users/jff/bin/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
/Users/jff/bin/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
/Users/jff/bin/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
/Users/jff/bin/apache-drill-1.17.0/bin/drill-config.sh: line 144: let: lineCount=: syntax error: operand expected (error token is "=")
/usr/local/drill/drillbit.pid file is present but drillbit is not running.

I haven't been able to find much help online. There are a few unresolved issues on Apache, including this one more recently. I can, however, use drill-embedded

I apologize if this isn't the proper forum for this. The book and the package are awesome.