Home
Up and Running with Secure Zookeeper
Introduction
The following assumes a blank Linux installation of a yum-based distribution such as Redhat or CentOS; specifically, I wrote this using the Amazon Linux AMI instance type.
Install Packages
[ec2-user@domU-12-31-39-0E-7D-D1 ~]$ sudo yum -y install java-1.6.0-openjdk emacs git ant krb5-serverNote that we would use the Sun JDK (as opposed to OpenJDK as above) in production.
Also feel free to substitute your favorite editor for emacs in the above.
If you are a developer, add to the install line:
ant-junit.. so that you can run ant test. In fact, you will not be able to run the complete ant test unless you install a lot of other C++-related packages; see Running Zookeeper's 'ant test' on AWS EC2 Linux.
Clone Zookeeper
Starting with release 3.4, Zookeeper has built-in the security features described here.
[ec2-user@domU-12-31-39-0E-7D-D1 ~]$ git clone git://github.com/apache/zookeeper.gitBuild
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ export JAVA_HOME=/usr/lib/jvm/java-openjdk
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ export PATH=$JAVA_HOME/bin:$PATH
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ ant clean compileSetup Kerberos KDC
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ sudo kdb5_util create -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'EXAMPLE.COM',
master key name 'K/M@EXAMPLE.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify: Start Kerberos KDC Daemon
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ sudo /etc/init.d/krb5kdc startStarting Kerberos 5 KDC: [ OK ]
Add Zookeeper Server principal
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ sudo kadmin.local
Authenticating as principal root/admin@EXAMPLE.COM with password.
kadmin.local: addprinc -randkey zookeeper/localhost
WARNING: no policy specified for zookeeper/localhost@EXAMPLE.COM; defaulting to no policy
Principal "zookeeper/localhost@EXAMPLE.COM" created.
kadmin.local: ktadd -k /home/ec2-user/zookeeper/conf/zookeeper.keytab zookeeper/localhostAdd a Zookeeper client principal
kadmin.local: addprinc -randkey myzkclient
WARNING: no policy specified for myzkclient@EXAMPLE.COM; defaulting to no policy
Principal "myzkclient@EXAMPLE.COM" created.
kadmin.local: ktadd -k /home/ec2-user/zookeeper/conf/myzkclient.keytab myzkclientSet keytab files ownership
..so that Zookeeper client and principal (running as ec2-user) can read them.
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ sudo chown ec2-user /home/ec2-user/zookeeper/conf/*.keytabConfigure Clients to use localhost as KDC
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ cat /etc/krb5.conf [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log[libdefaults] default_realm = EXAMPLE.COM dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true[realms] EXAMPLE.COM = { kdc = localhost admin_server = localhost }[domain_realm] .example.com = EXAMPLE.COM example.com = EXAMPLE.COM
Test Kerberos Authentication with KDC : Zookeeper Server
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ kinit -k -t conf/zookeeper.keytab zookeeper/localhost [ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ klist Ticket cache: FILE:/tmp/krb5cc_222 Default principal: zookeeper/localhost@EXAMPLE.COMValid starting Expires Service principal 04/15/11 18:37:20 04/16/11 18:37:20 krbtgt/EXAMPLE.COM@EXAMPLE.COM renew until 04/15/11 18:37:20
Test Kerberos Authentication with KDC : Zookeeper Client
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ kinit -k -t conf/myzkclient.keytab myzkclient [ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ klist Ticket cache: FILE:/tmp/krb5cc_222 Default principal: myzkclient@EXAMPLE.COMValid starting Expires Service principal 04/15/11 18:37:20 04/16/11 18:37:31 krbtgt/EXAMPLE.COM@EXAMPLE.COM renew until 04/15/11 18:37:31
Configure Zookeeper Server
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ cat conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/ec2-user/zk-data
clientPort=2181
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000The jaasLoginRenew line in the above will cause the Zookeeper to renew its Kerberos ticket once an hour (3,600,000 milliseconds = 3600 seconds = 60 minutes).
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ cat conf/java.env
export JVMFLAGS="-Djava.security.auth.login.config=/home/ec2-user/zookeeper/conf/jaas.conf"Note that, as we’ve configured it here, a single JAAS configuration file, conf/jaas.conf is used for SASL authentication by both the Client and Server. Below we use export JVMFLAGS to specify the location of this JAAS configuration file. The two configuration sections “Client” and “Server” need not be in the same file, however: if you want to separate them, simply use different values for the -Djava.security.auth.login.config in separate SERVER_JVMFLAGS and CLIENT_JVMFLAGS lines in java.env.
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ cat conf/jaas.conf
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/home/ec2-user/zookeeper/conf/zookeeper.keytab"
storeKey=true
useTicketCache=false
principal="zookeeper/localhost";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/home/ec2-user/zookeeper/conf/myzkclient.keytab"
principal="myzkclient"
useTicketCache=false
debug=true;
};_Note that I am using keytabs for clients in the above. You may also use: useTicketCache=true and useKeyTab=false. If so, you must do kinit with the appropriate arguments to initialize your ticket cache. You must also run kinit -R from your command line after this. This seems to be necessary due to a bug in the relevant Java libraries when parsing ticket cache. See this thread on the Kerberos development mailing list for more information.
Start Zookeeper Server
Test execution command-line with print-cmd (note the presence of -Djava.security.auth.login.config= : in your environment, make sure that its value points to the correct JAAS configuration file):
[[ec2-user@domU-12-31-39-14-39-61 zookeeper]$ bin/zkServer.sh print-cmd
JMX enabled by default
Using config: bin/../conf/zoo.cfg
java -Dzookeeper.log.dir="." -Dzookeeper.root.logger="INFO,CONSOLE" -cp "bin/../build/classes:bin/../build/lib/netty-
3.2.2.Final.jar:bin/../build/lib/log4j-1.2.15.jar:bin/../build/lib/jline-0.9.94.jar:bin/../zookeeper-*.jar:bin/../lib
/*.jar:bin/../src/java/lib/ivy-2.1.0.jar:bin/../conf:" -Djava.security.auth.login.config=/home/ec2-user/conf/jaas.con
f -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.Q
uorumPeerMain "bin/../conf/zoo.cfg"[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ bin/zkServer.sh start-foregroundJMX enabled by defaultUsing config: bin/../conf/zoo.cfg2011-01-27 20:25:54,413 [myid:] - INFO [main:QuorumPeerConfig@94] - Reading configuration from: bin/../conf/zoo.cfg 2011-01-27 20:25:54,417 [myid:] - WARN [main:QuorumPeerMain@106] - Either no config or no quorum defined in config, running in standalone mode
At the bottom of your console you should see:
2011-02-09 21:19:17,260 [myid:] - INFO [main:ServerCnxnFactory@204] - Using java.security.auth.login.config=/h
ome/ec2-user/zookeeper/conf/jaas.conf for doing server-side subject authentication.
2011-02-09 21:19:17,323 [myid:] - INFO [main:ServerCnxnFactory@212] - Server successfully authenticated.
2011-02-09 21:19:17,347 [myid:] - INFO [main:FileSnap@82] - Reading snapshot /home/ec2-user/zk-data/version-2/
snapshot.0
2011-02-09 21:19:17,351 [myid:] - INFO [main:FileTxnSnapLog@208] - Snapshotting: 0(in shell #2) Start Zookeeper Client
(Unlike the server’s command-line starter script bin/zkServer.sh, the client’s command-line starter script lacks a printcmd option).
[ec2-user@domU-12-31-39-0E-7D-D1 zookeeper]$ bin/zkCli.sh -server localhost:2181You should see a line similar to the following in the client output:
2012-02-12 09:10:56,458 [myid:] - INFO [main-SendThread(localhost.localdomain:2181):ZooKeeperSaslClient@98] - Found Login Context section 'Client': will use it to attempt to SASL-authenticate.(in shell #1) Watch client activity in Zookeeper Server log:
2011-04-15 18:58:54,422 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@207] - Accepted socket connection from /127.0.0.1:36815
2011-04-15 18:58:54,450 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@801] - Client attempting to establish new session at /127.0.0.1:36815
2011-04-15 18:58:54,459 [myid:] - INFO [SyncThread:0:FileTxnLog@197] - Creating new log file: log.1
2011-04-15 18:58:54,479 [myid:] - INFO [SyncThread:0:ZooKeeperServer@577] - Established session 0x12f5a7b20d80000 with negotiated timeout 30000 for client /127.0.0.1:36815
2011-04-15 18:58:54,628 [myid:] - INFO [SyncThread:0:ServerCnxnFactory$SaslServerCallbackHandler@349] - Successfully authenticated client: authenticationID=myzkclient@EXAMPLE.COM; authorizationID=myzkclient@EXAMPLE.COM.Note the Successfully authenticated client line in the above log output.
(in shell #2) Create a node that only a certain SASL-authenticated client can read
[zk: (CONNECTED) 5] create /test content sasl:myzkclient@EXAMPLE.COM:cdwra(in shell #2) Test that ACLs were set correctly.
[zk: (CONNECTED) 6] getAcl /test
'sasl,'myzkclient@EXAMPLE.COM
: cdrwaTest authorization with authorized client.
Please see the following JUnit tests, which verify that SASL-authenticated clients are granted the correct access by the Zookeeper server to the nodes to which they’ve been authorized:
Test authorization failure with unauthorized client.
Please see the following JUnit tests, which verify that clients which are denied access to resources to which they have not been been authorized: