Docker image: add a Docker Compose example file

[caliuf]

Now that we addressed the issue of first ensemble start adding an optional metaformat before running the bk server, we should add a docker compose example for helping new users.

[caliuf]

This issue it's a good opportunity to decide whether the example in Makefile should be kept or it's better to drop it.
And if we choose the letter option, should we drop Makefile completely?

[caliuf]

Should this docker compose example start the dice tutorial application as currently does the Makefile?

[aojea]

@ivankelly I was trying to follow your tutorial using the docker compose file in #701 but I'm facing several issues.

CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                    PORTS
                     NAMES
d27578fa2b35        caiok/bookkeeper-tutorial   "/bin/sh -c '/bin/..."   7 minutes ago       Up 7 minutes
                     docker_dice_4
0be4e70a0add        caiok/bookkeeper-tutorial   "/bin/sh -c '/bin/..."   37 minutes ago      Up 2 minutes
                     docker_dice_2
001e5c0f9282        caiok/bookkeeper-tutorial   "/bin/sh -c '/bin/..."   37 minutes ago      Up 2 minutes
                     docker_dice_3
277e59f29faa        caiok/bookkeeper-tutorial   "/bin/sh -c '/bin/..."   37 minutes ago      Up 3 minutes
                     docker_dice_1
43d787a08991        bookkeeper                  "/bin/bash /opt/bo..."   37 minutes ago      Up 37 minutes (healthy)   3181/tcp
                     docker_bookkeeper_2
ff2100d20ed5        bookkeeper                  "/bin/bash /opt/bo..."   37 minutes ago      Up 37 minutes (healthy)   3181/tcp
                     docker_bookkeeper_1
789ba349a4cc        zookeeper:3.4.9             "/docker-entrypoin..."   37 minutes ago      Up 37 minutes             2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   docker_zookeeper_1

If I start the dices containers at the same time that the application, they fail with the following error:

+ java -jar /local/bookkeeper-tutorial.jar --zookeeper-servers zookeeper:2181
Zookeeper Server: zookeeper:2181
log4j:WARN No appenders could be found for logger (org.apache.curator.CuratorZookeeperClient).
log4j:WARN Please initialize the log4j system properly.
Connecting to Zookeeper Server... Connected.
Exception in thread "main" org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /ledgers/available/readonly
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
        at org.apache.bookkeeper.client.BookieWatcher$ReadOnlyBookieWatcher.<init>(BookieWatcher.java:264)
        at org.apache.bookkeeper.client.BookieWatcher.<init>(BookieWatcher.java:85)
        at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:219)
        at org.apache.bookkeeper.Dice.<init>(Dice.java:54)
        at org.apache.bookkeeper.Dice.main(Dice.java:269)

However, if I run the docker-compose up and, once everything is working, I run new dice containers I'm able to see how the clients change from follower to leader, but I can't see in the logs the value of the dice.

Docker dice_4 is not the leader

$ docker logs docker_dice_4
+ java -jar /local/bookkeeper-tutorial.jar --zookeeper-servers zookeeper:2181
Zookeeper Server: zookeeper:2181
log4j:WARN No appenders could be found for logger (org.apache.curator.CuratorZookeeperClient).
log4j:WARN Please initialize the log4j system properly.
Connecting to Zookeeper Server... Connected.
Follow

Restart dice 1 2 and 3

$docker restart docker_dice_1; docker restart docker_dice_2; docker
restart docker_dice_3
docker_dice_1
docker_dice_2
docker_dice_3

Now, dice4 is the leader but can't see any value

 $  docker logs docker_dice_4
+ java -jar /local/bookkeeper-tutorial.jar --zookeeper-servers zookeeper:2181
Zookeeper Server: zookeeper:2181
log4j:WARN No appenders could be found for logger (org.apache.curator.CuratorZookeeperClient).
log4j:WARN Please initialize the log4j system properly.
Connecting to Zookeeper Server... Connected.
Follow
Becoming leader

[sijie]

@itsuugo it seems to be NoNodeException is caused because metaformat is not being executed.

[aojea]

I guess that I found the problem:

bookkeeper_1  | BK_CLUSTER_ROOT_PATH is
bookkeeper_1  | BK_bookiePort bookie service port is 3181
bookkeeper_1  | BK_zkServers is zookeeper:2181
bookkeeper_1  | BK_DATA_DIR is /data/bookkeeper
bookkeeper_1  | BK_CLUSTER_ROOT_PATH is

BK_CLUSTER_ROOT_PATH is empty and it fails to create it as you can see in the log of the container

bookkeeper_1  | 2017-11-22 21:24:36,524 - INFO  - [main:ClientCnxnSocket@236] - jute.maxbuffer value is 4194304 Bytes
bookkeeper_1  | create [-s] [-e] [-c] [-t ttl] path [data] [acl]

However, the README says it can be empty 🤔

[jiazhai]

@aojea, How could I reproduce it?

Right, it could be empty for BK_CLUSTER_ROOT_PATH. In most of the case, like in the docker/Makefile, it is empty.

The command zk create " " failure, should not break the process.

echo "create the zk root dir for bookkeeper"
/opt/bookkeeper/bin/bookkeeper org.apache.zookeeper.ZooKeeperMain -server ${BK_zkServers} create ${BK_CLUSTER_ROOT_PATH}

Maybe we could add " || true" after this command, if it was broken in your case.

[aojea]

@jiazhai

Download the docker-compose.yml file from #701 to the docker folder.

I assume that you have docker and docker-compose installed in your machine.

Run the ensemble with multiple clients and bookeepers:

docker-compose up -d --scale bookkeeper=2 --scale dice=3

You can see the output in each container with docker logs:

docker logs docker_dice_2
+ java -jar /local/bookkeeper-tutorial.jar --zookeeper-servers zookeeper:2181
Zookeeper Server: zookeeper:2181
log4j:WARN No appenders could be found for logger (org.apache.curator.CuratorZookeeperClient).
log4j:WARN Please initialize the log4j system properly.
Connecting to Zookeeper Server... Connected.
Follow
Becoming leader

[jiazhai]

@aojea, From your command, I see what is the problem now. It is not related to BK_CLUSTER_ROOT_PATH.

1, have at least 3 bookies is better, but in your command it said "bookkeeper=2",
2, it seems that dice not wait for bookkeeper to be ready, the depends_on need more work, please reference this link for more details

[aojea]

I tried again (adding a while loop to the dice to wait for bookkeeper port to be available)[https://github.com/aojea/bookkeeper-tutorial/commit/5b276f21abe119118fc2bdd26095bacef093c4c8] with the same result.

Also, I ran it only with one bookie and one dice docker-compose up, and the exception is raised when the bookie registers itself in zookeeper as you can see in the log below:

dice_1        | Connected to bookie: bookkeeper
dice_1        | + java -jar /local/bookkeeper-tutorial.jar --zookeeper-servers zookeeper:2181
bookkeeper_1  | 2017-12-04 10:17:36,716 - INFO  - [bookie-io-1:AuthHandler$ServerSideHandler$AuthHandshakeCompleteCallback@211] - Authentication success on server side
dice_1        | Zookeeper Server: zookeeper:2181
dice_1        | log4j:WARN No appenders could be found for logger (org.apache.curator.CuratorZookeeperClient).
dice_1        | log4j:WARN Please initialize the log4j system properly.
zookeeper_1   | 2017-12-04 10:17:36,999 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket connection from /172.19.0.4:39940
zookeeper_1   | 2017-12-04 10:17:37,005 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@938] - Client attempting to establish new session at /172.19.0.4:39940
zookeeper_1   | 2017-12-04 10:17:37,008 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@683] - Established session 0x100671263e30003 with negotiated timeout 4000 for client /172.19.0.4:39940
dice_1        | Connecting to Zookeeper Server... Connected.
zookeeper_1   | 2017-12-04 10:17:37,067 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@215] - Accepted socket connection from /172.19.0.4:39946
zookeeper_1   | 2017-12-04 10:17:37,067 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@938] - Client attempting to establish new session at /172.19.0.4:39946
zookeeper_1   | 2017-12-04 10:17:37,069 [myid:] - INFO  [SyncThread:0:ZooKeeperServer@683] - Established session 0x100671263e30004 with negotiated timeout 30000 for client /172.19.0.4:39946
dice_1        | Becoming leader
dice_1        | Lead
bookkeeper_1  | 2017-12-04 10:17:38,351 - INFO  - [bookie-io-2:AuthHandler$ServerSideHandler$AuthHandshakeCompleteCallback@211] - Authentication success on server side
dice_1        | Lead
bookkeeper_1  | 2017-12-04 10:17:39,391 - INFO  - [bookie-io-3:AuthHandler$ServerSideHandler$AuthHandshakeCompleteCallback@211] - Authentication success on server side
dice_1        | Lead
bookkeeper_1  | 2017-12-04 10:17:40,447 - INFO  - [bookie-io-4:AuthHandler$ServerSideHandler$AuthHandshakeCompleteCallback@211] - Authentication success on server side
dice_1        | Lead
bookkeeper_1  | 2017-12-04 10:17:41,533 - INFO  - [bookie-io-5:AuthHandler$ServerSideHandler$AuthHandshakeCompleteCallback@211] - Authentication success on server side
dice_1        | Lead
bookkeeper_1  | 2017-12-04 10:17:42,620 - INFO  - [bookie-io-6:AuthHandler$ServerSideHandler$AuthHandshakeCompleteCallback@211] - Authentication success on server side
dice_1        | Lead
bookkeeper_1  | 2017-12-04 10:17:43,720 - INFO  - [bookie-io-7:AuthHandler$ServerSideHandler$AuthHandshakeCompleteCallback@211] - Authentication success on server side
dice_1        | Lead
bookkeeper_1  | 2017-12-04 10:17:44,761 - INFO  - [bookie-io-0:AuthHandler$ServerSideHandler$AuthHandshakeCompleteCallback@211] - Authentication success on server side
dice_1        | Lead
bookkeeper_1  | 2017-12-04 10:17:45,846 - INFO  - [bookie-io-1:AuthHandler$ServerSideHandler$AuthHandshakeCompleteCallback@211] - Authentication success on server side
dice_1        | Lead
zookeeper_1   | 2017-12-04 10:17:45,999 [myid:] - INFO  [SessionTracker:ZooKeeperServer@354] - Expiring session 0x10066f0df0d000d, timeout of 10000ms exceeded
zookeeper_1   | 2017-12-04 10:17:46,001 [myid:] - INFO  [SessionTracker:ZooKeeperServer@354] - Expiring session 0x10066f0df0d000e, timeout of 10000ms exceeded
zookeeper_1   | 2017-12-04 10:17:46,002 [myid:] - INFO  [SessionTracker:ZooKeeperServer@354] - Expiring session 0x10066f0df0d0004, timeout of 10000ms exceeded
zookeeper_1   | 2017-12-04 10:17:46,003 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x10066f0df0d000d
zookeeper_1   | 2017-12-04 10:17:46,004 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x10066f0df0d000e
zookeeper_1   | 2017-12-04 10:17:46,005 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x10066f0df0d0004
bookkeeper_1  | 2017-12-04 10:17:46,036 - INFO  - [BookieStateService-0:Bookie@1035] - Registered myself in ZooKeeper at /ledgers/available/172.19.0.3:3181.
bookkeeper_1  | 2017-12-04 10:17:46,041 - INFO  - [main:BookieServer@427] - Register shutdown hook successfully
bookkeeper_1  | 2017-12-04 10:17:46,884 - INFO  - [bookie-io-2:AuthHandler$ServerSideHandler$AuthHandshakeCompleteCallback@211] - Authentication success on server side
dice_1        | Lead
bookkeeper_1  | 2017-12-04 10:17:46,901 - INFO  - [bookie-io-2:BookieRequestHandler@51] - Channel connected  [id: 0xa803fa82, L:/172.19.0.3:3181 - R:/172.19.0.4:40762]
zookeeper_1   | 2017-12-04 10:17:46,936 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x100671263e30003
zookeeper_1   | 2017-12-04 10:17:46,938 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1040] - Closed socket connection for client /172.19.0.4:39940 which had sessionid 0x100671263e30003
dice_1        | Exception in thread "main" org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException
dice_1        |         at org.apache.bookkeeper.client.BKException.create(BKException.java:58)
dice_1        |         at org.apache.bookkeeper.client.BookKeeper.createLedger(BookKeeper.java:521)
dice_1        |         at org.apache.bookkeeper.Dice.lead(Dice.java:128)
dice_1        |         at org.apache.bookkeeper.Dice.playDice(Dice.java:227)
dice_1        |         at org.apache.bookkeeper.Dice.main(Dice.java:270)
zookeeper_1   | 2017-12-04 10:18:05,999 [myid:] - INFO  [SessionTracker:ZooKeeperServer@354] - Expiring session 0x10066f0df0d0008, timeout of 30000ms exceeded
zookeeper_1   | 2017-12-04 10:18:06,000 [myid:] - INFO  [SessionTracker:ZooKeeperServer@354] - Expiring session 0x100671263e30000, timeout of 30000ms exceeded
zookeeper_1   | 2017-12-04 10:18:06,001 [myid:] - INFO  [SessionTracker:ZooKeeperServer@354] - Expiring session 0x10066f0df0d000b, timeout of 30000ms exceeded
zookeeper_1   | 2017-12-04 10:18:06,001 [myid:] - INFO  [SessionTracker:ZooKeeperServer@354] - Expiring session 0x10066f0df0d000c, timeout of 30000ms exceeded
zookeeper_1   | 2017-12-04 10:18:06,002 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x10066f0df0d0008
zookeeper_1   | 2017-12-04 10:18:06,003 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x100671263e30000
zookeeper_1   | 2017-12-04 10:18:06,004 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x10066f0df0d000b
zookeeper_1   | 2017-12-04 10:18:06,004 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@487] - Processed session termination for sessionid: 0x10066f0df0d000c