After starting the docker-quickstart, I can't see any Kafka topics

[dvasdekis]

After completing the quickstart, I run SHOW TOPICS; within KSQL and get the following output:

Kafka Topic        | Registered | Partitions | Partition Replicas
-------------------------------------------------------------------
 _confluent-metrics | false      | 12         | 1
 _schemas           | false      | 1          | 1
 ksql__commands     | true       | 1          | 1

The two demonstration topics - pageviews, and users don't appear. Running docker container ls from bash, I get the following output:

CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS              PORT                                  S                                                    NAMES
d2853fffdfac        confluentinc/ksql-cli:latest              "perl -e 'while(1)..."   13 minutes ago      Up 13 minutes                                                                                                  quickstart_ksql-cli_1
6f54121daf3e        confluentinc/cp-schema-registry:latest    "/etc/confluent/do..."   13 minutes ago      Up 13 minutes       0.0.                                  0.0:8081->8081/tcp                                   quickstart_schema-registry_1
7dedcd9fba36        confluentinc/cp-enterprise-kafka:latest   "/etc/confluent/do..."   13 minutes ago      Up 13 minutes       0.0.                                  0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp         quickstart_kafka_1
4337617d71f6        confluentinc/cp-zookeeper:latest          "/etc/confluent/do..."   13 minutes ago      Up 13 minutes       2181                                  /tcp, 2888/tcp, 3888/tcp, 0.0.0.0:32181->32181/tcp   quickstart_zookeeper_1

The remedy turns out to be running docker-compose up -d again from the /ksql/docs/quickstart directory:

user@server:~/ksql/docs/quickstart$ docker-compose up -d
quickstart_zookeeper_1 is up-to-date
quickstart_kafka_1 is up-to-date
quickstart_schema-registry_1 is up-to-date
Starting quickstart_ksql-datagen-users_1
Starting quickstart_ksql-datagen-pageviews_1
quickstart_ksql-cli_1 is up-to-date

I then get:

CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS              PORTS                                                                                    NAMES
d2853fffdfac        confluentinc/ksql-cli:latest              "perl -e 'while(1)..."   15 minutes ago      Up 15 minutes                                                                                                quickstart_ksql-cli_1
27a412186e20        confluentinc/ksql-examples:latest         "bash -c 'echo Wai..."   15 minutes ago      Up 18 seconds                                                                                                quickstart_ksql-datagen-pageviews_1
82ffabc651e3        confluentinc/ksql-examples:latest         "bash -c 'echo Wai..."   15 minutes ago      Up 17 seconds                                                                                                quickstart_ksql-datagen-users_1
6f54121daf3e        confluentinc/cp-schema-registry:latest    "/etc/confluent/do..."   15 minutes ago      Up 15 minutes       0.0.0.                                0:8081->8081/tcp                                   quickstart_schema-registry_1
7dedcd9fba36        confluentinc/cp-enterprise-kafka:latest   "/etc/confluent/do..."   15 minutes ago      Up 15 minutes       0.0.0.                                0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp         quickstart_kafka_1
4337617d71f6        confluentinc/cp-zookeeper:latest          "/etc/confluent/do..."   15 minutes ago      Up 15 minutes       2181/t                                cp, 2888/tcp, 3888/tcp, 0.0.0.0:32181->32181/tcp   quickstart_zookeeper_1

And can see the topics from ksql:

ksql> show topics;

 Kafka Topic        | Registered | Partitions | Partition Replicas
-------------------------------------------------------------------
 _confluent-metrics | false      | 12         | 1
 _schemas           | false      | 1          | 1
 ksql__commands     | true       | 1          | 1
 pageviews          | false      | 1          | 1
 users              | false      | 1          | 1

I'm running this on Ubuntu Server 16.04.

Looks like the scripts for the pageviews and users topics need to retry creation a few times?

[aayars]

Hi @dvasdekis,

Thanks for the report. I just stepped through the Docker quickstart on a fresh Ubuntu 16 image, using current docker and docker-compose. The pageviews and users topics show up within a few seconds of starting the compose cluster, so it's not clear what's happening here yet. Can you share the output of docker --version and docker-compose --version?

--Alex

[dvasdekis]

Thanks Alex. I suspect the reason is because my VM isn't as powerful (an Azure A1 machine) as your test rig.
In the code we have a sleep of 2 seconds - I tried increasing it to 10, but it still didn't work. It might be a memory issue - it appears to run first time when I increase the VM size.

docker --version output: Docker version 17.06.1-ce, build 874a737
docker-compose --version output: docker-compose version 1.8.0, build unknown
To install docker, I followed this tutorial yesterday, so it should be the latest stable versions.

Edit: Confirmed - no issues with a large Azure VM (D4Sv3 - 4 CPUs/16GB RAM)

[hjafarpour]

@dvasdekis Thanks for your feedback. If this issue resolved would you please close it.

[kaiwaehner]

I have the same issue on my Mac with up-to-date Mac OS and Docker installations. I have given Docker 8GB Memory and 2 CPUs... It never creates the topics. Even if I do docker-compose up several times...

I tried several times with laptop restart, new git clone, etc.

On the other side, the Clickstream Demo works fine for me.

[sigmazen]

Hi
Having the same issue as originally reported by @dvasdekis but rerunning docker-compose up -d multiple times isn't working.
Running the same setup as @kaiwaehner.
Just re-cloned ksql repo to see if it had been fixed ... no luck.
@miguno anything I can try next?
Cheers

[jeremyz23]

I am also having the same issue. I have run docker-compose up -d multiple times. Each time it prints:

quickstart_zookeeper_1 is up-to-date
quickstart_kafka_1 is up-to-date
quickstart_schema-registry_1 is up-to-date
Starting quickstart_ksql-datagen-users_1
Starting quickstart_ksql-datagen-pageviews_1
quickstart_ksql-cli_1 is up-to-date

But the topics users and pageviews are not being created.

[aayars]

Hi @jeremyz23 @sigmazen @kaiwaehner @dvasdekis

What I would expect to see on docker-compose ps after up is a list of all containers in the compose file -- even if the containers couldn't start up, they should still be listed there with an exit status (and something in the log...)

Not sure if it can account for the "missing" containers, but I did just spot an issue with a bad version slipping into the compose file, affecting those datagen containers. This has been corrected, please do pull the latest bits (0.1.x branch) and try again.

If you're running into missing topics for the Docker quickstart using the latest code, please chime in here with the output of docker-compose ps.

Thanks
--Alex

[jeremyz23]

The output of docker-compose ps is below:

Does anyone know a solution for this?

[aayars]

Thanks, @jeremyz23, sorry for the trouble. I don't think this is the same issue as originally reported, but the end result is the same.

Just to be sure you're on the latest, can you try git checkout 0.1.x and git pull? For those working with a forked personal repo (rather than https://github.com/confluentinc/ksql), you may need to sync with or point at confluentinc's remote to bring in the latest changes.

The culprit was an incorrect version string getting introduced to the compose file as the result of a funky merge. It should be fixed as of aba0f7c.

Can you verify if this is still happening after pulling down the most recent bits?

[jeremyz23]

Thank you @aayars . I have validated that this fix resolved my issue.

[sigmazen]

Hi @aayars
Unfortunately this issue is still occurring for me.
I even tried re-cloning using that specific branch in the command:
git clone -b 0.1.x https://github.com/confluentinc/ksql.git
The values for docker container ls are per the initial issue.
The values for docker-compose ps are per @jeremyz23 screenshot.
:-(

[sigmazen]

Hi @aayars
A bit of an update. I cleared everything out and re-cloned and ran docker-compose up -d
In a separate window I ran docker container ls which you can see shows that both quickstart_ksql-datagen-*** containers do exist albeit for a few seconds ... but something is killing them off:

$ docker container ls
CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS              PORTS                                                    NAMES
317f51bef658        confluentinc/ksql-cli:latest              "perl -e 'while(1)..."   4 seconds ago       Up 2 seconds                                                                 quickstart_ksql-cli_1
1741efd76e2a        confluentinc/ksql-examples:latest         "bash -c 'echo Wai..."   5 seconds ago       Up 3 seconds                                                                 quickstart_ksql-datagen-pageviews_1
553fb2abe229        confluentinc/ksql-examples:latest         "bash -c 'echo Wai..."   5 seconds ago       Up 4 seconds                                                                 quickstart_ksql-datagen-users_1
1d3fc5429075        confluentinc/cp-schema-registry:latest    "/etc/confluent/do..."   6 seconds ago       Up 5 seconds        0.0.0.0:8081->8081/tcp                                   quickstart_schema-registry_1
3bd3e981cd11        confluentinc/cp-enterprise-kafka:latest   "/etc/confluent/do..."   8 seconds ago       Up 6 seconds        0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp         quickstart_kafka_1
e66f2e0b40f8        confluentinc/cp-zookeeper:latest          "/etc/confluent/do..."   9 seconds ago       Up 7 seconds        2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:32181->32181/tcp   quickstart_zookeeper_1

$ docker container ls
CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS              PORTS                                                    NAMES
317f51bef658        confluentinc/ksql-cli:latest              "perl -e 'while(1)..."   11 seconds ago      Up 9 seconds                                                                 quickstart_ksql-cli_1
1d3fc5429075        confluentinc/cp-schema-registry:latest    "/etc/confluent/do..."   13 seconds ago      Up 12 seconds       0.0.0.0:8081->8081/tcp                                   quickstart_schema-registry_1
3bd3e981cd11        confluentinc/cp-enterprise-kafka:latest   "/etc/confluent/do..."   15 seconds ago      Up 13 seconds       0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp         quickstart_kafka_1
e66f2e0b40f8        confluentinc/cp-zookeeper:latest          "/etc/confluent/do..."   16 seconds ago      Up 14 seconds       2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:32181->32181/tcp   quickstart_zookeeper_1

and the docker-compose ps at this point shows the same as above, namely:

$ docker-compose ps
               Name                              Command               State                            Ports
--------------------------------------------------------------------------------------------------------------------------------------
quickstart_kafka_1                    /etc/confluent/docker/run        Up       0.0.0.0:29092->29092/tcp, 0.0.0.0:9092->9092/tcp
quickstart_ksql-cli_1                 perl -e while(1){ sleep 99 ...   Up
quickstart_ksql-datagen-pageviews_1   bash -c echo Waiting for K ...   Exit 1
quickstart_ksql-datagen-users_1       bash -c echo Waiting for K ...   Exit 1
quickstart_schema-registry_1          /etc/confluent/docker/run        Up       0.0.0.0:8081->8081/tcp
quickstart_zookeeper_1                /etc/confluent/docker/run        Up       2181/tcp, 2888/tcp, 0.0.0.0:32181->32181/tcp, 3888/tcp

Cheers

[miguno]

@sigmazen : What OS are you on? And what are the settings for your host machine's Docker installation? For example, how much RAM is available for Docker to use when running any containers?

[sigmazen]

Hi @miguno
Thanks for the reply.
I'm running it on a MacBook Pro 2.5GHz i7 16GB RAM, similar to @kaiwaehner above.
It has 2GB and 4 CPUs assigned to Docker, and here are the runtime values via docker stats which looks to be utilizing around 50% of allotted memory:

$ docker container ls
CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS              PORTS                                                    NAMES
317f51bef658        confluentinc/ksql-cli:latest              "perl -e 'while(1)..."   6 hours ago         Up 6 hours                                                                   quickstart_ksql-cli_1
1d3fc5429075        confluentinc/cp-schema-registry:latest    "/etc/confluent/do..."   6 hours ago         Up 6 hours          0.0.0.0:8081->8081/tcp                                   quickstart_schema-registry_1
3bd3e981cd11        confluentinc/cp-enterprise-kafka:latest   "/etc/confluent/do..."   6 hours ago         Up 6 hours          0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp         quickstart_kafka_1
e66f2e0b40f8        confluentinc/cp-zookeeper:latest          "/etc/confluent/do..."   6 hours ago         Up 6 hours          2181/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:32181->32181/tcp   quickstart_zookeeper_1

$ docker stats
CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
317f51bef658        0.00%               560KiB / 1.952GiB     0.03%               24.4kB / 0B         0B / 0B             1
1d3fc5429075        0.27%               311.8MiB / 1.952GiB   15.60%              16.2MB / 11.6MB     0B / 41kB           42
3bd3e981cd11        1.00%               902.4MiB / 1.952GiB   45.15%              11.9MB / 17.2MB     0B / 19.2MB         75
e66f2e0b40f8        0.17%               73.2MiB / 1.952GiB    3.66%               2.06MB / 1.22MB     0B / 406kB          28

and both quickstart_ksql-datagen-*** containers have the state of Exit 1
Cheers

[sigmazen]

... quick update:
Bumped it up to 4GB available to Docker and same thing happens ... all the containers start and then the two quickstart_ksql-datagen-** containers seem to die off with Exit 1 in a couple of seconds.

[bluemonk3y]

@sigmazen - I have experienced the same (where unix kills off processes to prevent the OS from a hardcrash). The following should reduce the amount of memory confluent platform uses. In conjunction, it would be worthwhile increasing to 6GB of memory.

# export KAFKA_HEAP_OPTS="-Xmx256M"

[sigmazen]

@bluemonk3y thanks for the quick reply.
I tried running the export and increasing to 6GB but to no avail. I've increased further to 8GB but no luck.
I ran the docker stats command in another shell-tab whilst re-running the docker-compose up -d command and can see the two quickstart_ksql-datagen-** processes begin created and beginning to consume around 12Mb and 18Mb each ... but then a second later they're killed like you mention.
In total all the processes would be consuming around 600Mb.
I'm not really sure what the next steps are other than try the non-Docker approach or maybe reinstall Docker?
Cheers

[bluemonk3y]

@sigmazen - depending on the linux flavor, killed processes should be logged in: /var/log/messages
Search for 'out of memory'

info: https://unix.stackexchange.com/questions/128642/debug-out-of-memory-with-var-log-messages

LMK if that helps

[sigmazen]

@bluemonk3y ... thanks again.
Looks like the folder is /var/log/system.log on mac but it didn't really show anything related to this.
However, I did try docker-compose up ie without the -d so that it outputs the logs on the fly.
During the ksql-datagen-** steps it reports this:

ksql-datagen-pageviews_1  | Expected 1 brokers but found only 0. Trying to query Kafka for metadata again ...
ksql-datagen-users_1      | Expected 1 brokers but found only 0. Trying to query Kafka for metadata again ...
ksql-datagen-users_1      | Waiting for Confluent Schema Registry to be ready...
ksql-datagen-pageviews_1  | Waiting for Confluent Schema Registry to be ready...

and then later I get a bunch of html output complaining about my proxy followed by:

quickstart_ksql-datagen-pageviews_1 exited with code 1
quickstart_ksql-datagen-users_1 exited with code 1

which all looks a bit suspect. I had actually tried running this at home as well (ie no proxy) a couple of days back, but I'll retry again tonight.
Fingers crossed.

[bluemonk3y]

@sigmazen - you will need to check the filesystem within the docker container and not the mac host - I hope that makes sense?

[sigmazen]

Hi
Very odd ... so those individual containers datagen-users_1 and datagen-pageviews_1 are failing with 407 proxy issues (see below).
However, this doesn't really make sense because I just git cloned and during the docker-compose up -d step, it downloads all the latest containers via the proxy :-(

Request (type=MetadataRequest, topics=) failed against node kafka:29092 (id: -1 rack: null).
org.apache.kafka.common.errors.DisconnectException
Request (type=MetadataRequest, topics=) failed on all bootstrap brokers [kafka:29092 (id: -1 rack: null)].
Expected 1 brokers but found only 0. Trying to query Kafka for metadata again ...
Waiting for Confluent Schema Registry to be ready...
Unexpected response with code: 407 

[sigmazen]

Hi again @bluemonk3y
Some additional information. As mentioned, the two datagen containers are dying because of a proxy 407 error. However, it might be something to do with them trying to connect to schema-registry.
If I docker-compose up -d then docker ps we can get the container IDs.
Next docker exec -it {zookeeper-containerID} bash and once logged in ss -ltu at which point we can see the correct ports assigned from docker-compose.yml, namely 32181.
Next docker exec -it {enterprise-kafka-containerID} bash and again ss -ltu which gives 9092 and 29092.
But when I'm in docker exec -it {schema-registry-containerID} bash I do not see 8081
Is this as expected?
Cheers

[bluemonk3y]

@aayars - hey alex any ideas on this?

[sigmazen]

Hi @aayars @bluemonk3y ... panic over :-/
In Docker 17.03 there is a 'No Proxy' option but I didn't have that option 17.09
However, there is a Bypass Proxy section.
Typed in localhost,127.*.*.*.
Relaunched and ran docker-compose up -d
Works a treat !!
So sorry for wasting peoples time but thank you all for your help.
Hopefully this last post might help others ;-)
Cheers
Simon

[aayars]

Thanks for the investigation and update @sigmazen! This is good to know.

[miguno]

Happy to hear that it works for you now @sigmazen -- and thanks for your patience to investigate and report back (including reporting back what finally resolved the issue for you). :)

[siralfbaez]

Hi mates,
for me quitting lots of opened/running apps like MS-word/excel even my spotify. Once these were close I restarted the killed processed from the Docker console.... FYI I'm running Docker with 8G or RUM out of 16; my Mac is at max...
Cheers
Alf