dockerized wordcount example #17
Conversation
|
Cool, thanks! Will give this a go tomorrow. |
| #RUN pip install -e git+https://github.com/confluentinc/confluent-kafka-python.git#egg=confluent-kafka | ||
| RUN pip install -e ../../ | ||
|
|
||
| CMD ["python", "source_client.py"] |
There was a problem hiding this comment.
Getting this:
$ docker logs docker_source_client_1
python: can't open file 'source_client.py': [Errno 2] No such file or directory
I guess this file is just missing, could you add it?
There was a problem hiding this comment.
Sorry, my fault...added now!
|
|
||
| WORKDIR /code/examples/wordcount/ | ||
| RUN pip --version | ||
| #RUN pip install -e git+https://github.com/confluentinc/confluent-kafka-python.git#egg=confluent-kafka |
There was a problem hiding this comment.
Probably the pip install below should pull this in, do you remember why you have the commented out pip install confluent-kafka-python here?
Getting this:
$ docker logs docker_wordcount_1
Traceback (most recent call last):
File "example.py", line 13, in <module>
from winton_kafka_streams.processor import BaseProcessor, TopologyBuilder
File "/code/winton_kafka_streams/processor/__init__.py", line 10, in <module>
from ._stream_thread import StreamThread
File "/code/winton_kafka_streams/processor/_stream_thread.py", line 10, in <module>
from confluent_kafka import KafkaError
ModuleNotFoundError: No module named 'confluent_kafka'
There was a problem hiding this comment.
I think the reason this does not get pulled in automatically for us might be a quirk in our local setup @ah-
There was a problem hiding this comment.
confluent_kafka is a dependency in your setup.py, so with RUN pip install -e ../../ it should be pulled in...Can you find out if it doesn't get installed (add RUN pip freeze to the Dockerfile in the wordcount folder) or if there is a different problem? you can build the service with docker-compose build wordcount
There was a problem hiding this comment.
This all runs and looks good and we are happy to merge as it is.
Thanks @sebastianneubauer !
|
Great thanks! If I have time I will look into the other examples too... |
|
Yes, that sounds like a good idea. Initially was going to look at https://github.com/dpkp/kafka-python/blob/master/test/fixtures.py and https://github.com/edenhill/trivup and just re-use one of them, but Docker would make it a bit easier to get it to work reliably. |
|
@sebastianneubauer : a quick note to say that we are considering requiring a CLA to be signed for contributions so I'll be in touch next week if this goes ahead. Looking forward to your docker-compose based test harness - this will be a big help to ensure consistent testing. |
This add a docker-compose project to the wordcount example. Additionally to the native execution, the very same example can now be executed with docker in several services:
If you like this, one can easily dockerize the other examples and even the integration tests can be automated using docker-compose
If you have any questions, just ask, this is an early demo, I am happy to incorporate any suggestions...