Skip to content

WallarooLabs/streaming-avro-over-kafka

Repository files navigation

Avro POC

HOWTO run the demo

  1. make sure you have machida in your $PATH, wallaroo.py in $PYTHON_PATH, and a virtualenv activated, with ./requirements.txt installed

  2. make start-kafka setup-topics (TERMINAL 1) Downloads kafka if not downloaded already, starts single broker, creates input and output topics.

  3. make start-app (TERMINAL 1) Start Wallaroo app app.py, which connects to the Kafka broker.

  4. make tail-output (TERMINAL 2) Opens up a Kafka console consumer and displays the output published by the Wallaroo app.

  5. make start-generator (TERMINAL 3) Starts up Python script generator.py which continuously sends random "sensor readings" to the input topic.

make test - runs unit tests for avro (de)serializer make reset - destroy kafka/zk state. WARNING! here be rm -rfs and pkills

About

A demonstration of processing data serialized with Apache Avro via Apache Kafka

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published