Stream Bitcoin Price Training Data on Apache Kafka using Schema Registry as Avro Format sink to BigQuery.
- Provide your Service Account Key File as 'creds.json' for BigQuery inside .credentials/.
docker compose up
to start everything.docker logs --follow app-producer
on new terminal. Please look at the end line after 'Start Producer App'.docker logs --follow app-consumer
on new terminal. Please look at the end line after 'Start Consumer App'.- You should see something similar as shown on imgs/.
- You can also check on the Kafka Control Center for better UI!
- Don't forget to end up everything
docker container stop $(docker ps -a -q); docker system prune; docker volume prune; docker container prune
. Because the data stream to the BigQuery, unless you have no worry for the Billing!
- A little bit lagging when streaming data to BigQuery (push after pull 2 messages), unless you use BigQuery Sink Connector.