avro-schema-kafka

Stream Bitcoin Price Training Data on Apache Kafka using Schema Registry as Avro Format sink to BigQuery.

1. Provide your Service Account Key File as 'creds.json' for BigQuery inside .credentials/.
2. docker compose up to start everything.
3. docker logs --follow app-producer on new terminal. Please look at the end line after 'Start Producer App'.
4. docker logs --follow app-consumer on new terminal. Please look at the end line after 'Start Consumer App'.
5. You should see something similar as shown on imgs/.

<sub>Notes:
- You can also check on the Kafka Control Center for better UI!
- Don't forget to end up everything docker container stop $(docker ps -a -q); docker system prune; docker volume prune; docker container prune.
Because the data stream to the BigQuery, unless you have no worry for the Billing!
- A little bit lagging when streaming data to BigQuery (push after pull 2 messages), unless you use BigQuery Sink Connector.</sub>