Distributed System in Docker with Apache Kafka and Spark for big data streaming and visualisation (NodeJS, TypeScript, React, NestJS, Java)
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
SparkStreamer
backend
docs
frontend
kafka
.editorconfig
.gitignore
README.md
docker-compose.yml
package-lock.json
package.json
pom.xml

README.md

Apache Kafka and Apache Spark - A Distributed Streaming Project

For detailed project report, please check out The Project Report. ;)

Running the project

You can try out this project running all components in a docker composed cluster, each component will run in a separated container and they will be connected with the default network inside the compose cluster.

Prerequisites:

  • Docker
  • Port 80 should be available or change the port mapping of the frontend app in docker-compose.yml

Run the project:

  • Clone this repository on your computer
  • Fire up the docker compose cluster:
$ docker-compose up
  • Open the frontend application in your browser: $ open http://localhost:80

Shutting down docker-compose (use an other terminal window for running this command):

$ docker-compose down

Notes:

  • The first time building the maven based project is take a while, be patient.
  • Kafka generated files mapped to ./kafka/volumes folder. If the streaming doesn't start when you launch this project first time, please shut down the docker-compose cluster and start again. Second time, all the mapped volume and folder will be available and Kafka can start properly.

Run the project in developer mode

You can run all components locally. In this way you can easily debug and add new features.

Prerequisites:

The setup npm script will install individual packages and prepare the project for you. The start:dev will run all component's development script concurrently in the same terminal.

$ npm run setup
$ npm run start:dev

Documents

Component's README files

Useful links