Apache Kafka and Apache Spark - A Distributed Streaming Project
For detailed project report, please check out The Project Report. ;)
Running the project
You can try out this project running all components in a docker composed cluster, each component will run in a separated container and they will be connected with the default network inside the compose cluster.
- Port 80 should be available or change the port mapping of the
Run the project:
- Clone this repository on your computer
- Fire up the docker compose cluster:
$ docker-compose up
- Open the frontend application in your browser:
$ open http://localhost:80
Shutting down docker-compose (use an other terminal window for running this command):
$ docker-compose down
- The first time building the maven based project is take a while, be patient.
- Kafka generated files mapped to
./kafka/volumesfolder. If the streaming doesn't start when you launch this project first time, please shut down the docker-compose cluster and start again. Second time, all the mapped volume and folder will be available and Kafka can start properly.
Run the project in developer mode
You can run all components locally. In this way you can easily debug and add new features.
- Locally installed Node.js (How to Install Node.js)
- Locally installed Java 8 and Maven 3.5
- Locally installed Kafka with Zookeeper (Kafka Quickstart)
- Locally installed Spark (Install Spark)
setup npm script will install individual packages and prepare the project for you. The
start:dev will run all component's development script concurrently in the same terminal.
$ npm run setup $ npm run start:dev
Component's README files
- API framework, Nest.js original README
- Frontend framework, React.js original README