bdea-workcloud

This project creates word-clouds based on a tfidf calculation for txt-files which you can upload via a web-UI by using a spark cluster with pyspark.

Project

Team

• Anton Bracke
• Jan Mayer
• Julian Becker

Documentation

docs/bdea_wordcloud.pdf

docs/presentation.pdf

Content

• Spark jobs: spark/
• Frontend: frontend/
• Backend: backend/
• Fake HDFS: fake-hdfs/

Development

Devcontainer

This project can be openend as .devcontainer with VS-Code.

Continue with Start backend / frontend

Local development (without devcontainer)

make docker-up

Open a bash inside the app container with: make docker-bash

Continue with Start backend / frontend

Start backend / frontend

make start # start webserver

make start-frontend # start frontend

TODO

• add frontend
• add backend
• add text file upload
• generate wordcloud
• embed wordcloud in frontend
• add database to store document-frequiencies of words
• add manual trigger for batch job to update document-frequiencies of words