BEES Data Engineering test run doc:

make .env file with your marvel API keys:

MARVEL_PUBLIC_KEY="<your public key>"
MARVEL_PRIVATE_KEY"<your private key>"

build the containers

docker-compose build

start the containers

docker-compose up -d

connecting to spark container (master or worker(s))

docker exec -it bees_spark-master_1 bash

execute task (from inside the bees_spark-master_1 container)

/opt/spark/bin/spark-submit --master spark://spark-master:7077 --driver-memory 1G --executor-memory 1G /opt/spark-apps/main.py

generate visualizations/tables

python3 dev_viz.py

cd visualizations dir

open each generated table in browser