By Anthony Vilarim Caliani
Brawl Stars Analytics project is a simple ETL Pipeline that uses the data from Brawl Stars Developer API.
I'm a Supercell fan and then I have decided to develop this small project using Brawl Stars thematic, of course respecting Supercell's Fan Content Policy. By the way, the content of this repository is not affiliated with, endorsed, sponsored, or specifically approved by Supercell and they are not responsible for it.
Before starting you need a existant GCloud project and the GCloud SDK installed on your machine.
After creating and installing GCloud stuff you have to update devops/.env file properly with your own GCloud data.
🧙♂️ "If in doubt... Always follow your nose" - Gandalf
First, to interact with GCloud via Docker container you have to create a Service Account. Then we can build and up the container.
# 01 - Create the service account
./devops/service-account.sh --create sa-docker
# 02 - Build docker image
docker-compose build
# 03 - Up the container
docker-compose up -d
# When you finish...
docker-compose downNow let's prepare the GCloud environment...
# Creating cloud storage buckets...
./devops/storage.sh --create bstars-repo
./devops/storage.sh --create bstars-logs
./devops/storage.sh --create bstars-transientDeploying "App Collector"...
# 💡 INFO
# - - - - - - - - - - - -
# Before deploying, set the API_TOKEN value in "app-collector/env/prod.env"
# The token can be found at "Brawl Stars Developer API"
./devops/app-deploy.sh app-collectorFinally, create some more services in GCloud...
# Compute Instance with static IP Address
./devops/ip-address.sh --create # Region: us-central1 && Save the IP Address
./devops/compute.sh --create 127.0.0.1 # Generated IP Address
# Cloud Pub/Sub Topic
./devops/pubsub.sh --create
# Cloud Function
./devops/function.sh --create start-instance
# Cloud Scheduler
./devops/schedule.sh --createApp Collector
This job is responsible to request user data from Brawl Stars Developer API and save the data into a JSON file. This job was developed to be executed in a Compute Engine Instance, but you can execute it locally as well.
# 💡 INFO
# - - - - - - - - - - - -
# Before executing locally, set the API_TOKEN value in "app-collector/env/local.env"
# The token can be found at "Brawl Stars Developer API"
docker-compose exec bstars /app/app-collector/run.shApp Processor
This job is responsible to clean, extract and transform the data. This Job was developed to be executed locally. Of course it is also prepared to be executed in GCloud, but this was not my objective.
# 01 - Download files from GCloud
docker-compose exec bstars /app/app-processor/scripts/get-files.sh
# 02 - Process files from transient zone until refined zone
docker-compose exec bstars /app/app-processor/run.sh local raw trusted refined
# 03 - Move file into analytics zone
docker-compose exec bstars /app/app-processor/scripts/single-file.shThis screenshot was taken from Google Data Studio report created by me \o/
