Data visualizer from SaveEcoBot. Calculate AQI

Читати Українською 🇺🇦

This software sets up and runs a nice dashboard with metrics from the CSV file and calculate AQI for PM2.5 and PM10.

Air Quality Index calculations are based on this document.

More screenshots and usage examples here.

---

MENU

• Required software
• Usage
  • Quick start
  • Daily usage
    • Start
    • Stop
  • Sensors data
    • Process new data
    • Add new data
    • Remove data
  • Full cleanup
• For contributors
• Future plans
• Want to help?
• License and Copyrights

---

Required software

• Docker
• docker-compose (for Linux)

Usage

Quick start

1. Clone this repo

git clone git@github.com:MaxymVlasov/eco-data-visualizer.git

or download it as zip and unpack.

2. Choose SaveEcoBot station on this map and click 'Details'
   

3. On the bottom you'll see Download raw data (CSV)
   
   click on the link and save CSV-file to ./data/original_data/ inside downloaded repo.

4. Open the terminal in the root of eco-data-visualizer and run:

# Data preparation
docker build -t data-transformer ./data-transformer-app
docker run -v "$PWD"/data/:/app/data/ --rm data-transformer
# Run Grafana and DBs
docker-compose up -d
# Add data of sensors to InfluxDB
docker build -t add_influx_data ./provisioning/influx
docker run -v "$PWD"/data/influx/:/influx-data/ --rm --network=eco-data-visualizer_default add_influx_data

<sup>Depending on your internet bandwidth, CPU, Storage I/O, CSV file size, and the number of processed files First Init may take different times.
For example, in laptop with 100Mbit/s bandwidth, Intel Core i7-8550U (max clock speed 4Ghz), SSD disk and:
- 2 CSV files (together: 620MB) it takes 8m11s (6m09s to transform data)
- 1 CSV file (513MB) - 5m27s (3m47s to transform data)
- 1 CSV file (107MB) - 3m42s (2m to transform data)</sup>

5. Open http://localhost/ for see visualizations!

Daily usage

Start

To start visualization open terminal in the root of the repo and run:

docker-compose up -d

Then open http://localhost/ to see visualizations.

Stop

To stop visualization open terminal in the root of the repo and run:

docker-compose stop

Sensors data

Process new data

1. Download CSV file from SaveEcoBot station
2. Move it to data/original_data folder in the repo.
3. Run:

# Remove temporary files
docker run -v "$PWD"/data/:/app/ --rm amancevice/pandas:1.0.3-alpine sh -c "rm -f /app/csv/*.csv /app/influx/*.influx"
# Data preparation
docker build -t data-transformer ./data-transformer-app
docker run -v "$PWD"/data/:/app/data/ --rm data-transformer

Add new data

To add new data, open terminal in the root of the repo and run:

# Start services
docker-compose up -d
# Add new data
docker build -t add_influx_data ./provisioning/influx
docker run -v "$PWD"/data/influx/:/influx-data/ --rm --network=eco-data-visualizer_default add_influx_data

Remove data

To remove sensors data, open terminal in the root of the repo and run:

docker-compose down
docker volume rm eco-data-visualizer_sensors-data

Full cleanup

To clean up, open terminal in the root of the repo and run:

# Stop services
docker-compose down
# Remove volumes with settings and sensors data
docker volume rm eco-data-visualizer_grafana-settings eco-data-visualizer_sensors-data
# Remove temporary files
docker run -v "$PWD"/data/:/app/ --rm amancevice/pandas:1.0.3-alpine sh -c "rm -f /app/csv/*.csv /app/influx/*.influx"

For contributors

Use same instructions as in Quick start section but in step 4 add --build-arg ENV=dev to data-transformer-app.

docker build --build-arg ENV=dev -t data-transformer ./data-transformer-app

Future plans

• Add Ukrainian localization
  • Code and message dashboard localization
  • Docs localization
• In Grafana Create personal graphs for each sensor with own good-bad color limits and so on as for AQI
• Grab exist metrics from 'phenomenon' colum, use SENSORS content only for user friendly names and localization
• Add AQI support for everything specified in the doc
• Optimize data-transformer-app
  • Parallel sensors operation execution
  • Use less Disk I/O operations

Want to help?

You can:

• Improve this software (see Future plans section).
• Donate to SaveEcoBot.
• Assemble or buy Air quality monitoring station and connect it to SaveEcoBot. SaveDnipro can assemble and connect it for you. Buy here.

License and Copyrights

This software is licensed by Apache License 2.0.

All data from SaveEcoBot is licensed by Creative Commons Attribution License 4.0 International.

Other data and sources can be licensed in a different way.

Also, by using the code provided in this repository you agree with the following:

• Russia has illegally annexed Crimea in 2014 and brought the war in Donbas followed by full-scale invasion of Ukraine in 2022.
• Russia has brought sorrow and devastations to millions of Ukrainians, killed hundreds of innocent people, damaged thousands of buildings, and forced several million people to flee.
• Putin khuylo!