Lessons Learned
Here, the learnings of this projects and suggested improvements can be found
Table of Contents
Overall, making the dash views and the application deployable took quite some time.
The other points were also challenging but the time needed was more predictable and therefore easier to manage. One solution could have been to use a ready-to-go Dashboard solution such as Grafana or Kibana. However, due to the larger scope of the projects and the five team members, using this more complex, but flexible solution was the better choice.
Overall, the team was very satisfied with Dash. First, writing everything in Python enabled the team to leverage their previous knowledge. Second, Dash had all the functionality needed to develop the dashboard.
On the other hand, there were also some negative points. First, the documentation and community was good, but not great. Second, the dashboard needed to be developed from scratch (there were no ready to use components). Third, implementing dynamic changes were quite time consuming and difficult to understand. They were implemented using so called Dash callbacks. These are functions that automatically update a certain content, if the input component's property changes.
- Dashboard creation using a new programming language (Python), different packages (Pandas, Plotly, usw.) and the Dash framework.
- Adding business value using data science and predictions.
- Adding interactivity to the dashboard using callbacks (e.g. for filtering graphs or adding visual effects).
- Make use of open communities (e.g. Dash and Plotly).
- Adjust expectations and adapt to the limitations of the used technologies and resources.
- Working with Python and different libraries/frameworks e.g. Pandas, Dash, Plotly.
- How to connect backend and frontend.
- How to create interactivity using callbacks.
Just as written above, it also applies here:
- Working with Python and different libraries/frameworks e.g. Pandas, Dash, Plotly.
- How to connect backend and frontend.
- How to create interactivity using callbacks.
Apart from that:
- creating differnt charts and graphs including calculations
- working with GitHub as agile team using Scrum, Kanban board and Wiki
- understanding the business value of defined target group (fleet manager and decision owner)
- working efficiently remote as a team
- Deeper inside into python libraries such as pandas, numpy and scikit-learn
- First time training a machine learning algorithm
- How do you prepare your data before training?
- How can you test the fit of your model?
- How to handle categorical variables when trying machine learning?
- Understanding of gradient boosting
- First time deploying an application on google cloud
- Learned a lot about the use of the console and about the system architecture
- Got better at handling multiple dependencies within one project
- Found a lot of new useful features within pycharm and optimized my workflows
- Things to look out for when selecting a dataset
- Would have been easier with a dataset where one row does not represent a completed journey.
- Improved my GitHub workflows
- Conducing the user research
- Creating design concept (design mockup)
- Researching dashboard solutions and data pipelines (batch vs streaming processing)
- Setting up the basic pages with the routing using Python (was new for me)
- Deploying the Dash application (Flask backend) on the Google Cloud Compute Engine using the Gunicorn Webserver
- Styling the dashboard with Bootstrap and Flexbox
- Project and documentation management
- Further learnings are written below:
This project's Dash application can be deployed on the Google Cloud Compute Engine as follows:
- Create a Google Cloud account and setup VM instance on Google Cloud
Smallest machine configuration (f1), leave boot disk on standard (Debian), Access scope (Allow full access to all Cloud APIs), Firewall (allow HTTP and HTTPS)
- Connect to the instance via the Cloud Shell oder Terminal
gcloud beta compute ssh --zone "europe-west3-c" "fleetboard-database" --project "abiding-orb-278309"
- Installing virtualenv and python (https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/)
sudo apt install python3.7 python3-dev python3-venv python3-pip libffi-dev libssl-dev git
- Cloning git repository
git clone https://github.com/Fleet-Analytics-Dashboard/Application.git
- Going into Application folder
cd Application/
- Creating a virtual environment for installing dependencies just for this application
python3 -m venv env
- Activating the virtual environment
source env/bin/activate
(8. Install dependencies -> step can be skiped, because dependency problems (9. and 10.) needs to be solved before)
pip3 install -r requirements.txt
- Installing libpq-dev for fixing "pg_config executable not found"
sudo apt-get install libpq-dev
- Installing wheel for installing Dash "Failed building wheel" (https://stackoverflow.com/questions/53204916/what-is-the-meaning-of-failed-building-wheel-for-x-in-pip-install)
pip3 install wheel
- Install fixed dependencies
pip3 install -r requirements.txt
- Run gunicorn web server
gunicorn -b 0.0.0.0:8080 main:server
- "main" points to the "main.py" file
- "server" points to the server of the "application" in the "main.py" file (see explanation here and the respective GtiHub file here )_
- Go to the website
- Example http://34.107.70.0:8080/
- the port and "/" at the end are important!!!
Changing branches may lead to address already in use issue. After changing a branch, a local Python application does not terminate the previous application. Therefore, it must be terminated (OSError: [Errno 48] Address already in use). The solution is as follows
- Open a terminal
- ps -fA | grep python (lists all running Python applications with its respective number)
- kill (kills the respective application)
- Now the app can again run on the local standard port (it was freed up)
Fix unable to install pscopg2 database connector on Mac OS as follows
- Change pscopg2 to pscopg2-binary in the requirements.txt
- Install per pip3 install -r requirements.txt again and done
Solving the problems with wheel packages which has the following error output: from psycopg2._psycopg import ( # noqa ImportError: dlopen(/Users/luciejuergens/PycharmProjects/untitled2/venv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-darwin.so, 2): Library not loaded: @rpath/libssl.1.1.dylib Referenced from: /Users/luciejuergens/PycharmProjects/untitled2/venv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-darwin.so Reason: image not found
- pip3 uninstall psycopg2
- pip install --pre -i https://testpypi.python.org/simple psycopg2
When executing the main.py, this error occures:
AttributeError: module 'enum' has no attribute 'IntFlag'?
This can be fixed by deactivating "Activate Google App Engine Support" in the preferences of Pycharm
Resource: stackoverflow - App Engine Problem
Branch naming
- Small letters
- Only dashes
- No whitespaces
- No special characters (e.g. #,%,&,...)
- Examples: 14-setup-dash-mvp