Vicky Dineshchandra's final year project for UCL's Urban Dynamics Lab.
The goal of this README is to inform the reader how to run some of the functionality associated with the project. A full explanation of what each component does and why it is required can be found in the final report.
The service is already installed and up and running as required by UDL on UDLTest1. This repository contains a Docker build test, which includes the build instructions for the interface.
Build instructions for Jupyter Notebook Spark Interface are provided in
/jupyter-service
. There are two configuration files provided:
jupyter_notebook_config.py
contains the settings for the JN installationjupyternotebook.service
contains the conf file for the systemd service
Once you've installed the bare bones jupyternotebook service, tap in your virtualenv and type
jupyter toree install --spark_home=$your_spark_path
toree should be installed with the requirements file I have provided.
The main goal of the Dockerfile found in jupyter_notebook/Dockerfile
is
to act like the instructions to install the service.
You can also perform a build test using Docker, it requires the ubuntu
image. (This is the OS for UDLTest1)
docker pull ubuntu
cd jupyter-service
docker build -t jntest
This pipeline is written in Java and uses ant as the build tool. The entire project is unit tested and follows the javadoc documentation style.
Java 8 and ant must be installed
This can be checked with
java -version
ant --version
Simple as
cd pipelines/fetching/AirAPI
ant
This will query relevant APIs and save the results into output
The folders are separated by "years", this can be configured in
pipelines/fetching/AirAPI/src/main/java/AirPollutionGatherer.java
This pipeline comes with unit tests for all classes that were written. To run the tests:
cd pipelines/fetching/AirAPI
ant test
This pipelines creates the postgres db server from the LDD database dump that is provided
Make sure to download the dump and save it in pipelines/dbserve/init
. The LDD
dump can be found at here
First the database server needs to be mounted using Docker
Simply run
cd pipelines/dbserve
docker_up.sh
Once the server is up, remember to call save_data.py
to make SQL
queries to the server, or make calls any other way. For dependencies, you can build a virtualenv from jupyter-experiments/requirements.txt
If there are port mapping issues, this can be simple set in the
docker-compose.yml
file.
Finally, remember to close the db server using
docker_down.sh