Cloud-Native Deployment

Cloud-native microservice application to predict humidity and temperature by defining a workflow in Airflow.

DAG (Directed Acyclic Graph)

• PrepareEnviroment: Create the folder specified in the Airflow variable named path_workflow in case it does not exist. This will be our main working directory.

  

  You can create such variable directly via web or from command line:

  airflow variables -s path_workflow <working_directory>
  

• GetDataHumidity/GettDataTemperature: Download and unzip the data related to humidity and temperature in our working directory.

• ProcessData: The downloaded data files are merged from the datetime column by extracting only that column and the San Francisco column for both humidity and temperature. The rows with missing data are then removed and the new dataset is stored in MongoDB.

• TrainARIMA/TrainRF: Extract datasets from the database, create models by training the humidity and temperature sets with ARIMA and Random Forest, and store them in different pickle files in .models folder.

• CloneRepo Clone this repository into <working_directory>/services folder.

• RunUnitTestsV1/RunUnitTestsV2: Run unit tests for both versions to check that everything is OK before deployments.

• DeployAPIv1/DeployAPIv2: Build Docker images from the Dockerfile deploying the microservices on different ports (port 5000 for ARIMA version and 8000 for RF version).

• CleanUp: Intermediate files that have been created and are no longer useful are deleted.

Deployments

Author

Mar Alguacil