AWS Server-less data pipelines with Terraform
Code supporting Blog post series AWS Server-less data pipelines with Terraform.
Though not a requirement, we recommend using Anaconda to manage python dependencies. Here are the steps that we will walk you though:
- Setup Anaconda.
- Setup conda environment
- Setup local env
To download Anaconda package manager, go to: https://www.continuum.io/downloads.
After installing locally the conda environment, proceed to setup this project environment.
Setup local conda environment
For dependency management we are using conda-requirements.txt and requirements.txt. Please "cd" into the current reposotory and build your conda environment based on those conda-requirements and requirements:
conda create -n analytics python=3.6 source activate analytics conda install --file conda_requirements.txt pip install -r pip_requirements.txt
To deactivate this specific virtual environment:
If you need to completely remove this conda env, you can use the following command:
conda env remove --name analytics
brew install terraform
- In order to automatic format terraform code (and have it cleaner), we use pre-commit hook. To install pre-commit.
- Run pre-commit install to setup locally hook for terraform code cleanup.
Planning & applying changes
In order to automate and simplify the provisioning steps, we provide a Makefile, which can be used for:
- deploy whole Terraform setup by simply running "make apply";
- build and deploy individual Lambda function(s);
Before running any 'make' command, we recommend exporting the specific environment you want to use. For example:
In order to safely view the executions steps that will be executed, it is a best practice running a "terraform plan" You can do so via the make file which we provide at the root level of this directory, simply by running the following command:
To apply Terraform changes, use:
See the list of contributors who participated in this project.