Docker Mongo Pipeline

Pipeline:

1. Export users collection from MongoDB as JSON
2. gzip the JSON file
3. Upload JSON data to Google Cloud Storage
4. Convert JSON to a BigQuery table

Setup

1. Fork/Clone this repo

2. Download Docker (if necessary)

3. Make sure you are using a Docker version >= 17:

   $ docker -v
   Docker version 17.03.0-ce, build 60ccb22

4. Create the .env file:

$ cp .env.sample .env

Workflow

1. Update the src/entrypoint.sh script along with the .env file

2. Build and tag the image:

   $ docker build -t sample ./src

3. Run the container:

   $ docker run --env-file .env -it sample