Climate Impact Lab Data Pipelines

The pipelines package is the place to request, test, and submit jobs at the Climate Impact Lab. For help submitting a job, see our docs or ask Justin.

Features

• Tested reshape operations on climate data
• A template for more portions of our pipeline

Usage

1. Create a new branch for your request (git branch my-new-run)
2. In the relevant sector's submodule (e.g. pipelines/climate), look for a template job, e.g. /pipelines/climate/jobs/job_bcsd_template.py

2. Copy this template to a directory for your project (e.g. gcp-labor or impactlab_website)

3. Make sure this folder has an __init__.py file in it. It can be blank.

4. Modify the template to your needs. If you need a new transformation, create one in transformations.py

5. Initialize your pipelines by importing your file, e.g.:

   python -m pipelines.climate.jobs.impactlab_website.my_new_job

6. Run tests: pytest

7. Push your changes to github and file a pull request

Requirements

For now, pipelines requires python 2.7. We're working on 3x support.

Todo

See issues to see and add to our todos.

Credits

This package was created by Justin Simcock and Michael Delgado of the Climate Impact Lab. Check us out on github.