Python Pipeline Sketch

An initial go and working our the methods, functions and general flow of a python only data ingress pipeline.

I've written this ina Jupyter notebook, to run it in this fashion, clone this repo and:

pipenv syv
pipenv run jupyter lab

Alternatively, you could just read the notebook.

The functions and methods here are all either empty or returning the minimal hard coded return. The intention here is to just work out the skeleton of what pytonh-only pipeline code in such a way that it makes sense regardless of the runner that's used to power the pipeline.

You should be able to follow the logic by reading though pipeline.ipynb.