Local airflow with load code for local csv converting to json

[hannelita]

local airflow with load code for local csv converting to json. Part of Epic #1

Acceptance

• Parameterized task on examples/folder; receives a csv file and outputs a pretty printed json file

Tasks

• Define airflow operator (PythonOperator for now)
• Libraries - converting with pandas
• Specify in/out directories in kwargs
• Json Pretty Printing
• wrap up DAG code

Analysis

Questions

• How to deal with invalid CSV files? This may deserve another issue on the tracker
  • ~rufus definitely deserves another item. Here you assume the CSV's are "tidy/good"
• tests? (unsure how to do them for this example case)
  • ~rufus - what are you testing? there's the core lib for processing and airflow itself. For airflow itself it would be more UAT like e.g. calling the API and checking there's some output on disk
• Formatting scripts/libraries? Ideally set them up with tests (i.e. run make test and also get feedback from formatting)
  • ~rufus: can you clarify what you mean by formatting scripts / lib.

[rufuspollock]

@hannelita can we get acceptance section items on this? I've also formatted sections as https://playbook.datopian.com/issues-tasks/ and answered the questions.

[hannelita]

Formatting libs => Create a pytest.ini with flake8, isort and other libraries.

[hannelita]

Update: @sglavoie will review the PR and help with tests setup

[rufuspollock]

FIXED. Merged in PR #6