Local airflow with load code working against CKAN - API DAG

[hannelita]

Build a DAG that loads a file to CKAN ( remote demo.ckan) using API only

Acceptance

• Run this DAG from a local airflow instance and have a file loaded on remote CKAN

Tasks

• Check https://gitlab.com/datopian/tech/data-loader-next-gen/-/blob/master/data_loader/load.py

• Copy over relevant parts (+ tests) into local /lib/ folder

• Create appropriate airflow tasks + DAG using this lib methods: ~rufus debatable if this is worth it

  • delete_datastore_table
  • create_datastore_table
  • load_csv

• Is there any retry policy? ~rufus on what conditions. Prob not for now

Analysis

• Can we use Airflow's SimpleHttpOperator/HttpSensor ? We may be able to do the integration purely via CKAN API
  • ~Rufus: not sure i understand? You are writing via the API but source data is one disk right ...

[rufuspollock]

As mentioned this could be local or remote CKAN - there's really very little difference I think. May be actually easier initially to run against demo.ckan.org.

Added responses to comments.

[hannelita]

@rufuspollock sorry, maybe I misunderstood some things. In datapuhser, we set the local url and the site_url. I thought we could use these HttpOperators from airflow to emulate datapuhser.

[hannelita]

See #7 (WIP and draft)

[hannelita]

Q: CKAN question: Am I not allowed to send this json {'resource_id': 'test1111', 'fields': [{'name': 'Goal', 'type': 'text'}], 'records': None, 'force': True} to api/3/action/datastore_create ?

I get a 400 from CKAN: {"help": "https://demo.ckan.org/api/3/action/help_show?name=datastore_create", "success": false, "error": {"data": {"fields": ["name", "type"], "force": "True", "resource_id": "test1111"}, "message": "Only lists of dicts can be placed against subschema ('fields',), not <type 'list'>", "__type": "Integrity Error"}}

[hannelita]

DUPLICATE / WONTFIX. Closing this for now, focusing on #9. Rewriting this task at #10