Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local airflow with load code working against CKAN - API DAG #4

Closed
2 of 9 tasks
hannelita opened this issue Jun 8, 2020 · 5 comments
Closed
2 of 9 tasks

Local airflow with load code working against CKAN - API DAG #4

hannelita opened this issue Jun 8, 2020 · 5 comments
Assignees
Labels
duplicate This issue or pull request already exists wontfix This will not be worked on

Comments

@hannelita
Copy link
Contributor

hannelita commented Jun 8, 2020

Build a DAG that loads a file to CKAN ( remote demo.ckan) using API only

Acceptance

  • Run this DAG from a local airflow instance and have a file loaded on remote CKAN

Tasks

Analysis

  • Can we use Airflow's SimpleHttpOperator/HttpSensor ? We may be able to do the integration purely via CKAN API
    • ~Rufus: not sure i understand? You are writing via the API but source data is one disk right ...
@rufuspollock
Copy link
Member

As mentioned this could be local or remote CKAN - there's really very little difference I think. May be actually easier initially to run against demo.ckan.org.

Added responses to comments.

@hannelita
Copy link
Contributor Author

@rufuspollock sorry, maybe I misunderstood some things. In datapuhser, we set the local url and the site_url. I thought we could use these HttpOperators from airflow to emulate datapuhser.

@hannelita hannelita changed the title Local airflow with load code working against local CKAN from a local CSV Local airflow with load code working against CKAN - API DAG Jun 8, 2020
@hannelita
Copy link
Contributor Author

See #7 (WIP and draft)

@hannelita
Copy link
Contributor Author

hannelita commented Jun 9, 2020

Q: CKAN question: Am I not allowed to send this json {'resource_id': 'test1111', 'fields': [{'name': 'Goal', 'type': 'text'}], 'records': None, 'force': True} to api/3/action/datastore_create ?

I get a 400 from CKAN: {"help": "https://demo.ckan.org/api/3/action/help_show?name=datastore_create", "success": false, "error": {"data": {"fields": ["name", "type"], "force": "True", "resource_id": "test1111"}, "message": "Only lists of dicts can be placed against subschema ('fields',), not <type 'list'>", "__type": "Integrity Error"}}

@hannelita
Copy link
Contributor Author

hannelita commented Jun 9, 2020

DUPLICATE / WONTFIX. Closing this for now, focusing on #9. Rewriting this task at #10

@rufuspollock rufuspollock added wontfix This will not be worked on duplicate This issue or pull request already exists labels Jun 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue or pull request already exists wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

2 participants