Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataStore load from local file (library + single node DAG) #10

Closed
11 tasks done
hannelita opened this issue Jun 9, 2020 · 3 comments
Closed
11 tasks done

DataStore load from local file (library + single node DAG) #10

hannelita opened this issue Jun 9, 2020 · 3 comments

Comments

@hannelita
Copy link
Contributor

hannelita commented Jun 9, 2020

Add DataStore loading via API to library and build a DAG that loads a (configurable) file to (configurable) CKAN instance (either local and/or demo.ckan) using API only.

Acceptance

Tasks

Analysis

Questions

@hannelita
Copy link
Contributor Author

hannelita commented Jun 12, 2020

@rufuspollock For the DAG, we will have a choice:

  1. an airflow oriented approach, which specific hooks/operators from Airflow. Pros: Fine grained control of each step, the DAG can be composed of several tasks (delete, create, load, create_indexes)
  2. A generic approach using the PythonOperator only. Can be turned into a DAG of a single task (IMHO we would not make a good use of airflow) or multiple tasks (I'd need to find out how to pass the response of a task to another)

As discussed, we may go with opt 2

Both will require manipulations in the load library (see #14 + improve method responses [create will need to return the resource_id; parameterize initial JSON path]).

@rufuspollock rufuspollock changed the title DAG refactor with API load DataStore load via from local or remote file (library + DAG) Jun 12, 2020
@hannelita
Copy link
Contributor Author

Awaiting review of #18 to possibly close this issue

@hannelita hannelita changed the title DataStore load via from local or remote file (library + DAG) DataStore load from local file (library + single node DAG) Jun 16, 2020
@hannelita
Copy link
Contributor Author

FIXED with #17 and #18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant