Migrate datastore-next-library code to this repo

[hannelita]

Acceptance

• previous datahub-next-loader code into lib/ folder
• DAG file invoking loader
• Mocks Moved to Mocks for library #15

Tasks

• Copy over existing datastore loader code from https://gitlab.com/datopian/tech/data-loader-next-gen
• Create invoking operator for each task individually and making sure they run properly
  • delete_datastore_table
  • create_datastore_table
  • load_csv_to_postgres_via_copy
• Create mocks for existing functions See Mocks for library #15

Analysis

Questions

• On load_csv_to_postgres_via_copy, we'll probably need a refactor, since it returns a generic error

except Exception as e:
                    return {
                        'success': False,
                        'message': 'Error during deleting index: {}'.format(e)
                    }

• The original library uses .env vars. Here I'm switching to airflow vars. Pros: Easy to manage everything in airflow UI; cons: code will be tied to airflow; there is a simple way to refactor it and make it entirely agnostic \cc @rufuspollock

[rufuspollock]

The original library uses .env vars. Here I'm switching to airflow vars. Pros: Easy to manage everything in airflow UI; cons: code will be tied to airflow; there is a simple way to refactor it and make it entirely agnostic \cc @rufuspollock

Sounds good.

[hannelita]

Questions: To run https://github.com/datopian/aircan/blob/master/lib/load.py#L155, a postgres connection is required. I am assuming, for a local ckan instance, it will be postgresql:///localhost/datastore_test, or postgresql:///ckan:ckan@localhost/datastore_test; not working. Do I need to perform any setup in my local isntance?

Tried: demo.ckan; I found it easier to understand the flow locally.

[rufuspollock]

@hannelita the postgres link will be set in the ini file of the local ckan setup i.e. check the docker file config or ckan ini for that info.

[hannelita]

Remarks:

• Testing on a local CKAN and on demo.ckan is not the same. With a local instance, I managed to get access to a postgres and understand the flow. Not sure how to do that for demo.ckan.
• Every datastore table comes with _id and _full_text fields. Fun fact: the COPY command for the CSV was throwing errors because it takes these two fields into consideration when parsing. This needs to be fixed. See load_csv_to_postgres_via_copy considers _id and _full_text #14
• We need better error messages/error handling
• Work on a better var config system, as addressed in DataStore load from local file (library + single node DAG) #10

[rufuspollock]

@hannelita there is no way to do direct to DB load on demo.ckan.org (that won't be possible).

[rufuspollock]

FIXED. All done - see description and moved mocks to #15