Harvest source import/export

[kimwdavidson]

User Story

As a data.gov operator, I want a script or process that can export harvest sources from catalog and import them into catalog-next so that I can keep catalog-next up to date with catalog even as harvest configuration changes.

Acceptance Criteria

• WHEN I follow the import/export process
• THEN catalog-next is configured with the same harvest sources and configuration as catalog. (When I go to /harvest on both, they look the same.)
• WHEN I follow the process additional times
• THEN I only see the changes and the up-to-date version, not another full replication of the database (Though it doesn't need to delete what's already there if it's deleted on the classic catalog)

Details/Tasks for simple import

• Define where these tools should live ---> Agreed in catalog.data.gov repo
• Reuse code from next gen harvester for importing Harvest sources
• Test it is working manually for a few sources
• Create the organization in CKAN needed for the harvester to ingest datasets to the correct organization
• Check that if you add new harvester sources, the import function only adds the diff.
• Check that if you add amend harvester sources, the import function only adds the diff (including the amended source)
• Check that if you delete harvester sources, the import function does not delete the previous harvester sources (if they need to be deleted, you do that manually, but this is not recommended)
• Review the current tests. If we think there should be more then we should add a ticket for this.
• Aaron approves PR

[avdata99]

PRs for this issue:

• #18 adds import/export script: Approved and merged
• #28 adds report for harvest sources
  • Merged
• #30 adds tests to the import script
  • Merged
• #31 update harvest sources when already exists
  • Pending update tests

[avdata99]

We can close this but before we should add issues to add new tests.
Required tests:

• Test if we import the sources correctly (e.g we detect config from extras)
• Test if the organization are imported correctly (e.g check images)

Added issue here: GSA/catalog.data.gov#42

[thejuliekramer]

@kimwdavidson in order to test this we need to run some commands on the sandbox, so I will need the help of @woodt for this

[avdata99]

To QA scripts @thejuliekramer @woodt:

Install the requirements file at /tools/harvest_source_import

Read sources:

cd tools/harvest_source_import
python list_harvest_sources.py --source_type=csw

To import sources (we need an API key from a user with permission to add harvest sources (packages) and organizations

cd tools/harvest_source_import
python import_harvest_sources.py \
    --origin_url=https://catalog.data.gov \
    --destination_url=http://ckan:5000 \
    --destination_api_key=xxxxx-xxxxx-xxxx-xxxxxx \
    --source_type=waf \
    --limit=10

[thejuliekramer]

@avdata99 I am able to run this command with the sandbox api key, but I do not see the datasets listed on the dataset page - the datasets are not private, what do you think is the cause here?

[avdata99]

@avdata99 I am able to run this command with the sandbox api key, but I do not see the datasets listed on the dataset page - the datasets are not private, what do you think is the cause here?

This command just import the harvest sources @thejuliekramer. Didn't run them. So, you only will see the harvest sources at /harvest

[thejuliekramer]

Harvest sources are listed here https://catalog-next.sandbox.datagov.us/harvest, I ran datajson, csw, waf, arcgis, and single doc

[avdata99]

Harvest sources are listed here https://catalog-next.sandbox.datagov.us/harvest, I ran datajson, csw, and waf

Also the orgs with images: