Data gov Team Workflow for Harvest Source Requests

Data.gov Team Workflow for Harvest Source Requests

Once an agency submits a request for a new DCAT-US 3 harvest source or a change to an existing harvest source, the Data.gov team reviews and processes the request through the Harvest Source Management workflow.

Initial Review and Testing

• Confirm and test the source URL link.
• Add the new harvest source to the Data.gov Harvester development environment.
• Send the Harvester development environment test report to the agency point of contact.
• Confirm with the agency point of contact whether the request will follow Path A or Path B.

Path A: Add a New DCAT-US 3 Harvest Source While Temporarily Maintaining the Existing Source

• Set up a new, additional harvest source record so the agency has an additional DCAT-US 3 harvest source.
• Turn on harvesting for the new harvest source.
• Allow the agency to transition datasets or other metadata records from the DCAT-US 1.1 harvest source to the DCAT-US 3 harvest source.
• Wait for the agency to confirm that all datasets or other metadata records have been transitioned.
• Clear and then delete the old harvest source.
• Send confirmation to the agency point of contact.
• Note any troubleshooting items in the request comments or tracking record.

Path B: Update an Existing Harvest Source to DCAT-US 3

• Confirm the date to stop harvesting the old harvest source.
• Make note of harvest source timing as daily, weekly, monthly, or other.
• Edit the harvest source to stop harvesting and send notification to the agency.
• Wait for the agency to switch, modify, or edit its JSON file and send confirmation back that changes have been made.
• Edit the harvest source to reference DCAT-US 3 and resume harvesting as previously noted.
• Send confirmation to the agency point of contact.
• Note any troubleshooting items in the request comments or tracking record.

Final Acceptance Criteria

• A new or updated metadata harvest source exists under the correct parent organization.
• The harvest source uses the confirmed source URL.
• The source is configured for the appropriate metadata format/profile.
• An initial harvest has completed successfully.
• Any harvest errors or validation issues have been reviewed and documented.
• Follow-up actions, if any, have been captured in the request issue or linked issues.