-
Notifications
You must be signed in to change notification settings - Fork 5
Harvest Source Change Requests
This page explains how agencies can request a new DCAT-US 3 harvest source or request a change to an existing harvest source.
Agencies should use the Harvest Source Change Request Form when they need Data.gov to add, update, transition, or otherwise modify a metadata harvest source.
| Topic | Details |
|---|---|
| Request form | https://touchpoints.app.cloud.gov/touchpoints/ffb7e9d0 |
| Use this form for | New DCAT-US 3 harvest sources or changes to existing harvest sources |
| Review current harvest sources | https://harvest.data.gov |
| Validate proposed harvest source | https://harvest.data.gov/validate/ |
| Request tracking | Harvest Source Management Board |
| Tracking board | https://github.com/orgs/GSA/projects/431 |
| Related implementation guidance | https://resources.data.gov/resources/dcat-us-3-implementation/ |
Resource:
https://touchpoints.app.cloud.gov/touchpoints/ffb7e9d0
The Harvest Source Change Request Form is the agency-facing intake mechanism for requesting either:
- A new DCAT-US 3 harvest source
- A change to an existing harvest source
Agencies should submit this form when they need the Data.gov team to create, update, transition, or modify a harvest source.
Agencies should use the Harvest Source Change Request Form when they need to:
- Request creation of a new DCAT-US 3 harvest source
- Request a change to an existing harvest source
- Update the source URL for a harvest source
- Change a harvest source from DCAT-US 1.1 to DCAT-US 3
- Correct or update harvest source configuration information
- Request other harvest-source-related changes needed to support metadata harvesting by Data.gov
Before submitting the Harvest Source Change Request Form, agencies should review implementation guidance, confirm current harvest source information, and validate the proposed or updated harvest source.
Resource:
https://resources.data.gov/resources/dcat-us-3-implementation/
The DCAT-US 3 Implementation Guide is the primary reference for agencies implementing the DCAT-US 3 metadata standard.
Agencies should use this guide to understand:
- DCAT-US 3 requirements
- Metadata field expectations
- Required, recommended, and optional elements
- Metadata structure
- Data.gov harvesting considerations
- Data.gov display or discoverability considerations
Resource:
https://harvest.data.gov
Agencies can view their current harvest sources at harvest.data.gov.
Agencies should review existing harvest source information before submitting a request. This helps determine whether the agency needs to:
- Add a new DCAT-US 3 harvest source
- Update an existing harvest source
- Replace an existing harvest source
- Retire an old harvest source
- Transition from DCAT-US 1.1 to DCAT-US 3
- Correct current harvest source configuration information
Resource:
https://harvest.data.gov/validate/
Agencies can use the Harvest Source Validation Tool to check whether a proposed or updated harvest source is expected to pass validation.
Agencies should validate before submitting a request to help:
- Confirm that the source URL is accessible
- Confirm that the metadata file is structured correctly
- Identify validation errors or warnings
- Resolve metadata issues before Data.gov review
- Reduce follow-up questions during request processing
If validation issues remain, agencies should document known issues and be prepared to discuss them with the Data.gov team.
When submitting a harvest source change request, agencies should be prepared to provide information such as:
- Agency name
- Agency point of contact
- Current harvest source information, if applicable
- Proposed source URL
- Whether the request is for a new source or a change to an existing source
- Whether the request involves transition from DCAT-US 1.1 to DCAT-US 3
- Validation results, if available
- Known validation errors or warnings, if applicable
- Desired timing or transition considerations
- Any additional context needed for Data.gov review
Before submitting the Harvest Source Change Request Form, agencies should complete the following checklist.
- Reviewed the DCAT-US 3 Implementation Guide.
- Reviewed the DCAT-US 3 to M-25-05 Crosswalk, if applicable.
- Checked the DCAT-US 3 Technical Implementation FAQs.
- Created or updated the DCAT-US 3 compliant metadata file.
- Reviewed current harvest sources at https://harvest.data.gov.
- Confirmed whether the request is for a new harvest source or a change to an existing harvest source.
- Confirmed whether an existing source should be updated, replaced, or retired.
- Validated the proposed or updated source at https://harvest.data.gov/validate/.
- Reviewed validation results.
- Addressed validation errors or warnings where possible.
- Confirmed the source URL.
- Identified the agency point of contact for follow-up.
- Documented any known issues or transition considerations.
- Submitted the Harvest Source Change Request Form at https://touchpoints.app.cloud.gov/touchpoints/ffb7e9d0.
Agencies should submit the Harvest Source Change Request Form here:
Harvest Source Change Request Form:
https://touchpoints.app.cloud.gov/touchpoints/ffb7e9d0
Once submitted, the request will be reviewed by the Data.gov team.
After a harvest source change request is submitted, the Data.gov team reviews and processes the request.
Requests submitted through the Harvest Source Change Request Form are tracked on the Harvest Source Management Board:
https://github.com/orgs/GSA/projects/431
The Harvest Source Management Board is used to track:
- Requests submitted through the Harvest Source Change Request Form
- Requests for new DCAT-US 3 harvest sources
- Requests to update or change existing harvest sources
- Agency communications
- Notes received from agencies
- Internal notes related to request review or implementation
- Current stage or status of each request
- Follow-up actions or pending questions
- Troubleshooting items
- Coordination needed with agency representatives or Data.gov team members
After receiving a harvest source change request, the Data.gov team generally completes the following initial review and testing tasks:
- Confirm and test the source URL link.
- Add the new harvest source to the Data.gov Harvester development environment.
- Send the Harvester development environment test report to the agency point of contact.
- Confirm with the agency point of contact whether the request will follow Path A or Path B.
The purpose of this initial review is to confirm that the proposed source URL is accessible, can be tested in the Harvester development environment, and is ready for the appropriate transition approach.
After initial testing, the Data.gov team and agency point of contact will determine whether the request should follow Path A or Path B.
Path A is used when an agency will have a new, additional DCAT-US 3 harvest source created while the existing harvest source remains available temporarily.
This allows the agency to transition datasets or other metadata records from the existing DCAT-US 1.1 harvest source to the new DCAT-US 3 harvest source over time.
- Set up a new, additional harvest source record so the agency has an additional DCAT-US 3 harvest source.
- Turn on harvesting for the new harvest source.
- Allow the agency to transition datasets or other metadata records from the DCAT-US 1.1 harvest source to the DCAT-US 3 harvest source.
- Wait for the agency to confirm that all datasets or other metadata records have been transitioned.
- Clear and then delete the old harvest source.
- Send confirmation to the agency point of contact.
- Note any troubleshooting items in the request comments or tracking record.
Path B is used when an existing harvest source will be stopped, updated, and resumed using the DCAT-US 3 metadata format/profile.
This path requires coordination with the agency to confirm timing and ensure the agency has made the necessary changes to its JSON file before harvesting resumes.
- Confirm the date to stop harvesting the old harvest source.
- Make note of harvest source timing as daily, weekly, monthly, or other.
- Edit the harvest source to stop harvesting and send notification to the agency.
- Wait for the agency to switch, modify, or edit its JSON file and send confirmation back that changes have been made.
- Edit the harvest source to reference DCAT-US 3 and resume harvesting as previously noted.
- Send confirmation to the agency point of contact.
- Note any troubleshooting items in the request comments or tracking record.
A harvest source request may be considered complete when the following acceptance criteria have been met:
- A new or updated metadata harvest source exists under the correct parent organization.
- The harvest source uses the confirmed source URL.
- The source is configured for the appropriate metadata format/profile.
- An initial harvest has completed successfully.
- Any harvest errors or validation issues have been reviewed and documented.
- Follow-up actions, if any, have been captured in the request issue or linked issues.
Agencies may need to submit a harvest source change request for several reasons.
An agency has created a DCAT-US 3 compliant metadata file and needs Data.gov to add it as a new harvest source.
An agency needs to update the source URL for an existing harvest source.
An agency is transitioning its metadata from DCAT-US 1.1 to DCAT-US 3 and needs Data.gov to update or add a harvest source.
An agency needs to replace an existing harvest source with a new one.
An agency has completed transition to a new harvest source and needs the old source cleared and deleted.
An agency identifies incorrect harvest source information that needs to be corrected.
Agencies may bring questions to DCAT Interagency Q&A Office Hours if they need help before submitting a harvest source change request.
Examples of office hours topics include:
- Whether a harvest source change request is needed
- How to interpret validation results
- Whether a proposed source should be added as a new source or used to update an existing source
- How metadata decisions may affect harvesting
- How metadata may display or be discovered on Data.gov
- DCAT-US 3 implementation questions
- Transition questions related to DCAT-US 1.1 and DCAT-US 3
Meeting schedule updates are posted at:
https://resources.data.gov/resources/dcat-us-3-updates/
The Harvest Source Change Request Form should not be used for every type of DCAT-US 3 or Data.gov issue.
Use the Inventory.data.gov Access Request Form to request or renew access to Inventory.data.gov:
https://touchpoints.app.cloud.gov/touchpoints/9145dd7e
Once an Inventory.data.gov account is established, agencies must re-submit that same form once per quarter to maintain access.
Use the DCAT-US GitHub Issues page to request or discuss changes to the DCAT-US metadata standard itself:
https://github.com/GSA/dcat-us/issues
Examples include:
- Requesting a change to the DCAT-US metadata standard
- Proposing updates to fields, definitions, usage notes, or requirements
- Identifying ambiguities or gaps in the standard
- Raising standard-level metadata modeling questions
Technical implementation issues, enhancements, and defects for Data.gov products are tracked on the Data.gov Team Board:
https://github.com/orgs/GSA/projects/11/views/1
This includes technical work for products such as:
- Harvester
- Catalog
- Inventory.data.gov
- Other Data.gov services
| Need | Use This Resource |
|---|---|
| Request a new DCAT-US 3 harvest source | Harvest Source Change Request Form |
| Change an existing harvest source | Harvest Source Change Request Form |
| View current harvest sources | harvest.data.gov |
| Validate a proposed or updated harvest source | Harvest Source Validation Tool |
| Request or renew Inventory.data.gov access | Inventory.data.gov Access Request Form |
| Request a change to the DCAT-US metadata standard | DCAT-US GitHub Issues |
| Track agency harvest source request status | Harvest Source Management Board |
| Track Data.gov product defects or enhancements | Data.gov Team Board |
Agencies should submit harvest source change requests using this form:
https://touchpoints.app.cloud.gov/touchpoints/ffb7e9d0
Agencies can view current harvest sources at:
Agencies can validate a proposed or updated harvest source at:
https://harvest.data.gov/validate/
After a request is submitted, the Data.gov team reviews the request, tests the source URL, adds the source to the Harvester development environment, sends a test report to the agency point of contact, and coordinates the appropriate transition path.
Requests are tracked on the Harvest Source Management Board:
https://github.com/orgs/GSA/projects/431
Path A creates a new, additional DCAT-US 3 harvest source while temporarily
- DCAT-US 3 Implementation Resources
- Agency Workflow for DCAT-US 3 Harvest Source Requests
- Inventory.data.gov Access
- Harvest Source Review and Validation
- Harvest Source Change Requests
- FAQ Overview
- DCAT-US 3 Technical Implementation FAQ
- Harvest Source Request FAQ
- Inventory.data.gov Access FAQ
- Meeting FAQ
- Tracking and Issue Routing FAQ
- DCAT Interagency Maintenance Team
- DCAT IMT CHARTER
- Maintenance Process Overview
- Roles and Responsibilities
- Issue Triage and GitHub Workflow
- Semi-Annual Release Cycles
- Agency Feedback and Participation
- Release Communications and Outreach
- Annual Retrospective
- Maintenance Meeting Schedule
- Maintenance Templates