-
Notifications
You must be signed in to change notification settings - Fork 5
Agency Workflow for Harvest Source Requests
This page describes the recommended agency workflow for preparing, validating, and submitting a request for a new DCAT-US 3 harvest source or a change to an existing harvest source.
Agencies should use this workflow to support DCAT-US 3 implementation, reduce validation issues, and help the Data.gov team process harvest source requests efficiently.
| Step | Action | Resource |
|---|---|---|
| 1 | Review DCAT-US 3 implementation resources | DCAT-US 3 Implementation Guide |
| 2 | Review the DCAT-US 3 to M-25-05 Crosswalk, if applicable | DCAT-US 3 to M-25-05 Crosswalk |
| 3 | Check technical implementation FAQs | DCAT-US GitHub Wiki |
| 4 | Determine whether Inventory.data.gov access is needed | Inventory.data.gov Access Request Form |
| 5 | Create or update the DCAT-US 3 compliant file | Agency-selected metadata process or Inventory.data.gov |
| 6 | Review current harvest sources | harvest.data.gov |
| 7 | Validate the proposed or updated harvest source | Harvest Source Validation Tool |
| 8 | Submit a harvest source change request | Harvest Source Change Request Form |
| 9 | Coordinate with the Data.gov team during review | Agency point of contact and Data.gov team |
| 10 | Respond to follow-up questions or troubleshooting items | Agency point of contact and Data.gov team |
Agencies should begin by reviewing the primary DCAT-US 3 implementation resources.
Resource:
https://resources.data.gov/resources/dcat-us-3-implementation/
The DCAT-US 3 Implementation Guide is the primary reference for agencies implementing the DCAT-US 3 metadata standard.
Agencies should use this guide to:
- Understand DCAT-US 3 requirements
- Review metadata field expectations
- Prepare DCAT-US 3 compliant metadata files
- Align agency data inventories and catalogs with the DCAT-US 3 standard
- Support metadata quality
- Prepare for validation
- Prepare for Data.gov harvesting
- Understand how metadata may display or be discovered on Data.gov
Agencies should consult the implementation guide before submitting harvest source change requests or bringing implementation questions to interagency meetings.
Resource:
https://resources.data.gov/resources/dcat-us-3-crosswalk/
The DCAT-US 3 to M-25-05 Crosswalk provides a reference for understanding how DCAT-US 3 metadata elements relate to M-25-05-related requirements or considerations.
Agencies may use this crosswalk to support:
- Implementation planning
- Metadata review
- Internal coordination
- Alignment between agency metadata practices and related federal expectations
- Discussions between data, policy, technical, and governance teams
Agencies should use the crosswalk alongside the DCAT-US 3 Implementation Guide.
Resource:
https://github.com/GSA/dcat-us/wiki/DCAT-US-3-Technical-Implementation-FAQ
Frequently asked questions related to technical implementation of the DCAT-US 3 metadata standard are being posted to the DCAT-US GitHub wiki as they are developed.
Agencies should review this resource for:
- Common DCAT-US 3 implementation questions
- Technical interpretation of metadata fields
- Validation issues and troubleshooting
- Metadata formatting questions
- Harvesting-related implementation considerations
- Clarifications raised through interagency discussions
- Frequently encountered edge cases or implementation patterns
If an agency has a question that is not addressed in the implementation guide, crosswalk, or FAQs, the agency may bring the question to DCAT Interagency Q&A Office Hours.
Resource:
https://touchpoints.app.cloud.gov/touchpoints/9145dd7e
Agencies that need access to Inventory.data.gov to create a DCAT-US 3 compliant file can request access by submitting the Inventory.data.gov Access Request Form.
Inventory.data.gov may be used by agencies as part of their metadata creation and management process, including the creation of files intended to comply with the DCAT-US 3 metadata standard.
Agencies may use this form to:
- Request initial access to Inventory.data.gov
- Support creation of a DCAT-US 3 compliant file
- Maintain continued access to Inventory.data.gov
- Complete the required quarterly access renewal process
Once an Inventory.data.gov account is established, agencies must re-submit the same form once per quarter to maintain access.
Inventory.data.gov Access Request Form:
https://touchpoints.app.cloud.gov/touchpoints/9145dd7e
Agencies should create or update their metadata file using their selected metadata management process or tool.
This may include:
- Inventory.data.gov
- An agency-managed metadata system
- An agency data inventory platform
- A catalog or publishing system
- Another internal process used to produce DCAT-US 3 compliant metadata
The metadata file should be prepared in accordance with the DCAT-US 3 Implementation Guide.
Agencies should confirm that the file:
- Uses the appropriate DCAT-US 3 structure
- Includes required metadata fields
- Includes recommended fields where applicable
- Uses valid values and formats
- Reflects current agency dataset and distribution information
- Is available at the source URL that will be submitted for harvesting
Resource:
https://harvest.data.gov
Agencies can view their current harvest sources at harvest.data.gov.
Agencies should review current harvest sources before requesting a new DCAT-US 3 harvest source or submitting a change to an existing harvest source.
Reviewing current harvest sources can help agencies determine whether they need to:
- Request a new DCAT-US 3 harvest source
- Update an existing harvest source
- Correct harvest source information
- Confirm the status or configuration of existing harvest sources
- Identify whether a current source should be replaced, updated, or retired
- Determine whether they are transitioning from DCAT-US 1.1 to DCAT-US 3
Resource:
https://harvest.data.gov/validate/
Agencies can use the Harvest Source Validation Tool to check whether their proposed or updated harvest source is expected to pass validation.
Agencies should validate their harvest source before submitting a request for a new DCAT-US 3 harvest source or a change to an existing harvest source.
Using the validation tool can help agencies:
- Test whether a proposed harvest source passes validation
- Identify metadata errors or warnings
- Confirm that a DCAT-US 3 harvest source is structured correctly
- Resolve validation issues before requesting that Data.gov add or update the harvest source
- Reduce back-and-forth during the harvest source review process
If the harvest source does not pass validation, agencies should review and correct the identified issues before submitting a new or updated harvest source request, where possible.
Resource:
https://touchpoints.app.cloud.gov/touchpoints/ffb7e9d0
The Harvest Source Change Request Form is the agency-facing intake mechanism for requesting either a new DCAT-US 3 harvest source or a change to an existing harvest source.
Agencies should use this form when they need to:
- Request creation of a new DCAT-US 3 harvest source
- Request a change to an existing harvest source
- Update the source URL for a harvest source
- Change a harvest source from DCAT-US 1.1 to DCAT-US 3
- Correct or update harvest source configuration information
- Request other harvest-source-related changes needed to support metadata harvesting by Data.gov
Before submitting the Harvest Source Change Request Form, agencies should:
- Review the DCAT-US 3 Implementation Guide
- Review their current harvest sources at https://harvest.data.gov
- Validate the proposed or updated source at https://harvest.data.gov/validate/
- Confirm the correct source URL
- Identify the agency point of contact for follow-up
- Confirm whether the request is for a new harvest source or a change to an existing harvest source
- Note whether the request relates to a transition from DCAT-US 1.1 to DCAT-US 3
After a harvest source change request is submitted, the Data.gov team will review and process the request.
Requests submitted through the Harvest Source Change Request Form are tracked on the Harvest Source Management Board:
https://github.com/orgs/GSA/projects/431
The Harvest Source Management Board is used to track:
- Request details
- Agency communications
- Notes received from agencies
- Internal notes
- Follow-up questions
- Troubleshooting items
- Current stage or status of each request
- Coordination needed with agency representatives or Data.gov team members
The Data.gov team may contact the agency point of contact for clarification, testing results, timing, or follow-up information.
Agencies should be prepared to respond to follow-up questions from the Data.gov team during the review and implementation process.
Follow-up may include requests to:
- Confirm the source URL
- Review the Harvester development environment test report
- Correct validation errors
- Clarify metadata structure or field usage
- Confirm whether the request should follow Path A or Path B
- Confirm timing for stopping or resuming harvesting
- Confirm that agency-side changes have been made
- Confirm that datasets or metadata records have been transitioned
- Review harvest errors or warnings
- Confirm final acceptance of the new or updated harvest source
After initial testing, the Data.gov team and agency point of contact will determine whether the request should follow Path A or Path B.
Path A is used when an agency will have a new, additional DCAT-US 3 harvest source created while the existing harvest source remains available temporarily.
This allows the agency to transition datasets or other metadata records from the existing DCAT-US 1.1 harvest source to the new DCAT-US 3 harvest source over time.
Under Path A:
- A new DCAT-US 3 harvest source is created.
- The existing harvest source remains available temporarily.
- The agency transitions datasets or metadata records from the old source to the new source.
- The agency confirms when the transition is complete.
- The old harvest source is cleared and deleted.
- Confirmation is sent to the agency point of contact.
Path A may be appropriate when an agency needs time to transition records gradually from an existing DCAT-US 1.1 harvest source to a new DCAT-US 3 harvest source.
Path B is used when an existing harvest source will be stopped, updated, and resumed using the DCAT-US 3 metadata format/profile.
This path requires coordination with the agency to confirm timing and ensure the agency has made the necessary changes to its JSON file before harvesting resumes.
Under Path B:
- The Data.gov team confirms the date to stop harvesting the old harvest source.
- The current harvest schedule is noted.
- The harvest source is edited to stop harvesting.
- The agency switches, modifies, or edits its JSON file.
- The agency confirms that changes have been made.
- The Data.gov team edits the harvest source to reference DCAT-US 3.
- Harvesting resumes according to the previously noted schedule.
- Confirmation is sent to the agency point of contact.
Path B may be appropriate when an agency is updating an existing harvest source in place rather than creating a new additional source.
Before submitting a harvest source change request, agencies should complete the following checklist.
- Reviewed the DCAT-US 3 Implementation Guide.
- Reviewed the DCAT-US 3 to M-25-05 Crosswalk, if applicable.
- Checked the DCAT-US 3 Technical Implementation FAQs.
- Determined whether Inventory.data.gov access is needed.
- Requested or renewed Inventory.data.gov access, if needed.
- Created or updated the DCAT-US 3 compliant metadata file.
- Reviewed current harvest sources at https://harvest.data.gov.
- Validated the proposed or updated harvest source at https://harvest.data.gov/validate/.
- Reviewed and addressed validation errors or warnings, where possible.
- Confirmed the source URL.
- Identified the agency point of contact for follow-up.
- Determined whether the request is for a new harvest source or a change to an existing harvest source.
- Submitted the Harvest Source Change Request Form at https://touchpoints.app.cloud.gov/touchpoints/ffb7e9d0.
Agencies may bring questions to DCAT Interagency Q&A Office Hours if they need help with:
- Interpreting DCAT-US 3 fields
- Applying DCAT-US 3 to agency-specific datasets or catalogs
- Understanding validation errors or warnings
- Determining whether a harvest source change request is needed
- Understanding how metadata decisions may affect Data.gov harvesting
- Understanding how metadata may display, search, filter, or be discovered on Data.gov
- Discussing transition options from DCAT-US 1.1 to DCAT-US 3
- Raising related metadata standards questions
Meeting schedule updates are posted at:
https://resources.data.gov/resources/dcat-us-3-updates/
Resource:
https://github.com/GSA/dcat-us/issues
Use the DCAT-US GitHub Issues page when the question or request is related to the DCAT-US metadata standard itself.
Examples include:
- Requesting a change to the DCAT-US metadata standard
- Proposing updates to DCAT-US fields, definitions, usage notes, or requirements
- Raising questions about the structure or interpretation of the standard
- Identifying inconsistencies, ambiguities, or gaps in the standard
- Discussing standard-level metadata modeling considerations
- Suggesting improvements to DCAT-US documentation or guidance
Do not use DCAT-US GitHub Issues for agency-specific harvest source requests. Those should be submitted through the Harvest Source Change Request Form:
https://touchpoints.app.cloud.gov/touchpoints/ffb7e9d0
https://resources.data.gov/resources/dcat-us-3-updates/
Use this page for meeting schedule updates and related DCAT-US 3 engagement information.
https://resources.data.gov/resources/dcat-us-3-implementation/
Primary reference for agencies implementing the DCAT-US 3 metadata standard.
https://resources.data.gov/resources/dcat-us-3-crosswalk/
Reference for understanding how DCAT-US 3 metadata elements relate to M-25-05-related considerations.
https://github.com/GSA/dcat-us/wiki/DCAT-US-3-Technical-Implementation-FAQ
Frequently asked questions related to technical implementation of the DCAT-US 3 metadata standard are posted here as they are developed.
https://touchpoints.app.cloud.gov/touchpoints/9145dd7e
Use this form to request or renew access to Inventory.data.gov. Once an account is established, this same form must be re-submitted once per quarter to maintain access.
Use this site to view current agency harvest sources.
https://harvest.data.gov/validate/
Use this tool to test whether a proposed or updated harvest source is expected to pass validation.
https://touchpoints.app.cloud.gov/touchpoints/ffb7e9d0
Use this form to request a new DCAT-US 3 harvest source or a change to an existing harvest source.
https://github.com/orgs/GSA/projects/431
Used by the Data.gov team to track harvest source requests received through the Harvest Source Change Request Form, including agency communications, notes, request details, troubleshooting items, and the current stage or status of each request.
https://github.com/orgs/GSA/projects/11/views/1
Used to track technical implementation issues, enhancements, and defects across Data.gov products such as Harvester, Catalog, Inventory.data.gov, and related services.
https://github.com/GSA/dcat-us/issues
Used to request or discuss changes to the DCAT-US metadata standard itself.
- DCAT-US 3 Implementation Resources
- Agency Workflow for DCAT-US 3 Harvest Source Requests
- Inventory.data.gov Access
- Harvest Source Review and Validation
- Harvest Source Change Requests
- FAQ Overview
- DCAT-US 3 Technical Implementation FAQ
- Harvest Source Request FAQ
- Inventory.data.gov Access FAQ
- Meeting FAQ
- Tracking and Issue Routing FAQ
- DCAT Interagency Maintenance Team
- DCAT IMT CHARTER
- Maintenance Process Overview
- Roles and Responsibilities
- Issue Triage and GitHub Workflow
- Semi-Annual Release Cycles
- Agency Feedback and Participation
- Release Communications and Outreach
- Annual Retrospective
- Maintenance Meeting Schedule
- Maintenance Templates