Skip to content

Harvester APIs

Jump to bottom
David Aguiar edited this page Aug 14, 2026 · 2 revisions

Three APIs provide programmatic access to Data.gov's harvest metadata, run by two different teams for two different audiences. This page describes each one and the current direction for future integrations.

flowchart TD
    subgraph datagovteam["data.gov team"]
        HA["Harvest API<br/>harvest.data.gov/api/v1<br/><i>source of truth</i>"]
    end

    subgraph apidatagovteam["api-data-gov team"]
        GSA["api.gsa.gov Harvester API<br/>/technology/datagov_harvest/v2<br/><i>public front door</i>"]
    end

    subgraph shared["shared infra, ~25 agencies"]
        ADG["api.data.gov<br/>key issuance + gateway (API Umbrella)"]
    end

    Public(["Public / other agencies"]) -->|x-api-key| ADG
    ADG --> GSA
    GSA -.->|"reads from (exact mechanism TBD)"| HA
    Internal(["Data.gov admin app, internal tools"]) --> HA
Loading

Dotted line = relationship not yet confirmed with the api-data-gov team (see "Open questions" below).

Harvest API (harvest.data.gov)

This is the harvester's own API, built into the datagov-harvester admin app and managed by the data.gov team. This is the source of truth and every other API on this page ultimately reads this.

Warning

Hitting /api/<path> without a version prefix (e.g. /api/harvest_sources/ instead of /api/v1/harvest_sources/) redirects to whichever version is currently newest (app/routes.py, API_VERSIONS). Today that redirect goes to v1 but when a v2 ships anyone depending on the unprefixed path gets moved onto it. Integrations should always pin to an explicit version (/api/v1/...).

Source: app/api/ in the datagov-harvester repo. See harvest.data.gov for the pipeline this API sits in front of.

Versioning policy

The API stays on v1 by default. A version bump only happens when a specific endpoint needs a breaking change, and only that endpoint moves to the new version; unaffected endpoints stay on v1. The API is not bumped wholesale for a change that only touches one endpoint.

Breaking, requires a new version:

  • Removing or renaming a field or endpoint
  • Changing a field's type or meaning
  • Changing required parameters or the shape of a response
  • Tightening validation on input that previously succeeded

Non-breaking, no version bump needed:

  • Adding a new endpoint
  • Adding a new optional field
  • Adding a new enum value that existing clients can ignore

This split follows the same breaking/non-breaking definition used in Stripe's API versioning guide, among other public API versioning practices.

Process for moving one endpoint to a new version:

  1. Add the changed endpoint under /api/v2/<endpoint>; leave /api/v1/<endpoint> unchanged and serving the old behavior.
  2. Write a new, separate test suite for the v2 endpoint. v1 and v2 tests are not shared, each version is tested against its own contract, since the two versions' behavior differs by definition.
  3. Leave every other endpoint registered only under v1.
  4. Update this page and the endpoint's Swagger docs to note which version(s) it's on.

Note: today's version mechanism (API_VERSIONS in app/routes.py) mounts the entire api blueprint per version, so it versions the whole API, not individual endpoints. Supporting one endpoint on v2 while the rest stay on v1 will require that mechanism to change to per-route overrides rather than whole-blueprint mounting.

api.gsa.gov (Harvester API)

This is a separate API for the same underlying harvest data, hosted at api.gsa.gov and managed by the api-data-gov team, not the data.gov team. It's meant to be the public front door: the intended entry point for the general public and other agencies, instead of harvest.data.gov directly.

  • Docs: resources.data.gov/harvester-api
  • Base URL: https://api.gsa.gov/technology/datagov_harvest/v2/
  • Auth: a key registered at open.gsa.gov/api/datadotgov, sent in the x-api-key header on every request
  • Covers: harvest sources, jobs, organizations, and records (including raw/transformed payloads), plus DCAT-US catalog validation (1.1 only today; a 3.0 validator is documented as planned but not live)

Open questions, not yet chased down:

  • Why this is versioned v2 while the Harvest API above is only at v1. Unclear whether that's a coincidence or whether this wraps a different, newer internal interface.
  • Exactly how (and how often) this pulls from or proxies harvest.data.gov's own API. Not confirmed with the api-data-gov team.

api.data.gov

This isn't a harvest-specific API. It's GSA's shared API key and gateway platform (built on API Umbrella), used by about 25 federal agencies across 450+ APIs, per api.data.gov. Its job is issuing a single x-api-key and enforcing shared rate limits. The actual data still lives on each agency's own domain.

The api.gsa.gov Harvester API appears to sit behind this shared gateway, meaning an api.data.gov key provides access to api.gsa.gov/technology/datagov_harvest/v2/... rather than a separate credential. This relationship is not yet confirmed with the api-data-gov team and needs verification.

Future direction

Going forward, external and public consumers should be steered toward api.gsa.gov, since that's the public-facing surface (harvest.data.gov's API was built for the admin app itself). Longer-term, the harvest.data.gov API endpoints may be locked down, restricted to internal or authenticated use, once api.gsa.gov is the established front door.

Related pages

Written 2026-08-14 from published docs (resources.data.gov/harvester-api, api.data.gov) and the datagov-harvester codebase (PR #845, "Route /api to the latest API version"). The api.gsa.gov and api.data.gov sections reflect an outside understanding of those systems, not yet confirmed with the api-data-gov team. Update as more is learned.

Clone this wiki locally