Test URL and data

[MstrDan]

1. Can a generic prod/live frontend version of the url be available and point to appropriate environment, Prod or Test. So we can code our request to it without having to change code later. If not ready, when will it be and what is the most recent version, v0? Assuming any major changes to this URL and versions will be communicated directly as we've pretty much completed development and waiting for final stable version.
2. In vw_general endpoint, is there only audit year 2022 available? When will other audit years be available for testing?
   https://api.data.gov/TEST/audit-clearinghouse/v0/dissemination/vw_general?&api_key=private&audit_year=eq.2021&limit=1000&offset=0
   no data returned.

Dan Amare
AmeriCorps

[mogul]

Hi Dan,

Addressing only the first point:

Can a generic prod/live frontend version of the url be available and point to appropriate environment, Prod or Test. So we can code our request to it without having to change code later. If not ready, when will it be and what is the most recent version, v0? Assuming any major changes to this URL and versions will be communicated directly as we've pretty much completed development and waiting for final stable version.

We've been talking about the API versioning lifecycle and want to run this by you and get some feedback...

1. Think of what we've got as v0 of the API.
2. We'll freeze this as v1 of the API.
3. We'll start making v2-alpha, v2-beta etc. available on an opt-in basis for volunteers as we make changes in the future. Volunteers would opt in to using those newer API versions by supplying the Accept-Profile: and Content-Profile: headers, as described here.
4. We would announce v2 as non-default, opt-in version, along with much fanfare and a timeline for when it will become the default. During that period, people can opt-in to using v2, again by supplying those headers in their requests, in order to verify that their stuff will continue to work once v2 becomes the default.
5. After the v2 timeline ends, v2 will be the default unless people explicitly "opt-out" by specifying that they still want to want to use v1 in their headers.

From then on, the process looks much the same to move from v2->v3, etc. Previous versions would remain available for as long as technology allows (eg so long as we can map the frozen API views to newer versions of the underlying tables and have them still make sense).

How does that sound?

Bret
GSA/TTS/OROS/FAC rabblerouser

[MstrDan]

Thanks, Bret. This makes sense and shouldn't be a problem. We just need to be able to use newer versions with minimal changes to our code. With that, will the url path change to v0->v1->v2 when newer version becomes the default? So once you're on v2, we'll need to update code to that effect. Project priorities change so resource may not be available to volunteer test versions but if we know ahead of time when the new version becomes the default we'll plan accordingly, and test as needed. More importantly, any major changes to underlying views and models obviously impacts our code so probably should be avoided as you might agree. I understand what you currently have is pretty stable version. We've had to deal with sudden model changes, fields dropped etc and we scramble to figure out what changed and fix it. But the process you outlined should avoid this.
Thanks for the detail explanation!
Dan

[mogul]

With that, will the url path change to v0->v1->v2 when newer version becomes the default? So once you're on v2, we'll need to update code to that effect.

You can optionally indicate exactly which version you want to use via the Accept-Profile: and Content-Profile: headers. So the code change wouldn't be in the URL path, but in the headers included with your requests.

More importantly, any major changes to underlying views and models obviously impacts our code so probably should be avoided as you might agree. I understand what you currently have is pretty stable version. We've had to deal with sudden model changes, fields dropped etc and we scramble to figure out what changed and fix it.

Sorry about that, we're still moving pretty fast... Yes, versioning the API will address that problem. You would no longer be using an API driven entirely from the shape of the underlying tables; freezing an API version will involve us creating views which are a snapshot of the underlying table structure at that time. Those views won't change shape (in terms of the resulting columns) even if the underlying tables change to support newer versions of the API.

We haven't discussed this much yet, but there's also potential for us to name API versions using semver conventions. That would give you an indication of the work required to update to newer API versions, since breaking changes would be clearly communicated. Let us know what you think.

[MstrDan]

Let me see if I understand this correctly.
The base url would be generic one, not the current API_GOV_URL="https://api.data.gov/TEST/audit-clearinghouse/v0/dissemination", which has the hardcoded V0 in it.
With this generic url we can specify in the header the version of the API to use with header = { 'Accept-profile': 'api'}?
It works now with the current url and accept-profile = 'api' setting. I'm just trying to understand how it'll be when you go live. So of my questions I'm sure have to do with basic GET request terminology I'm relatively unfamiliar with. Thanks.

[mogul]

The base url would be generic one, not the current API_GOV_URL="https://api.data.gov/TEST/audit-clearinghouse/v0/dissemination", which has the hardcoded V0 in it.

Correct. It will likely be something like https://api.fac.gov/dissemination

With this generic url we can specify in the header the version of the API to use with header = { 'Accept-profile': 'api'}?

Yes, although instead of api it will be either a specific version (such as v1, v2-beta, etc) or no header (which means "just use the latest/default version").

So of my questions I'm sure have to do with basic GET request terminology I'm relatively unfamiliar with.

For GET and HEAD requests, you specify the Accept-Profile: header.

For all other requests, you specify the Content-Profile: header.

[MstrDan]

Great. Thanks for confirming, Bret!

[jadudm]

Hi Dan,

Bret covered your first question very nicely. Thank you, Bret!

To the second: we're still loading data and working out process in the transfer with Census.

We do not yet have all of the data (e.g. PDFs), and we're currently patching some import errors that will trigger a reload of '22. Once that passes regression tests, we'll proceed backwards through time loading '21 and '20.

Our intent is to have a 5-year lookback loaded by or shortly after go-live.

[MstrDan]

That works. Please update us once other audit years are available. Thanks!