Extensions: Consider normative content about how extensions are published

[jpmckinney]

We presently assume that extensions are published in a web-accessible directory (or API that behaves like a directory), such that extension.json can be trimmed from the end of a URL and replaced with a schema or codelist file.

Some publishers publish via APIs (e.g. Visual Studio) where the file path is a query string parameter.

The available documentation on creating extensions is:

• https://standard.open-contracting.org/latest/en/guidance/map/extensions/
• https://github.com/open-contracting/standard_extension_template
• https://extensions.open-contracting.org/en/publishers/

Options:

1. Add normative content establishing our assumption as the rule.
2. Change the publication format to be a ZIP file (GitHub, GitLab, Visual Studio, etc. auto-generate a ZIP). The extensions array would also have to be links to ZIP files.
3. Change the schemas and codelists keys in extension.json) to be absolute URLs and/or relative URLs.
4. Relax our assumption so that just replacing extension.json in the URL with the desired path should work.

Considerations:

1. This will be fine for the vast majority of publishers, but might frustrate the 1-2 exceptions.
2. This will require a change to all publications that use extensions. Otherwise, it should be fine for all publishers, as all known extensions (as far as I remember) are published on platforms that provide ZIP files automatically.
3. This will require a change to all publications that use extensions. This also has its own complexities, e.g. resolving relative URLs, testing extensions (absolute URLs are likely to change between development and production), etc.
4. This should be fine for all publishers. A challenge is that the rule might be hard to word. We can maybe word it the same as (1), and then add the substitution case as an alternative.

[jpmckinney]

Closed #1291 as duplicate. (Please read in case anything additional to cover.)

#606 is also related and can maybe be closed at the same time.

[odscjen]

Considering all of those options:

1. Checking through the publishers in the Data Registry who aren't marked as no longer publishing, 42/56 use OCP maintained extensions, and 25/56 use local extensions. Of the local extensions that are linked to from the Data Registry pages all but one are hosted on Github, Gitlab or Bitbucket and so meet the need that

extension.json can be trimmed from the end of a URL and replaced with a schema or codelist file.

2. Given how many publishers use extensions (local or OCP maintained, see above) I don't think this is a desirable option.

3. as for 2

4. The only one of the local extensions linked to in the Data Registry that was somewhere else is used (and maintained) by Chile Compra and is in Visual Studio with an API query string style path. But even then because there are no codelists to get from one file to another it is still just a case of replacing the file name in the url, it's just the url includes a ?path= token before the file name. e.g. https://chilecompracl.visualstudio.com/Documentaci%C3%B3n%20OCDS/_git/ocds_contractDateCreated_extension?path=/extension.json (as you mention in the initial issue description)

My feeling is that it shouldn't be too difficult to word it sufficiently, going for (1) and establishing the assumption as the normative rule but adding the substitution rule as a permissible alternative seems the way to go.

[jpmckinney]

Sounds good.

We already implemented (4) in the profile builder, so the permissible alternative has no additional development cost.

Noting that a simpler version of (3) would have been to add support for full URLs. The main downside is the development time to update libraries to handle this scenario (not difficult for us, but not obvious for others).

[odscjen]

Noting that all 3 location links in the initial issue description that contain the extension guidance are currently explicitly named as non-normative pages in https://standard.open-contracting.org/staging/1.2-dev/en/governance/normative/#normative-content.

• https://standard.open-contracting.org/latest/en/guidance/map/extensions/ - is very minimal on creating extensions and mostly just points to the github extension template
• https://github.com/open-contracting/standard_extension_template - contains all the technical details needed to create an extension
• https://extensions.open-contracting.org/en/publishers/ - extensive discursive content on the steps needed to create an extension.

So we should explicitly declare either https://extensions.open-contracting.org/en/publishers/ or https://github.com/open-contracting/standard_extension_template/blob/master/README.md as normative as this is where the new guidance will be added. My feeling is https://extensions.open-contracting.org/en/publishers/ should be the normative guidance, and the bit added to https://github.com/open-contracting/standard_extension_template/blob/master/README.md can be the non-normative bit with the permissible alternative described?

[odscjen]

@jpmckinney First stab at the wording where in both instances I've put the new words in italics and used Hemingway to try and remove passive voice and sentences it thought were "very hard to read".

In https://extensions.open-contracting.org/en/publishers/#document where this will be normative content

"You will need to find somewhere to host your extension online at a stable URL. You must publish your extension in a public web-accessible directory (or API that behaves like a directory). Using this kind of location means that the URL path of each file ends in the file name, but is otherwise identical . For example, replacing extension.json at the end of the URL for that file with release-schema.json would take you to the schema file. Creating a GitHub repository for your extension is a good option. Github repositories provide the necessary directory while offering version control and issue tracking.
"

And in https://github.com/open-contracting/standard_extension_template/blob/master/README.md using non-normative version of the above

"† Use of git for version control is recommended, but optional. You need to host your extension on a public HTTP server, with a file structure that matches that in this template repository. You need to publish your extension in a web-accessible directory (or API that behaves like a directory). Using this kind of location means that the URL path of each file ends in the file name, but is otherwise identical . For example, replacing extension.json at the end of the URL for that file with release-schema.json would take you to the schema file. If you cannot use such a location, it is permissible to use a location that allows for replacing extension.json at the end of the URL with the path to the other files in the extension."

@duncandewhurst feel free to weigh in on wording and location too :)

[duncandewhurst]

Regarding the location of the normative statement, I would put it in https://standard.open-contracting.org/staging/1.2-dev/en/schema/conformance_and_extensions/#extensions. I took a pass at wording the actual rule:

It must be possible to access an extension's schema and codelist files by replacing extension.json in the extension's URL with a file's path, e.g. release-schema.json or codelists/codelist.csv. For more information, refer to the Extension Explorer's guidance on documenting extensions.

The non-normative documentation can provide guidance on the practical options for conforming to the rule.

Rather than documenting the guidance in two places, I would document it in https://extensions.open-contracting.org/en/publishers/#document and then update the note in the extension template to refer to that guidance, e.g. For guidance on how to publish your extensions, refer to...

[jpmckinney]

@duncandewhurst's proposed wording looks good.

The two open issues on the template repo are essentially about eliminating its readme, so we can also pursue that option as part of (or as follow-up to) this issue: https://github.com/open-contracting/standard_extension_template/issues

Normative content can be added to https://standard.open-contracting.org/staging/1.2-dev/en/schema/conformance_and_extensions/#extensions as suggested, and it can be repeated at https://extensions.open-contracting.org/en/publishers/#document

[odscjen]

Great thanks both, agree with @duncandewhurst's wording and where to put it.

I think it makes sense to partly tackle the template repo issues as part of this, so

• a PR to add the new guidance to https://standard.open-contracting.org/staging/1.2-dev/en/schema/conformance_and_extensions/#extensions,
• a second PR to add the new guidance to https://extensions.open-contracting.org/en/publishers/#document and copy over any necessary guidance from the template README
• a third PR to update the template to just point to the extension explorer guidance and remove the bulk of the README

[odscjen]

Actually I now think we're better treating this as two separate issues. So for this issue just:

• add the new guidance to https://standard.open-contracting.org/staging/1.2-dev/en/schema/conformance_and_extensions/#extensions see PR conformance_and_extensions.md: add normative guidance about extension… #1682
• add the new guidance to https://extensions.open-contracting.org/en/publishers/#document see PR publishers.html: add guidance on URL directory paths extension-explorer#82

We can then close this issue and tackle the issue of moving the template README content over to the extension explorer and adding more to the README on what makes a good README

[jpmckinney]

I agree with your approach!

[jpmckinney]

I added the template's two issues to the OCDS 1.2 project.

[odscjen]

closing this as remaining issues are detailed in open-contracting/standard_extension_template#41 and open-contracting/standard_extension_template#37