[Framework] Fetch spec info from W3C API and Specref #207
Conversation
Major re-write of the spec info extraction script to fetch info from the W3C API and from Specref. A valid W3C API key is required to query the W3C API. It must be written to a `w3cApiKey` property in a `config.json` file in the root folder or as a `W3C_API_KEY` environment variable. Also took the opportunity to inject all (non implementation related) info defined in data files into the generated `tr.json` file so that loading individual data files from the Web page becomes useless. That simplifies the logic of the JS generation a bit. The code also drops the need to distinguish between the `TR`, `editors` and `ls` URLs of a spec in the data file. A `url` property should be enough if the spec exists in the W3C spec database or in Specref (the code will also consider that the filename is a W3C shortname if that property does not exist to generate the spec's URL). README updated accordingly. The code still supports the former properties. Data files have not yet been updated to use `url` instead of the former properties. Most of the `wgs` and `title` properties can also be dropped now that the code fetches info for Specref. The Editor's Draft URL is now also included in `tr.json`. The UI does not make use of that info for now.
Travis CI removes encrypted environment variables on pull requests from a fork for security reasons: https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions This would effectively prevent checking pull requests before they get merged. Or we'd have to send PR from the same repository, which I'm not a huge fan of. The W3C API allows leaving the W3C API key in plain text, so let's stick to that for now. I'll try to restrict the key to the right set of domain names later on (not exactly sure how to do that for now)
Since the W3C API key appears in plain text in `.travis.yml` (to allow checks to run in pull requests from forks of the repo), it is highly recommended to restrict the API key to a specific set of domain names. The code now sends an `Origin: https://www.w3.org` HTTP header, and the key was restricted to `w3.org`. This should prevent reuse of the API key in contexts that automatically set the `Origin` header (such as in browser contexts).
| * `polyfills`: for specifications for which there are polyfills available that would be worth reporting, the `polyfills` property lists these polyfills. It should be an array of objects that have a `url` property that links to the polyfill's home page on the Web, and a `label` property with the name of polyfill. | ||
| * `feature`: in case the reference to the specification would benefit from being more specific than the specification as a whole, the `feature` property allows to add the name of the specific feature (see e.g. the [reference to the HTMLMediaElement interface in the HTML5 specification](data/htmlmediaelement.json)) | ||
| * `title`: when the specification is unknown to the [W3C API](https://w3c.github.io/w3c-api/) and to [Specref](https://www.specref.org/), the `title` property should be set to the title of the specification | ||
| * `edDraft`: when the specification is unknown to the [W3C API](https://w3c.github.io/w3c-api/) and to [Specref](https://www.specref.org/), or when these APIs do not know the URL of the Editor's Draft for the specification, the `edDraft` property should contain the URL of the Editor's Draft of the specification. |
There was a problem hiding this comment.
Because I'd like to be able to use editors later on to list the editors of a specification, as done in ReSpec (actually, I hesitated to name the property edDraftURI as in ReSpec, but URI seemed a bit pompous).
There was a problem hiding this comment.
edDraft sounds like... editors' draft Draft to me. It depends on how you interpret the lowercase letter "d". But anyway, it doesn't matter that much and I think I can live with this ^_^
There was a problem hiding this comment.
Would editorDraftUrl or editorDraft look better for you? I don't really like edDraft either, I just wanted to remain close to what Respec uses. But then, since I already dropped URI...
There was a problem hiding this comment.
editorsDraft looks better to me, but maybe we can apply the principle of least astonishment for ReSpec users here. (I didn't think of the ReSpec issue.)
| * `feature`: in case the reference to the specification would benefit from being more specific than the specification as a whole, the `feature` property allows to add the name of the specific feature (see e.g. the [reference to the HTMLMediaElement interface in the HTML5 specification](data/htmlmediaelement.json)) | ||
| * `title`: when the specification is unknown to the [W3C API](https://w3c.github.io/w3c-api/) and to [Specref](https://www.specref.org/), the `title` property should be set to the title of the specification | ||
| * `edDraft`: when the specification is unknown to the [W3C API](https://w3c.github.io/w3c-api/) and to [Specref](https://www.specref.org/), or when these APIs do not know the URL of the Editor's Draft for the specification, the `edDraft` property should contain the URL of the Editor's Draft of the specification. | ||
| * `wgs`: when the specification is unknown to the [W3C API](https://w3c.github.io/w3c-api/) and to [Specref](https://www.specref.org/), the `wgs` property should be an array of objects describing the groups that are producing the spec; each such object should have a `url` property with a link to the group's home page, and a `label` property with the name of the group. |
There was a problem hiding this comment.
IIRC Specref doesn't have data about working groups?
There was a problem hiding this comment.
Specref has some notion of working groups, see e.g. info about Web-bluetooth, but that's not fantastic: it gives the URL of the group's home page which is good, but then it only gives a shortname for the group, which does not mean much as far as I can tell.
I was planning to build on that but then forgot about it, so thanks for the reminder ;) Mixing the info in the deliveredBy property with the info in the publisher property should work most of the time. Also, using the publisher property directly when there is no deliveredBy property should work as well. I'll update the logic in a follow-up PR.
README.md
Outdated
| @@ -231,8 +231,10 @@ The `gh-pages` branch is the branch published on [`https://w3c.github.io/web-roa | |||
|
|
|||
| If you would like to visualize the contents of a roadmap locally as it would appear on the published version, you will need to: | |||
|
|
|||
| 1. Run the [Makefile](Makefile) to update information and implementation data. This should generate `specs/tr.json` and `specs/impl.json` files. Note you'll need Node.js v8.0.0 or above and you'll need to run `npm install` first. | |||
| 2. Serve the root folder of the repository over HTTP (any HTTP server should work). In particular, opening the file directly with your Web browser will not work because the JavaScript code needs to send cross origin requests, which are not supported for `file://` URLs. | |||
| 1. Create a [W3C account](https://www.w3.org/accounts/request) and a [W3C API key](https://www.w3.org/users/41989/apikeys) if not already done | |||
There was a problem hiding this comment.
Change the API key URL to https://www.w3.org/users/myprofile/apikeys ?
There was a problem hiding this comment.
Dammit, I hadn't noticed that URL contained my W3C user ID ;)
|
Made a few comments on the docs. The code looks good to me. Thank you! |
Addresses #130, #105, and partly #77.
Major re-write of the spec info extraction script to fetch info from the W3C API and from Specref. A valid W3C API key is required to query the W3C API. It must be written to a
w3cApiKeyproperty in aconfig.jsonfile in the root folder or as aW3C_API_KEYenvironment variable.Also took the opportunity to inject all (non implementation related) info defined in data files into the generated
tr.jsonfile so that loading individual data files from the Web page becomes useless. That simplifies the logic of the JS generation a bit.The code also drops the need to distinguish between the
TR,editorsandlsURLs of a spec in the data file. Aurlproperty should be enough if the spec exists in the W3C spec database or in Specref (the code will also consider that the filename is a W3C shortname if that property does not exist to generate the spec's URL). README updated accordingly. The code still supports the former properties.Data files have not yet been updated to use
urlinstead of the former properties. Most of thewgsandtitleproperties can also be dropped now that the code fetches info for Specref.The Editor's Draft URL is now also included in
tr.json. The UI does not make use of that info for now.