Bug 1507521 - Finish library per RFC 128

.taskcluster.yml

@@ -11,7 +11,7 @@ tasks:
           - push
     payload:
       maxRunTime: 3600
-      image: node:8
+      image: node:10
       command:
         - /bin/bash
         - '--login'

README.md

@@ -4,8 +4,8 @@ including manifests, API references, exchange references, and JSON schemas.
 It exports a class, `References`, that manages reading and writing this data in
 several formats, as well as performing transformations and consistency checks.
 
-It also serves as the canonical repository for a few miscellaneous and
-historical schemas.
+It also serves as the canonical repository for a few shared schemas and
+meta-schemas under `schemas/`.
 
 # Data Formats
 
@@ -32,17 +32,12 @@ during the cluster build process:
 This is a subset of the
 [taskcluster-lib-docs](https://github.com/taskcluster/taskcluster-lib-docs)
 documentation tarball format, and `metadata.json` is defined there.  The
-library will load `exchanges.json` as an alternative to `events.json`, if
-present.
+library will in fact load any `.json` files in `references` and interpret them
+according to their schema.  So, the `$schema` property of every file in
+`references` must point to the relevant schema using a `/`-relative URI such as
+`/schemas/common/api-reference-v1.json`.
 
-TODO:
- * events/api.json
-   * $schema uri format
-   * references to schemas
- * schemas
-   * $ref must be relative
-
-*NOTE* the library can only read this format, not write.
+*NOTE* this library can only read data in this format, not write it.
 
 ## URI-Structured
 
@@ -70,57 +65,107 @@ Taskcluster rootUrl.
     └── ...
 ```
 
-TODO:
- * add a single-JSON-file format
+## Serializable
+
+The serializable format is similar to the URI-structured format, but as a
+JSON-serializable data structure consisting of an array of `{filename,
+content}` objects.
+
+```json
+[
+  {
+    "filename": "references/manifest.json",
+    "content": {
+      "$schema": ...
+    }
+  }, ...
+]
+```
 
 # Abstract and Absolute References
 
-A URI-Structured representation of References can either be "abstract" or
-"absolute". The abstract form contains URIs of the form
-`taskcluster:/references/..` and `taskcluster:/schemas/..`, abstract from any
-specific rootUrl. The absolute form contains URIs that incorporate a specific
-rootUrl and thus could actually be fetched with an HTTP GET (in other words,
-the URIs are URLs).
+A References instance can either be "abstract" or
+"absolute". The abstract form contains `/`-relative URIs of the form
+`/references/..` and `/schemas/..`, abstract from any specific rootUrl.  These
+appear anywhere a link to another object in the references or schemas appears,
+including `$schema`, `$id`, and `$ref`.
 
-The library can load either form, but an in-memory References instance is
-always abstract.  Writing an absolute form always requires a rootUrl.
+The absolute form contains URIs that incorporate a specific rootUrl and thus
+could actually be fetched with an HTTP GET (in other words, the URIs are URLs).
 
-# File Contents
+The library can load and represnt either form, converting between them with
+`asAbsolute` and `asAbstract`.  Some operations are only valid on one form.
+For example, json-schema requires that `$schema` properties be absolute URLs,
+so schema operations on schemas should not be performed on the abstract form.
 
-## Manifests
+# API
 
-The `references/manifest.json` file contains a JSON document with the following
-structure (documented in `manifest-vN.json` as linked from its `$schema`
-property):
+To create a References instance, use one of the following methods:
 
-```json
-{
-  "services": [
-    {
-      "serviceName": "someservice",
-      "apis": [
-        {"version": "v1", "reference": "/references/someservice/v1/api.json"}
-      ],
-      "pulse": [
-        {"version": "v2", "reference": "/references/fake/v2/exchanges.json"}
-      ]
-    }
-  ]
-}
+```js
+const References = require('taskcluster-lib-references');
+
+// Build from a built services format (which is always abstract)
+references = References.fromBuiltServices({directory: '/build/directory'});
+
+// Build from a uri-structured format; omit rootUrl when on-disk data is abstract
+references = References.fromUriStructured({directory: '/app', rootUrl});
+
+// Build from a serializable data structure; omit rootUrl if data is abstract
+references = References.fromSerializable({serializable, rootUrl});
+```
+
+To validate the references, call `references.validate()`.
+If validation fails, the thrown error will have a `problems` property containing the discovered problems.
+
+To convert between abstract and absolute references, use
+
+```js
+const abstract = references.asAbstract();
+const absolute = abstract.asAbsolute(rootUrl);
 ```
 
-The services are listed, each with a name and links to the api references and pulse (event) references.
-The links (`reference`) are relative to the rootUrl.
+Note that both of these operations create a new object; Reference instances are
+considered immutable.
+
+To write data back out, use
+
+```js
+// write URI-structured data
+references.writeUriStructured(directory);
+
+// create a serializable data structure
+data = references.makeSerializable();
+```
 
-## Schemas
+To build an [Ajv](https://github.com/epoberezkin/ajv) instance containing all schemas and metaschemas, call `references.makeAjv()`.
+This is only valid on an absolute References instance.
 
-Schemas are available at `schemas/<service>/<version>/<name>.json` in the URI-structured format.
-The the schema's `$id` contains this path.
+To get a specific schema, call `references.getSchema($id)`, with an absolute or abstract `$id` as appropriate.
 
-## Reference Documents
+# Validation
+
+Validation occurs automatically in most operations on this class.
+
+Every schema must:
+* have a valid `$id` within a Taskcluster deployment (so, a relative path of `/schemas/<something>/<something>.json#`);
+* have a valid `$schema`, either a json-schema.org URL or another schema in the references; and
+* validate against the schema indicated by `$schema`.
+
+All `$ref` links in schemas must be relative and must refer only to schemas in
+the same service (so `v1/more-data.json` may refer to
+`../util/cats.json#definitions/jellicle` but not to
+`/schemas/anotherservice/v1/cats.json`). As an exception, refs may refer to
+`http://json-schema.org` URIs.
+
+Every reference must:
+* have a valid `$schema`, defined in the references;
+* have a schema that, in turn, hsa metaschema `/schemas/common/metadata-metaschema.json#`;
+* validate against the schema indicated by `$schema`.
+
+# File Contents
 
-Reference documents, as generated by `taskcluster-lib-api` and `pulse-publisher`, are available at the URLs referenced from `manifest.json`.
-These are the same as the URLs generated by `taskcluster-lib-urls`' `apiReference` and `exchangeReference` functions.
+The contents of the files handled by this library are described in the schemas under `schemas/` in the source repository, or under `/schemas/common` in a Taskcluster deployment.
 
 # Development
 

package.json

@@ -4,9 +4,13 @@
   "main": "src/index.js",
   "engine-strict": true,
   "engines": {
-    "node": "^8.0.0",
+    "node": "^10.0.0",
     "yarn": "^1.0.0"
   },
+  "files": [
+    "src",
+    "schemas"
+  ],
   "scripts": {
     "lint": "eslint src/*.js test/*.js",
     "test": "mocha test/*_test.js",
@@ -16,12 +20,13 @@
   "author": "Dustin J. Mitchell <dustin@mozilla.com>",
   "license": "MPL-2.0",
   "dependencies": {
+    "ajv": "^6.5.5",
+    "js-yaml": "^3.12.0",
     "lodash": "^4.17.10",
     "mkdirp": "^0.5.1",
+    "regex-escape": "^3.4.8",
     "rimraf": "^2.6.2",
-    "taskcluster-lib-urls": "^10.0.0",
-    "taskcluster-lib-validate": "^11.0.2",
-    "walk": "^2.3.13"
+    "taskcluster-lib-urls": "^11.0.0"
   },
   "devDependencies": {
     "eslint-config-taskcluster": "^3.1.0",

schemas/action-schema-v1.yml

@@ -1,4 +1,5 @@
 $schema: http://json-schema.org/draft-06/schema#
+$id: "/schemas/common/action-schema-v1.json#"
 title: Schema for Exposing Actions
 description: |
   This document specifies the schema for the `public/actions.json` used by

schemas/api-reference-v0.yml

@@ -1,6 +1,10 @@
-$schema: 'http://json-schema.org/draft-06/schema#'
+$schema: "/schemas/common/metadata-metaschema.json#"
+$id: "/schemas/common/api-reference-v0.json#"
 title: API Reference File
 description: Reference of methods implemented by API
+metadata:
+  name: api
+  version: 0
 type: object
 definitions:
   scopeExpressionTemplateString:
@@ -79,7 +83,7 @@ definitions:
         additionalProperties: false
 properties:
   version:
-    description: API reference version
+    description: (deprecated)
     type: integer
     enum:
       - 0
@@ -237,7 +241,7 @@ properties:
         - description
 additionalProperties: false
 required:
-  - version
+  - apiVersion
   - $schema
   - title
   - description

schemas/exchanges-reference-v0.yml

@@ -1,12 +1,20 @@
-$schema: 'http://json-schema.org/draft-06/schema#'
+$schema: "/schemas/common/metadata-metaschema.json#"
+$id: "/schemas/common/exchanges-reference-v0.json#"
 title: Exchange Reference File
 description: Reference of exchanges published
+metadata:
+  name: exchanges
+  version: 0
 type: object
 properties:
   version:
-    description: Exchange reference version
+    description: (deprecated)
     enum:
       - 0
+  apiVersion:
+    description: Version of the API
+    type: string
+    pattern: '^v[0-9]+$'
   $schema:
     description: >-
       Link to schema for this reference. That is a link to this very document.
@@ -107,7 +115,7 @@ properties:
         - schema
 additionalProperties: false
 required:
-  - version
+  - apiVersion
   - $schema
   - title
   - description

schemas/manifest-v2.yml

@@ -1,10 +1,9 @@
 $schema: http://json-schema.org/draft-06/schema#
+$id: "/schemas/common/manifest-v2.json#"
 title: "Taskcluster Service Manifest"
 description: |-
   Manifest of taskcluster service definitions available in a taskcluster service deployment.
   These manifests are served from `$ROOT_URL/references/manifest.json`.
-
-  See https://github.com/taskcluster/taskcluster-references for further information.
 type: object
 properties:
   services:

schemas/manifest-v3.yml

@@ -0,0 +1,20 @@
+$schema: "/schemas/common/metadata-metaschema.json#"
+$id: "/schemas/common/manifest-v3.json#"
+title: "Taskcluster Service Manifest"
+description: |-
+  Manifest of taskcluster service definitions available in a taskcluster service deployment.
+  These manifests are served from `$ROOT_URL/references/manifest.json`.
+metadata:
+  name: manifest
+  version: 3
+type: object
+properties:
+  references:
+    type: array
+    description: "Array of URLs of reference documents"
+    items:
+      type: string
+      formt: uri
+additionalProperties: false
+required:
+  - references

schemas/metadata-metaschema.yml

@@ -0,0 +1,35 @@
+$schema: "http://json-schema.org/draft-06/schema#"
+$id: "/schemas/common/metadata-metaschema.json#"
+title: "JSON-Schema Meta-Schema, with the addition of a `metadata` property"
+allOf:
+  - {$ref: "http://json-schema.org/draft-06/schema#"}
+  - type: object
+    properties:
+      metadata:
+        title: "Metadata for this schema"
+        description: |
+          Metadata identifying the documents that the schema document describes,
+          giving both a name (a category of document) and a version (to allow
+          several versions of the same category).  Consumers of the documents can
+          consult the schema metadata to determine how to process the document.
+
+          Any changes to a schema that require changes to consumers of the described
+          documents should be accompanied by a version increase.
+        type: object
+        properties:
+          name:
+            title: "Name of the document category"
+            description: |
+              This is used to identify the category of document for later consumption.
+              It is also used to determine schema id's.  Common values for Taskcluster
+              references are `manifest`, `exchanges`, and `api`.
+            type: string
+          version:
+            title: "Version of the document format"
+            type: integer
+          additionalProperties: false
+        required:
+          - version
+          - name
+    required:
+      - metadata