v6 hyper-schema: extended templating (vars+template)

[handrews]

Originally written by @geraintluff at https://github.com/json-schema/json-schema/wiki/Extended-templating-syntax-(v5-proposal)

Proposed keywords

Uses existing keyword - proposes extension of href in LDOs.

(Should this syntax also be allowed inside "$ref" to allow templating of references?)

Purpose

Currently, the only values available for templating in href are the object itself, and the immediate children of the object (which must be referred to by their exact name).

This proposed new syntax would allow more powerful templating, specifying values from the data using Relative JSON Pointer.

It would also allow the re-naming of template variables. This is useful because in some templates, the variable name is actually included in the results, e.g.

• /prefix/{?foo,bar,baz} -> /prefix/?foo=1&bar=2&baz=3

Values

In addition to the existing string values/behaviour for href, the following is proposed:

The value of href may be an object, containing the following properties:

• template - containing a URI template
• vars - an object, where all the values are Relative JSON Pointers

Behaviour

To obtain the URI for the link, the URI template in template is expanded. When a variable is referenced by the template, its value is obtained like so:

• if the variable name is a defined property in vars, then:
  • the corresponding value in vars is interpreted as a Relative JSON Pointer, and resolved relative to the current data instance.
  • the result of resolving the relative pointer is used as the value for the variable in the template
• otherwise, the variable name is percent-decoded and taken as the name of an immediate property.

(Note the complete lack of pre-processing rules - they are not needed here, due to the expressive power of Relative JSON Pointers.)

Example

Data:

{
    "author": {"id": 241, "name": "Jenny"},
    ...
}

Schema:

{
    "links": [
        {
            "rel": "author",
            "href": {
                "template": "/users/{authorId}",
                "vars": {
                    "authorId": "0/author/id"
                }
            }
        }
    ]
}

Concerns

Mismatched pre-processing rules

This syntax is in many ways much simpler than the existing syntax, because there is no need for escaping rules. (The current syntax does pre-processing using (, ) and $.)

We are faced with a choice, then - to make the new syntax equally complex, or to have complex pre-processing rules for URI Templates in some situations but not others. (Or, of course, remove the old plain-string syntax, which will impact brevity as well as backwards-compatability.)

Usage inside $ref

Allowing templating inside $ref would force all validators to implement link-parsing - currently, validators can ignore all hyper-schema aspects, which is convenient.

Use inside $ref would limit static analysis for schemas. However, (like $data) allowing this keyword in $ref would open up quite a lot of expressive power.

Use inside $ref would also ruin our ability to describe $ref relationships in the meta-schema. Currently, the $ref behaviour is characterised by a full link, but allowing templating would undermine that.

Current behaviour, and rel="describedby"

The behaviour of templating $ref can currently be mirrored by adding a rel="describedby" link:

{
    "links": [{
        "rel": "describedby",
        "href": "/schemas/{type}"
    }]
}

The only difference is that validators are not obliged to take any notice of links. "Hyper-validators" should, but it is not expected that plain validators would.

[handrews]

Opinions on various points:

vars should allow schemas in addition to relative JSON Pointers

The vars section should not only describe how to resolve the variables from an instance, but also support validation of user-supplied data when resolving the templates in other ways. This should work as follows, taken from Riverbed Technology's service definition format which made extensive use of this mechanism:

An entry in the "vars" field can either adjust the starting point for that resolution through a relative JSON pointer, or provide a schema, indicating that the value must be provided from an external source. Any schema, whether implied through the resource's schema or given explicitly, must describe a type that can be resolved through the rules for that variable as described in the URI templates standard (RFC 6570). Otherwise the resulting behavior is undefined.

While we found it to be a best practice to always be able to fully resolve the self link from the instance data, it doesn't seem to be something that should be strictly required.

Legacy pre-processing rules

Toss them. They're ugly and confusing and not worth the burden of carrying them forwards. It's also not horribly difficult to migrate from them to vars. People who don't want to update them can keep using the Draft 04 meta-schema until they are ready to move forwards.

Templating $ref

No use case is given for this, and it introduces considerable complication. I move that this be dropped unless/until compelling use cases are identified. It clearly could do a lot of things, but it's not clear that those things are desirable.

[handrews]

Having split the most urgent portion of this out into #142, I am going to close this. It's an omnibus that is hard to discuss all at once.

Templating $ref is not something I have heard anyone else request. If someone wants that, please open it as its own issue. The question in that issue should just be whether to template $ref, yes or no. The question of whether templates use pre-processing, mapping, or some other mechanism should be decided separately and applied to all template uses (currently href and base, possibly also $ref and rel).

My comment tacked on more proposals about user input and validation. That's a complex topic, the requirements for which are being explored in #108 which is a better vehicle for the discussion (it's at least garnered a few comments, while this issue has not).