Why does hrefSchema consider instance data last?

[jdesrosiers]

An "href" with at least one template variable and with an "hrefSchema" allows using external data to resolve the template, and falls back to resolving any remaining variables from the instance.

What is the reason for resolving instance variables last?

It seems to me that it should be the other way around. I used to be able to resolve a hyper-schema against it's instance JSON and get a set of links that I can use to make requests with no dependency on instance data. Now making a request has a dependency on instance data that I didn't used to have.

[handrews]

@jdesrosiers I am re-writing this whole section b/c the changes around forms require a different overview. What you are worried about is not what I meant (I'm pretty sure) so I will fix the wording- thanks for filing this! We'll leave it open until you can verify the PR (I'll put this in the Draft 06 milestone).

The idea is:

• If there is no "hrefSchema", or if "hrefSchema" is false (or any other value that will never successfully validate anything), then client input is not allowed. The behavior is identical to what was present before "hrefSchema" was introduced.

• For a given variable, if it corresponds to a false or otherwise never-validating subschema of "hrefSchema", then that variable does not allow client input. It behaves as it would have in earlier drafts.

• For a given variable, if there is a subschema in "hrefSchema" that will successfully validate some instance(s), then if the client supplies input, it is used. Otherwise it resolves from the instance.

• For a given variable, if there is a subschema allowing input, and the property matching that variable is marked as required in "hrefSchema", then client input MUST be used in order for the link to resolve.

Key observation: Unless you make fields in "hrefSchema" required, you can completely ignore it and everything resolves from instance data by the same rules there were before.

Does that address your concern?

If you want to pre-populate input, but allow the client to override it, that gets into the use of "default", which is it's own otherwise unrelated morass, see #204.

[jdesrosiers]

No, this doesn't change anything.

if the client supplies input, it is used. Otherwise it resolves from the instance

I can't resolve variables from instance data until after I have client input. Links shouldn't be dependent on instance data. In Hyper-Schema, LDOs are more like link templates. They must be resolved against instance data before they can be used. Once they have been resolved, they can be used like regular links where the only dependency is client input. That is no longer possible with the way hrefSchema is defined.

[handrews]

I can't resolve variables from instance data until after I have client input.

I don't know how to say this more clearly: If you don't have client input, it resolves from instance data. You can require client data, but that is there to allow you to simulate an HTML GET form:

[EDIT: But see a few comments down for discussion of client data-but-falling-back-to-instance data scenarios, which I don't address at all in this comment]

Here is a paginated collection of "bars" nested under a specific "foo". First, let's look at the Draft 04 schema:

{
    "href": "/foos/{id}/bars{?offset,limit}",
    "method": "get",
    "schema": {
        "type": "object",
        "properties": {
            "offset": {"type": "integer", "minimum": 0},
            "limit": {"type": "integer", "minimum": 1}
        }
    }
}

Now, in the proposed Draft 06:

{
    "href": "/foos/{id}/bars{?offset,limit}",
    "hrefSchema": {
        "type": "object",
        "properties": {
            "id": false,
            "offset": {"type": "integer", "minimum": 0},
            "limit": {"type": "integer", "minimum": 1}
        }
    }
}

These schemas behave exactly identically.

The point of this is to allow more flexibility with URI design and evolution. HTML GET forms artificially treat the query string as malleable and every other part of the URI as fixed. There is nothing inherent in RFC 3896 that suggests that should be true. It's just what HTML decided to do with forms because it addressed the most obvious use cases.

But I need to be able to rearrange URIs without breaking things. Specifically because nested collections turn out to be extremely limiting as a design. So I want to be able to change to this schema:

{
    "href": "/bars{?offset,limit,fooId}",
    "hrefSchema": {
        "type": "object",
        "properties": {
            "fooId": false,
            "offset": {"type": "integer", "minimum": 0},
            "limit": {"type": "integer", "minimum": 1}
        }
    }
}

Now I can work with bars as a top-level collection, but in this case I want to pre-filter the collection (presumably because this link is from a foo and the link relations indicates the semantics of "the bars that this foo owns".

This also allows changes in the other direction. Here is a URI design where pages are nested in the path (no, I would never do this, please go with the spirit of the example and ignore whether it's good collection design or not- it's not).

{
    "href": "/foos/{id}/bars/{offset}{?limit}",
    "hrefSchema": {
        "type": "object",
        "properties": {
            "id": false,
            "offset": {"type": "integer", "minimum": 0},
            "limit": {"type": "integer", "minimum": 1}
        }
    }
}

All three of these Draft 06-style schemas behave identically, even though I keep moving template variables back and forth between the path component and the query string component. This is a major technique for correcting bad URI design and absolutely critical for long-term maintainability of APIs.

[awwright]

We definitely should be able to generate a list of links (including link templates with user variables) with all instance variables already bound and substituted. Otherwise, JSON Hyper-schema isn't properly abstracting the generic hypermedia concerns away from the JSON-specific concerns.

Can we include a little paragraph on how to do this?

[handrews]

@awwright when you talk about "user variables" and "instance variables", do you mean that some variables in the template are instance-only and some are user-only? Because I think that the examples in my last comment meet that criteria. Are those examples sufficient?

I do touch on this in #290 but if it is not clear I definitely agree with you that it should be.

[handrews]

@awwright @jdesrosiers If the confusing case is accepting user input but falling back to instance data, let me explain my use case: I am open to alternate solutions, and will talk a bit about that.

Comment forthcoming, wanted to note that I'm open to discussion on that part while I work up the example.

[handrews]

I think the scenario where each variable is either only instance-resolved or only client data-resolved is pretty solid. The scenarios where a variable can be resolved from either depending on what data is available are less solid.

Here is a use case that is clear, but may be better handled by defining multiple links with different relations (@jdesrosiers, for this example, you are more than welcome to suggest alternate schema design approaches! :-)

This is a collection that takes a search query string, plus an offset and limit for pagination. All three are optional.

The collection's representation includes the value of all three variables. This is so that the "self" link can be re-constructed from the instance data (this is a design requirement that I use for resource representation design).

Here is the schema written out, requiring all of that information from the instance. I'm pretty certain this version is a reasonable design:

{
    "type": "object",
    "properties": {
        "elements": {"items": {...}},
        "searchQuery": {"type": "string"},
        "offset": {"type": "integer", "minimum": 0}, 
        "limit": {"type": "integer", "minimum": 1}
    },  
    "links": [
        {   
            "rel": "self",
            "href": "/foos{?searchQuery,offset,limit}"
        }
    ]   
}

---

So let's look at a very dubious version of this. Here is a schema that allows a client to override the search and pagination variables to look at a different subset of the collection:

{
    "type": "object",
    "properties": {
        "elements": {"items": {...}},
        "searchQuery": {"type": "string"},
        "offset": {"type": "integer", "minimum": 0}, 
        "limit": {"type": "integer", "minimum": 1}
    },  
    "links": [
        {   
            "rel": "self",
            "href": "/foos{?searchQuery,offset,limit}",
            "hrefSchema": {
                "type": "object",
                "properties": {
                    "searchQuery": {"type": "string"},
                    "offset": {"type": "integer", "minimum": 0}, 
                    "limit": {"type": "integer", "minimum": 1}
                }   
            }   
        }   
    ]   
}

This is definitely useful and powerful. But it's definitely an abuse of the "self" link relation. There is only one possible resolution of the template that actually points to "self", so allowing a user to override the parameters is strange.

I noted with the earlier examples that HTML GET forms artificially encourages thinking of the URI components up through the path as the "real" URI, and the query string as data. Once upon a time, that caused me to think of the "self" link as only applying up through the path component. In which case being able to override the query string parameters through the self link made sense (at the time- not now).

Probably the correct approach here is to not allow users to override, and instead only define client-settable variables in the link that brings you from somewhere else to this resource. So this would be the "collection" link from each item, and/or some sort of "tag:example.com,2017:foos" link from either the entry point, or from some other resource that has related foos.

So I do see this as an abusable feature, which is perhaps your concern, @jdesrosiers? But that doesn't mean it's a bad feature.

---

However, there are other use cases. The "next" link would want to keep the same search query, and would use a specific offset (existing offset + existing limit + 1). But you might want to allow the client to select a different limit for the next page. Yes, I know this is not a rock-solid example of awesome design, but I don't have time to work out a better one today.

So here's what that would look like:

{
    "type": "object",
    "properties": {
        "elements": {"items": {...}},
        "searchQuery": {"type": "string"},
        "offset": {"type": "integer", "minimum": 0}, 
        "limit": {"type": "integer", "minimum": 1},
        "nextOffset": {"type": "integer", "minimum": 0}
    },  
    "links": [
        {   
            "rel": "next",
            "href": "/foos{?searchQuery,nextOffset,limit}",
            "hrefSchema": {
                "type": "object",
                "properties": {
                    "searchQuery": false,
                    "nextOffset": false, 
                    "limit": {"type": "integer", "minimum": 1}
                }   
            }   
        }   
    ]   
}

The way this is written, if the client supplies a new limit, that will be used. Otherwise the next page will be the same size as the current page. This makes semantic sense: the "next" link relation does not impose uniform pagination constraints as far as I can tell (and if it does, an API could define a custom variable-size-next relation of some sort).

So there is a valid use case for this functionality. Whether this is a reasonable design with "next" or not, it should not be too hard to imagine that this general pattern is useful: there is a related resource, some aspect of which is suggested as a default by the context resource. But the link is valid with any value for that aspect.

Oh, hey... "default"...

---

I originally proposed this sort of thing in #108. As proposed in that issue setting a schema of false for a variable made is unusable. Instead, you used "$data" with either "const" or "default".

Here's what the "next" link looks like in that approach:

{
    "type": "object",
    "properties": {
        "elements": {"items": {...}},
        "searchQuery": {"type": "string"},
        "offset": {"type": "integer", "minimum": 0}, 
        "limit": {"type": "integer", "minimum": 1},
        "nextOffset": {"type": "integer", "minimum": 0}
    },  
    "links": [
        {   
            "rel": "next",
            "href": "/foos{?searchQuery,nextOffset,limit}",
            "hrefSchema": {
                "type": "object",
                "properties": {
                    "searchQuery": {"const": {"$data": "0/searchQuery"}},
                    "nextOffset": {"const": {"$data": "0/nextOffset"}}, 
                    "limit": {
                        "type": "integer",
                        "minimum": 1,
                        "default": {"$data": "0/limit"}
                    }
                }   
            }   
        }   
    ]   
}

These semantics are arguably more clear anyway. The "searchQuery" and "nextOffset" keywords are pinned to an instance value with "const", while "limit" is given a default based on the instance.

This approach is a lot more flexible, since it does not force an order of operations on the client. Because of the vague semantics of "default", a client may use that a value to pre-populate a user form before presenting it to the user. Or a client may silently use the default in the absence of input, particularly if the client is not directly handling user interactions (b/c it is a lower-level programmatic client). Neither of these usages are required, but a given application may decide that either of them are appropriate. Or it may ignore the default altogether, assuming that it is just documenting server-side behavior.

---

So I actually much prefer the "$data" approach, and if I am correct that @jdesrosiers objects to the order of operations defined by "hrefSchema", it addresses that concern. But it involves "$data" and Relative JSON Pointer, both of which are highly controversial. So when @awwright discussed #108 a while ago, he suggested something closer to the current "hrefSchema" to get some feedback on the idea of providing a schema for URI templates without having to deal with the "$data" can of worms. That seemed very reasonable to me, so I worked out the details and proposed "hrefSchema".

My preference would be to leave "hrefSchema" as it is for draft-wright-json-schema-hyperschema-01, so that others can encounter this ambiguity and give us feedback on it. That will help us understand what need, if any, there is for the "$data" solution.

The current definition of "hrefSchema" was always understood to be incomplete- for instance, it does not solve the problems addressed by the old pre-processing scheme, nor does it solve the numerous scenarios that pre-processing didn't solve either (mapping a template variable to property elsewhere in the instance). So we will make further changes here, and I expect the area of resolving/defaulting data to be an area that changes.

I can add a CREF note explaining that we think this part of "hrefSchema" is at least somewhat problematic and may change. @jdesrosiers would that make you more comfortable?

[jdesrosiers]

@handrews You're still completely misunderstanding. Let's assume we have this instance data.

{
  "id": 1,
  "foo": 2
}

Now let's look at a few hyper-schemas and see how they would resolve against that instance data.

{
  "links": [
    { "rel": "self", "href": "/foo/{id}" }
  ]
}

... resolves to ...

{
  "links": [
    { "rel": "self", "href": "/foo/1" }
  ]
}

The resolved LDO can now be used without any dependency on the instance data. Nothing new or special here. Here's another.

{
  "links": [
    {
      "rel": "some-rel",
      "href": "/foo/{id}/bar/{bar}",
      "hrefSchema": {
        "type": "object",
        "properties": {
          "bar": { "type": "integer" }
        }
      }
    }
  ]
}

... resolves to ...

{
  "links": [
    {
      "rel": "some-rel",
      "href": "/foo/1/bar/{bar}",
      "hrefSchema": {
        "type": "object",
        "properties": {
          "bar": { "type": "integer" }
        }
      }
    }
  ]
}

Again, this resolved LDO has no dependency on instance data. No problem here. Now let's get to the point.

{
  "links": [
    {
      "rel": "some-rel",
      "href": "/foo/{id}/foo/{foo}",
      "hrefSchema": {
        "type": "object",
        "properties": {
          "foo": { "type": "integer" }
        }
      }
    }
  ]
}

... resolves to ... Oops, we can't resolve it because we don't know if we are going to get client data or not. We now have to keep the instance data in scope in case we need it to finish constructing the href. We can't resolve the instance data until after we have client input (or know that we won't have client input). I think we should have a very good reason if we are going to impose this dependency on hyper-schema. This issue what about learning what that reason is and discussing whether it was a good enough reason to justify the architectural drift.

---

The example you wrote about that used $data and Relative JSON Pointer would address this problem if $data and Relative JSON Pointer become part of JSON Schema.

{
  "links": [
    {
      "rel": "some-rel",
      "href": "/foo/{id}/foo/{userFoo}",
      "hrefSchema": {
        "type": "object",
        "properties": {
          "userFoo": { "type": "integer", "default": { "$data": "foo" } }
        }
      }
    }
  ]
}

... resolves to ...

{
  "links": [
    {
      "rel": "some-rel",
      "href": "/foo/1/foo/{userFoo}",
      "hrefSchema": {
        "type": "object",
        "properties": {
          "userFoo": { "type": "integer", "default": 2 }
        }
      }
    }
  ]
}

I specifically had to choose a variable name that didn't conflict with the instance data, but this does allow for falling back on instance data without breaking resolution.

[handrews]

@jdesrosiers I think I see what's going on here, and the problem is a gap in the specification rather than a true contradiction. There are two ways to use a link:

1. Perform an operation with it (HTTP method for an http:// link, read the data from a data: link, etc.)
2. Render the link to a user so that they can decide whether or how to use it

I have been focusing on case 1, for which (I think) all of this works. But I'm guessing you are thinking in terms of case 2, and I agree that it is not clear that "hrefSchema" works for that.

In case 1, you MUST have the schema, the instance, and any client data in order to use the link. There is no keeping anything in scope, because it's a single action- you take what you have and you perform the operation. If the URI template fails to resolve because a required variable doesn't appear in either data source, that's an error.

In case 2, you have the schema and an instance, but you are waiting for possible data. This is the only situation I can think of where your "keep the instance data in scope" phrase absolutely makes sense. You have done part of the work, you know you'll need to finish the work (case #1) after you get the client input, but you don't know what input you'll get yet. So you need to check it, and maybe check the instance data again.

In the "$data" approach this is easily solved because one way to use "default" is to pre-populate the form field (or however you are rendering the link). Since the default comes from the instance data, you know that you'll get either the instance data back or real client input. If the hrefSchema allows nulling out the field, then you can even use such an interface to allow a user to unset the variable.

The current proposal doesn't talk about how to handle this, but there is nothing in the spec that prevents you from looking at the instance data and treating it as a default for form rendering.

The spec is written entirely from the perspective of case 1, so it takes the form "once you have data, first you look at the data, then you look at the instance."

I think it needs additional wording to indicate that a client is free to treat an instance value that can be overridden as if it were a "default". That does not require any change to the current syntax, and avoids the whole "$data" problem, but gives you exactly the functionality that is needed to present a form, and then accept the form data without re-checking whether a given variable needs to do any sort of fallback.

Would that address your concern?

[handrews]

@jdesrosiers a key observation here is that a minimal hyper-schema implementation (generic client) MUST handle case 1. If it is given (all at once) everything needed to perform an operation with a link, then it MUST be able to perform the operation.

But a minimal hyper-schema implementation would not directly implement case 2. It would support exposing all of the information necessary to implement case 2, but the application would need to decide what to do with that information, and how to present it to the user.

The only real difficulty I can think of at this point is if the application wants to present the resolved-from-instance URI to the user, but some variables have both an input schema and an instance value fallback, the application has two choices:

1. Display the as-fully-resolved-as-possible URI and ensure that the user-submitted data will contain all values, so no further fallback:
   • Resolve the instance-only variables
   • Pre-fill the form with the instance value for the fallback variables
   • Act from that point as if all unresolved form fields are user input only when it comes to rendering the target URI

OR

2. Notice that resolving everything that can resolve from the error may produce a misleading URI due to the possibility of user override, and raise an error because this violates the application's expectation despite being a legal schema.

[jdesrosiers]

you MUST have the schema, the instance, and any client data in order to use the link

This is exactly what I am saying is the problem. Using the link should not depended on the instance.

I see no difference in how a client should behave in case 1 vs case 2 (minimal or not). Maybe there is a difference, but presenting URLs to a user is not a use case that motivated this discussion. My concern is (to use @awwright's words) "abstracting the generic hypermedia concerns away from the JSON Schema-specific concerns". The difficultly in presenting to the user is a manifestation of this coupling, not simply a different use case.

[handrews]

@jdesrosiers then I think we see JSON Hyper-Schema in fundamentally different ways :-(

To me, JSON Hyper-Schema is a specification for using a JSON Schema vocabulary together with a JSON instance as a composite hypermedia document. The specification describes how an implementation uses the two things together, in some cases also with outside data (which may or may not come from a "user").

There is no using hyper-schema without an instance. There are specific scenarios where you may be able to build out links in a single pass, discard the instance data, and work with the links, but I do not recall anything anywhere that makes it a requirement for all aspects of Hyper-Schema, and I do not see any reason why it should be.

In my view, discarding the instance at any point is fundamentally opposed to the entire JSON Schema paradigm. JSON Schema vocabularies describe sets of possibilities, which resolve to actual concrete things when paired with an instance. The strength of JSON Hyper-Schema is in its dynamic nature: you don't pre-compute everything and then use it, you build things as needed based on the instance (and external data) at hand.

While it's possible that I will suddenly get why you think it should work this way, right now I think it's best that I step back from this issue to de-escalate. Someone else may come by and understand your argument and you can work it out with them, and I will be happy to jump back in when I see a way to contribute positively.

But I think I'd be doing more harm than good continuing to debate, so hopefully if I stop talking someone else will be able to help resolve your concerns.

[Relequestual]

Having read the majority of this issue, I'm on the same page as @handrews with this...

There is no using hyper-schema without an instance.

I'm not sure I get the use case of using hyper-schema WITHOUT the instance @jdesrosiers. After all, a link describes things related to the instance. How can I get related things to an instance without the instance data?

[Relequestual]

Using the link should not depended on the instance.

@jdesrosiers

What's the purpose of such a link?

[jdesrosiers]

@handrews I agree that this discussion needs a break. If nothing else, at least we have identified what the core disagreement is. It seems we don't always get even that far, so I think that this already going better than the last one.

Someone else may come by and understand your argument and you can work it out with them

@awwright seemed to understand the concept. Maybe he can jump in an explain his perspective.

[jdesrosiers]

There is no using hyper-schema without an instance.

I'm not sure I get the use case of using hyper-schema WITHOUT the instance

@Relequestual, I am not suggesting that hyper-schema should (or even can) be used without the instance. Nor do I think @handrews was implying that I was making such an assertion with that statement.

Let me make an OO analogy. It's not a perfect analogy, but I'm going to risk it. A hyper-schema is like a class. Just as a class is a template for creating objects, a hyper-schema is a template for creating links. The instance data is used to instantiate the hyper-schema as a set of links. The links can then be used to make requests without dependency on the instance data. With the way hrefSchema is currently defined, we can on longer instantiate links without a dependency on instance data. This property has always existed in hyper-schema until hrefSchema was proposed. The question is whether the decoupling of links from instance data is an accidental property of hyper-schema that we can change without consequence, or whether it is an essential property of hyper-schema and should be preserved.

[handrews]

The question is whether the decoupling of links from instance data is an accidental property of hyper-schema that we can change without consequence, or whether it is an essential property of hyper-schema and should be preserved.

Just for the record to show that I am following your discussion here, I agree that this is the fundamental question, and I strongly believe that it was an accidental property, and furthermore not a desirable one.

[handrews]

I think we're all OK with this not being resolved for draft-06? I'm taking it out of the milestone but if there are disagreements on that point please just speak up.

[handrews]

@jdesrosiers I think I may finally be getting what you're talking about here. Need to work few a few more things but will update soon. Also, I'm deleting [EDIT: have deleted] the last two comments as I now think they're in the wrong direction and just add noise.

[handrews]

OK, here's my latest attempt at getting this right :-)

The feature of accepting client data and falling back to instance data was intended to be roughly analogous to setting value on a non-hidden HTML form <input> element.

The spec forbids relying on any particular behavior for "default", so it is not analogous to value. With value, if the client takes no action before submitting the form, the server will get the value back. With "default", a conforming client may use the value, or may not. Or may make it available but require an action on the part of the user-agent to accept it and use it.

hidden form <input> fields are analogous to setting the "hrefSchema" value for those fields to false.

Non-hidden form <input> fields with no initial value are analogous to an "hrefSchema" property with no corresponding instance property (either it doesn't exist at all, or it is omitted from the particular runtime instance due to not having a value yet).

In all cases, JSON Hyper-Schema allows this for any portion of the URI, while HTML forms only work with the query string. As discussed elsewhere, this is correct because the query string is not special in terms of which parts of a URI can and cannot be templated. It's just the most flexible, so that's what HTML used.

---

So, I see why you were confused and concerned, and the "hrefSchema" section probably needs rewriting to be more clear, but in terms of the intent, it is no more burdensome than supporting value.

I think I was overly specific when I gave a particular order of implementation steps. The spec should talk more about the functionality (the server sets a value which the client can change, but that will be sent back unchanged if the client takes no action) than about how it is implemented.

Another problem with the current language is that it is unclear whether the instance data should be presented to the user agent as the value that will be used if they do not take action. That's an important distinction, because "default" already handles the case where the user agent does not necessarily see a value, but the behavior is documented to be equivalent to setting the default. Here, we want to reliably pre-initialize the input.

If those things are clarified and the specification of exactly what steps the implementation needs to take in order to meet the requirements is loosened, how would that work for you?

[handrews]

@jdesrosiers ...and now I see that I actually did work all of this out before, and PR #291 is almost exactly this solution, but you shot it down.

Fundamentally, you just don't seem to want to allow the server to pre-set values that the client can override. But that's an essential and extremely common use case.

If you want anything else from this, you need to propose an actionable alternative that satisfies the use case. Otherwise I'm going to close this. The feature behaves as intended, and has an exact analogy in HTML forms. While I refuse allow HTML's frequently antiquated and always human user-oriented restrictions to constrain JSON Hyper-Schema, I also refuse to allow JSON Hyper-Schema to be less functional than HTML.

I'm probably going to re-submit that PR with a few improvements to make the use case and motivation more clear, as I think it is a helpful clarification. I may instead submit a more comprehensive rewrite to address several clarity issues. But either way, I don't see another solution to the use case. If you have one, please propose it. I am open to suggestion.

[handrews]

Closing per plan stated in previous comment. If you have an actionable proposal for this that meets the stated use cases, please file it as a new issue as this is too long and confusing to attract comments from anyone else.