Allow regular JSON pointers for $data reference (draft5)

[HotelDon]

Right now, the proposed $data reference only supports relative JSON pointers. While this is useful and more compact for referencing data nearby, it's a big pain in the ass for referencing data that is further away. As an example (note that I'm using YAML for readability):

type: object
properties: 
    foo:
        enum:
            $data: "1/enumList"
    bar:
        type: object
        properties:
            baz:
                enum: 
                    $data: "2/enumList"
    enumList: 
        type: array     

At a small scale, the only real issue is the lack of code reuse, as you can't reuse the enum if the data is closer or further from the root. However, the problem gets progressively worse the larger and more complex the schema gets. It's not, however, an insurmountable problem. The real problem is when you start using recursive schemas:

type: object
properties:
    foo:
        $ref: "#/definitions/recursiveSchema"
    enumList: 
        type: array

definitions:
    recursiveSchema:
        type: object
        properties:
            bar:
                enum:
                    $data: "2/enumList"
            foobar:
                $ref: "#/definitions/recursiveSchema"

Example Data:

foo:
    bar: "Allowed"
    foobar:
      bar:  "Not Allowed"
      foobar:
        bar:  "Allowed"
enumList: ["Allowed"]

Now, what I expected to happen here was the relative pointer only works on the first level, and then simply fails to resolve the enum for every level after. What actually happens is AJV realizes the relative pointer is completely out of whack and assumes something is wrong with the JSON, and rejects any data you give it that follows the schema. Remove either the recursion or the enum, and whatever is left works just fine.

With regular JSON pointers, the fix is easy: instead of "[number]/enumList", it would simply be "/enumList", and it would resolve properly regardless of where you are in the document. All absolute pointers start with a slash, and all relative pointers start with a number, so there would never be any confusion about which is which.

Adding in JSON pointers shouldn't be hard for the people who've already implemented the $data reference, but it would be nice if it was part of the actual specification.

[handrews]

+1 to this. It should be possible to use both absolute and relative JSON pointers anywhere instead of inconsistently using one or the other or both.

[HotelDon]

I helped to add this feature to AJV (which I think is the only library supporting v5 as of right now? If not, someone tell me so I can get to work). I also edited the $data proposal on the old repo's wiki, since it looks like nobody is paying attention to it right now. If you wanna try out absolute pointers, you can with Json Schema Lint.

[handrews]

I went back and looked at the Relative JSON Pointer draft RFC, which says:

JSON Pointer is a syntax for specifying locations in a JSON document,
starting from the document root. This document defines an extension
to the JSON Pointer syntax, allowing relative locations from within
the document.

So I think it's appropriate to treat any mention of Relative JSON Pointer as automatically including support for absolute JSON pointers.

[awwright]

@HotelDon There's no such thing as "v5" yet... And the next draft won't contain any new features.

I don't see anything wrong with testing out features in implementations, but keep in mind it's strictly for experimentation

[HotelDon]

@awwright There is an entire page on the old repo dedicated to listing proposed features for the draft 5 spec. I understand that nothing is official yet, but are you telling me that ALL of those ideas are being thrown out?

[awwright]

@HotelDon Proposals are being moved/should be moved to issues on this tracker, I think that's described in #16

[HotelDon]

@awwright If they'll eventually be moved over, what's the rationale for not including any new features into the next draft? If there's some sort of deadline that we're up against, I would be more than happy to help get those turned into feature requests on the issue tracker.

[awwright]

json-schema/json-schema#167 has most of the discussion on the aim and the problems we've run into... Basically the latest released Internet-Draft (draft-04) has so many problems with it, it's causing disagreements and problems with feature suggestions and among implementors.

So we're spending a bit of time to step back fixing definitions, making things internally consistent, consistent with other standards we're supposed to be playing nicely with, and updating references to their latest published versions where possible.

I don't know how that's going to impact a lot of the suggestions right now. Most of them seem to go beyond structural validation (and $data seems to be among this group), into the realm of runtime validation.

This isn't a bad thing, but it suggests it's more likely to be covered by a specialized JSON Schema vocabulary like JSON Hyper-schema, instead of JSON Schema Validation.

[HotelDon]

I would feel less trepidation about going that route if it wasn't for the fact that JSON Hyper-schema, in it's current form, feels completely divorced from the rest of the spec. While it's kept the core JSON Schema Validation pure, it's also made it easier to just completely ignore Hyper-schema entirely - there aren't even any examples on the main site to help people understand what it's for, you have to go digging into the spec page itself. I don't disagree with it being a separate thing, but you could make the argument it belongs in the core standard - JSON is/was built for web browsers, after all.

The current suggestions for draftN+1 contain a lot of useful ideas, and their usage feels a lot less niche than Hyper-schema does. It would be a shame if 9X% of validators didn't implement them simply because they were an optional part of the spec.

[handrews]

@HotelDon I think having JSON Hyper-schema slightly separate is helpful for thinking about the different pieces. I agree that hyper-schema does not get enough attention (and I have some concerns about it which I will file as specific issues for draftN+1). I think the appropriate action is to promote hyper-schema more clearly as its own thing rather than folding it back in with validation.

[handrews]

@HotelDon , @awwright : Can we close this issue since (I think) the original concern has been addressed? If we want to have a longer discussion about how Hyper-schema is managed, that deserves its own issue (or an email thread followed by an issue for any action determined).

[HotelDon]

I was using hyper schema to illustrate a point, I wasn't trying to completely derail the topic at hand. But I already made the change to the proposal, nobody objected or reverted the change, and since debating the proposals is currently on the backburner, you are technically correct that this issue is resolved. Consider it closed.

[awwright]

@HotelDon I don't think anything was derailed. Though I'm not sure what else to say because I don't see "$data" as a very good solution to whatever problem it is that's trying to be solved.

It seems many of the cases either could either be handled with really basic (non-turing-complete) scripting (e.g. {"assert": "{start} >= 0 && {start} < {end}"}), or aren't really worth asserting (too runtime-specific, like asserting a reference to a database record exists).

If there's a specific use case, maybe open up an issue for that, and then we can look and see if it's worth solving, and if so, how.

[HotelDon]

@awwright I'm going to wait for the proposals to be moved over before I do anything else.

[handrews]

@HotelDon I'll move the $data one over next.