Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

$data #51

Closed
handrews opened this issue Sep 16, 2016 · 31 comments
Closed

$data #51

handrews opened this issue Sep 16, 2016 · 31 comments

Comments

@handrews
Copy link
Member

@handrews handrews commented Sep 16, 2016

Originally written by @geraintluff at https://github.com/json-schema/json-schema/wiki/%24data-(v5-proposal)

NOTE: JSON Relative Pointer is defined as an extension of JSON Pointer, which means that an absolute JSON pointer is legal anywhere that a relative pointer is mentioned (but not vice versa).

Absolute JSON Pointers always begin with /, while relative JSON pointers always begin with a digit. Resolving a pointer beginning with / behaves the same whether it is being resolved "relative" to a specific location or not, just as resolving a URI "/foo/bar" is resolved the same whether there is an existing path component to the URI or not.

Proposed keywords

  • $data

This keyword would be available:

  • inside any schema
  • contained in an object ({"$data": ...}) for the following schema properties:
    • minimum/maximum
    • exclusiveMinimum/exclusiveMaximum
    • minItems/maxItems,
    • enum
    • more...
  • contained in an object ({"$data": ...}) for the following LDO properties:
    • href
    • rel
    • title
    • mediaType
    • more...

Purpose

This keyword would allow schemas to use values from the data, specified using Relative JSON Pointers.

This allows more complex behaviour, including interaction between different parts of the data.

When used inside LDOs, this allows extraction of many more link attributes/parameters from the data.

Values

Wherever it is used, the value of $data is a Relative JSON Pointer.

Behaviour

If the $data keyword is defined in a schema, then before any further processing of the schema:

  • The value of $data is interpreted as a Relative JSON Pointer.
  • The pointer is resolved relative to the current instance being validated/processed/etc.
  • The resolved value is taken to be the value of the schema for all further processing.

When used in one of the permitted schema/LDO properties, then before any further processing of the schema/LDO:

  • The value of $data is interpreted as Relative JSON Pointer.
  • The pointer is resolved relative to the current instance being validated/processed/etc.
  • The resolved value is substituted as the property value.

Example

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "properties": {
        "smaller": {"type": "number"},
        "larger": {
            "type": "number",
            "minimum": {"$data": "1/smaller"},
            "exclusiveMinimum": true
        }
    },
    "required": ["larger", "smaller"]
}

In the above example, the "larger" property must be strictly greater than the "smaller" property.

Concerns

Theoretical purity

Currently, validation is "context-free", meaning that one part of the data has minimal effect on the validation of another part. This has an effect on things like referencing sub-schemas. Changing this is a big issue, and should not be done lightly.

Some interplay of different parts of the data can currently be specified using oneOf (and the proposed switch) - but crucially, these constraints are specified in the schema for a common parent node, meaning that sub-schema referencing is still simple.

The use of $data also (in some cases) limits the amount of static analysis that can be done on schemas, because their behaviour becomes much more data-dependent. However, the expressive power it opens up is quite substantial.

Not available for all keywords

It's also tempting to allow its use for all schema keywords - however, not only is that a bad idea for keywords such as properties/id, but it also might present an obstacle to anybody extending the standard.

Not available inside enum values

It should be noted that while {"enum": {"$data":...}} would extract a list of possible values from the data, {"enum": [{"$data":...}]} would not - it would in fact specify that there is only one valid value: {"$data":...}.

Similar concerns would exist with an extra keyword like constant - what if you want the constant value to be a literal {"$data":...}? However, perhaps constant could be given this data-templating ability, and if you want a literal {"$data":...}, then you can still use enum.

Describing using the meta-schema

The existing mechanics of $ref can be nicely described using a rel="full" link relation.

The mechanics of $data, however, would be impossible to even approach in the meta-schema. We could describe the syntax, but nothing more. Is this a problem?

@handrews
Copy link
Member Author

@handrews handrews commented Sep 17, 2016

I feel that there are several use cases here, and it might be best to split them up.

URI Template resolution

For hypermedia interactions, where the instance data must be referenced in order for hyperlinking to work at all, the extended templating syntax from issue #52 covers the necessary use cases. It narrowly targets hyperlinking and does not involve instance data in any other aspect of JSON Schema, as it applies only in situation where we were already referencing instance data.

Link title

I can see how a link title might reference data in the same way as the URI Template. The template may include the id of a related thing, while the title may include the related thing's name. Either way, the URI and the title are both things that are presented back to the user, which should be affected by the instance data as they are describing a relation involving that data.

Therefore I would prefer to see the URI Template extended syntax (with "vars") be used here rather than a more generic approach that applies to more than just hypermedia values.

rel and mediaType

The use cases for mediaType and rel are not immediately obvious to me. The relation type should not, in my mind, change based on the instance data. Only the specific instance to which the relation points should change. A mediaType specified at runtime from instance data would not be of use in planning what a program can do with different representations. It's not clear to me why you wouldn't just list out the possibilities. If some media type links may only be present some of the time, there are other ways to express that using "oneOf" (or possibly "switch") to associate links with only certain variations of the content.

Interactions during validation

I feel like this should somehow express the constraints in terms of the relations among the fields rather than pulling in data that will produce the desired result. So somehow explicitly saying that "larger" should be strictly greater than "smaller" without loading the data into the schema before validation.

I'm not 100% sure what that would look like. In this case, possibly very much like the $data example, as either way you need to reference the related field. But I would be more comfortable with something that clearly states "this is describing relationships among data" rather than "this loads a value from instance data, and treats it as part of the schema, whatever that happens to mean."

I don't feel like I'm articulating this well, but I'm going to go ahead and post this comment in the interest of provoking discussion :-)

@HotelDon
Copy link

@HotelDon HotelDon commented Sep 19, 2016

@handrews
So, for me, my use cases sit entirely in the interactions during validation category, so that's what I'm going to speak to.

I think I have a basic understanding of what you're trying to say - you'd prefer a solution where JSON Schema validators use $data (or some other similar feature) as a pointer whose value is checked during validation, instead of recompiling the schema before validation even begins, where it inserts the value of those pointers directly into the schema.

Would it be possible to modify the proposal to remove the portions about modifying the schema directly, and include language elsewhere in v6+ that validation shouldn't modify schemas for any reason? Then, $data would be functionally identical to the way it works now, but doesn't encourage "bad behavior" among the various JSON Schema validators.

@handrews
Copy link
Member Author

@handrews handrews commented Sep 19, 2016

@HotelDon this is what I mean by "I'm not articulating this well"

It's not so much the reading/loading of the data (which likely has to be done lazily because of $refs), it's what you can do with it after it is loaded. Although thinking about this more I may be OK with it.

The way the proposal is written, with a list of allowed properties that trails off with "more..." left me very concerned about the scope. However what's not explicitly called out, but I think would be better than listing the fields, is that all of the fields proposed for $data take a literal value, and not a schema. We really need to make sure that data is never interpreted as a schema- that's a security nightmare- just use it to shove in links to all of your favorite malware sites!

But I think the intent here is that $data is only used to load data in place of a literal value. I can get behind that.

I still think it is valuable to separate the hypermedia template resolution cases out and use vars as specified in issue #52 for that. Since the values in vars are already assumed to be pointers into the instances, so requiring them to be little{"$data": "/pointer"} objects instead of just pointers is overkill.

@HotelDon
Copy link

@HotelDon HotelDon commented Sep 19, 2016

I had never considered the possibility of someone trying to load schema's with $data, so I guess that is what got me confused.

So maybe fix it to sound more like this:

This keyword would be available:

  • inside any schema
  • contained in an object ({"$data": ...}) for most schema properties that accept literal values. For example:
    • minimum/maximum
    • minItems/maxItems
    • pattern
    • enum
    • etc...

I'm still having a hard time wrapping my head around the hypermedia/LDO portion of this proposal, so I don't have much of an opinion on it. It might be helpful if @geraintluff chimed in to defend his original proposal a bit, assuming he still has any interest in doing so.

@epoberezkin
Copy link
Member

@epoberezkin epoberezkin commented Oct 28, 2016

My 2¢: people seem to use it a lot with Ajv, judging by the questions. So it must be useful.

I think relative JSON pointer should be extended to allow navigating array items (see #115)

@handrews
Copy link
Member Author

@handrews handrews commented Oct 31, 2016

I've become more receptive of this proposal while working with some of the more difficult hyper-schema problems such as discussed in #108

@handrews handrews changed the title v6 validation and hyper-schema: $data validation and hyper-schema: $data Nov 24, 2016
@awwright
Copy link
Member

@awwright awwright commented Dec 3, 2016

I'm solidly of the opinion that checking data consistency is solidly out of the scope of JSON Schema. Although it's certainly an option for validators that do want to offer the feature.

And if it's a popular feature then... maybe it's something we have to look into, perhaps as a separate document though.

@handrews
Copy link
Member Author

@handrews handrews commented Dec 3, 2016

@awwright $data has important uses in hyper-schema whether it is available in general validation or not.

@Relequestual
Copy link
Member

@Relequestual Relequestual commented Jan 5, 2017

I can see this could be useful. A few clear usecases might be helpful if anyone has the time or inclination.

@handrews
Copy link
Member Author

@handrews handrews commented Jan 5, 2017

@Relequestual I'd like to see whether PR #179 is accepted or not before digging into use cases here. If it is accepted, that will clarify how to present the future use cases. If it is not, I'll need to come up with a different approach anyway.

@handrews
Copy link
Member Author

@handrews handrews commented Aug 30, 2017

I'm moving this out of draft-07/wright-*-02. It is a huge topic that has seen no progress and almost no real discussion in the past year. And there is no clear advocate with time available to move it forward.

@handrews handrews modified the milestones: draft-future, draft-07 (wright-*-02) Aug 30, 2017
@handrews handrews removed the hypermedia label Sep 2, 2017
@handrews handrews changed the title validation and hyper-schema: $data $data Sep 2, 2017
@handrews
Copy link
Member Author

@handrews handrews commented Sep 26, 2017

Random thought: Would it make sense to define $data as part of a separate vocabulary for data interaction? (for lack of a better term)

If we went this route, it would also add to the use cases for #314 for understanding multiple vocabularies in use simultaneously.

@epoberezkin
Copy link
Member

@epoberezkin epoberezkin commented Oct 6, 2017

Separate vocabulary seems overkill...

@handrews
Copy link
Member Author

@handrews handrews commented May 25, 2018

Bringing over some commentary from #541 and #549 since this is the oldest issue in this area of functionality:

I'm proposing that all of the possible $data-tagged features (including but not limited to loading instance data into the schema, loading external data into the schema, and asserting relationships among instance locations) be worked on as a new vocabulary. All of the proposals in this area add substantial complexity, and also do not need to change the existing core and validation specification concepts. With vocabulary support being added in draft-08, having one or more vocabularies for this area would allow it to develop independent from core and validation, which we hope are approaching a final draft. I expect this area would be pretty active with new ideas and feedback, which would delay finalization significantly if added to core or validation (and it's unrelated to hyper-schema, which has its own mechanisms for working with instance data in URI Templates).

@equaeghe
Copy link

@equaeghe equaeghe commented Dec 6, 2018

@handrews Nitpicking: In your example

        "minimum": {"$data": "1/smaller"},
        "exclusiveMinimum": true

should be replaced by

        "exclusiveMinimum": {"$data": "1/smaller"},

lest people get confused.

@handrews
Copy link
Member Author

@handrews handrews commented Dec 6, 2018

@equaeghe that example was copied from the original proposal, and both when written and at the time I filed it here was correct as that is the draft-04 and earlier syntax.

garlick added a commit to garlick/flux-core that referenced this issue Jan 9, 2019
Add a json-schema (https://json-schema.org/) schema
for jobspec, as defined in
- Flux RFC 14 json content rules, examples, and text
- Valid and invalid sharness test input
- (some) peeking at jobspec.cpp

Some caveats:
- "complex count" cannot be checked for max < min
  (see json-schema-org/json-schema-spec#51)
garlick added a commit to garlick/flux-core that referenced this issue Jan 9, 2019
Add a json-schema (https://json-schema.org/) schema
for jobspec, as defined in
- Flux RFC 14 json content rules, examples, and text
- Valid and invalid sharness test input
- (some) peeking at jobspec.cpp

Some caveats:
- "complex count" cannot be checked for max < min
  (see json-schema-org/json-schema-spec#51)
garlick added a commit to garlick/flux-core that referenced this issue Jan 9, 2019
Add a json-schema (https://json-schema.org/) schema
for jobspec, as defined in
- Flux RFC 14 json content rules, examples, and text
- Valid and invalid sharness test input
- (some) peeking at jobspec.cpp

Some caveats:
- "complex count" cannot be checked for max < min
  (see json-schema-org/json-schema-spec#51)
@askirmas
Copy link

@askirmas askirmas commented Jun 28, 2019

Why does this and many similar 'features' couldn't be done with $ref?

@handrews
Copy link
Member Author

@handrews handrews commented Jun 28, 2019

@askirmas $ref connects schemas to schemas. Meaning you need to know what other schema you are loading at the time that you write the schema containing the $ref. $data loads information from the instance into the schema. So it is (potentially) different for every instance and cannot be determined before the instance is provided. These are fundamentally different behaviors.

@askirmas
Copy link

@askirmas askirmas commented Jun 29, 2019

@handrews

$ref connects schemas to schemas
$data loads information from the instance

For validator both are load some data on demand when pointer see this key. Difference is only in context

{
  "$schema": "http://json-schema.org/draft-07/schema",
  "$ref": "#/definitions/type-object",
  "properties": {
    "type": {
      "$ref": "#/definitions/type-object",
      "$data": "Deprecated? It is description layer",
      "default": {
        "$data": "#/definitions/type-object",
        "$ref": "Deprecated? It is data layer. The only thing - if I need exactly $ref property, not as an instruction to validator",
        "\\$data": "But same question for $data and any other",
        "$$ref": "Some escaping should be used in any case, but it is not $ref vs $data question" 
      }
    }
  },
  "definitions": {
    "type-object": {
      "type": "object"
    }
  }
}
@johandorland
Copy link
Collaborator

@johandorland johandorland commented Jun 29, 2019

The context is of huge importance. Most validators split validation into two parts, parsing the schema and validating a document with this parsed schema. With $data you are missing some values when parsing the schema as their content is part of the document. It's not extremely difficult to implement in a validator, as it indeed uses the same reference mechanics as $ref, but it does require a fundamental change in the way schemas are processed.

@askirmas
Copy link

@askirmas askirmas commented Jun 29, 2019

Most validators

Does another way exists? I expect all validators

The context is of huge importance

And it's obvious for validator. That's why I see bingo

@askirmas
Copy link

@askirmas askirmas commented Jun 29, 2019

Also I see huge question for $data and not so critical for $ref - merge strategy. For $ref it is replace as Object.assign({}, $ref, holder) but for data I'd prefer to use all the variety (and 1 option per project depend on it's specification).
Ajv propose $merge and $patch but my opinion same like for $data - it is just

entities should not be multiplied unnecessarily

{
  "$schema": "http://json-schema.org/draft-07/schema",
  "properties": {
    "$data": {
      "oneOf": [
        {"type": "string", "format": "uri-reference"},
        {
          "type": "object",
          "required": ["uri"],
          "properties": {
            "uri": {
              "type": "string", "format": "uri-reference"
            },
            "strategy": {
              "type": "string",
              "default": "replace",
              "enum": ["replace", "merge", "replace_recursive", "merge_recursive"]
            },
            "mergeRecursiveTypeConflicts": {
              "description": "For next step of processing $data will be instruction. Let's say like $ref/type. So merge ['boolean', 'integer'] with 'string' can produce ['boolean', 'integer', 'string'] that is OOP extending"
            }
          }
        }
      ]     
    }
  }
}
@handrews
Copy link
Member Author

@handrews handrews commented Jan 15, 2020

See @awwright 's Scope of JSON Schema Validation document for why this is unlikely to be taken up.

@awwright
Copy link
Member

@awwright awwright commented Feb 25, 2020

Can we close this out since the general consensus is against this?

@handrews
Copy link
Member Author

@handrews handrews commented Feb 25, 2020

@awwright yes, I think it's time. The most relevant current discussions are in #855 and #549.

@yww325
Copy link

@yww325 yww325 commented Jul 26, 2020

See @awwright 's Scope of JSON Schema Validation document for why this is unlikely to be taken up.

I can't understand why the scope document is against this proposal.
in the scope it says: Validating data consistency may involve: Scanning the rest of the document for a referenced value
Isn't that all this proposal about?

@awwright
Copy link
Member

@awwright awwright commented Jul 26, 2020

@yww325 Find this paragraph:

Many applications simply wish to test that an ID is defined in another part of the same document. Even though this case would be compute-bound, it is still outside the scope of JSON Schema validation for several reasons...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.