Need support of a local description of a property using $ref

[nitmws]

I'm working for the organisation IPTC creating and maintaining standards for metadata about media assets - and the metadata should be expressed using different formats, one of them JSON. That's the background for this use case:

Use of JSON-Schema: schema objects are defined for a reuse by properties under definitions. They represent generic things like a string with format uri or an object with (sub-)properties which are required by many properties. Then properties are specified with a $ref pointing at the required schema object.

Use of description for documentation: a key feature of these standards is to include a definition of all properties - expressed by a description in the JSON Schema. All properties must have their specific definition, all (sub-)properties in a referenced object must have their specific definition.

The issue I encountered in JSON Schema v 5 is: The Core specification defines in section 7 All other properties in a "$ref" object MUST be ignored. First: this language is misleading as it apparently means that all local keywords of a property using $ref should be ignored - and nothing in the referenced object should be ignored.
But the key issue regarding documentation is: if a property using $ref has a local description and/or title they are ignored and description and/or title of the referenced objects replace them.
This is implemented by vendors of JSON Schema editors.

Example:

{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"description": "Example of the local/global description issue - created by Michael Steidl on 2017-01-31",
	"type": "object",
	"properties": {
		"uriProperty1": {
			"description": "Local description of the uriProperty1",
			"title": "Local title of uriProperty1",
			"$ref": "#/definitions/aURI"
		},
		"uriProperty2": {
			"description": "Local description of the uriProperty2",
			"title": "Local title of uriProperty2",
			"$ref": "#/definitions/aURI"
		},
		"aLocalUriProperty": {
			"title": "Title of aLocalUriProperty",
			"description": "Description of aLocalUriProperty",
			"type": "string",
			"format": "uri"
		}
	},
	"additionalProperties": false,
	"definitions": {
		"anObject": {
			"title": "Title of anObject",
			"description": "Description of anObject",
			"type": "object",
			"properties": {
				"prop1": {
					"type": "string"
				},
				"prop2": {
					"type": "number"
				}
			},
			"additionalProperties": false
		},
		"aURI": {
			"title": "Title of aUri",
			"description": "Description of aURI",
			"type": "string",
			"format": "uri"
		}
	}
}

Result regarding the definitions of properties:

• the properties uriProperty1 and uriProperty2 have to use as description the one of the aURI schema object. A local description like "Local description of the uriProperty1" is not shown by JSON Schema editor software complying with the JSON Schema specification in a "Desing View", only looking at the pure text version shows these description - and by the specs they do not exist. I guess this is confusing.
• aURI is a generic object and its generic description cannot replace the specific semantics of uriProperty1 and uriProperty2. This is in fact a violation of the need of a party defining a schema to define/describe each property individually.
• at least the JSON Schema editor I'm using is not indicating this import of the description of the referenced object. This way it may surprise a user that editing the description of such a property results in a change of the description of all properties referencing the same schema object.

Requested change of the JSON Schema specification section "Schema references with '$ref'":

• All other properties in a "$ref" object MUST be ignored. should be removed
• This language should be added: If a schema reference is used by a property of a schema the values of the keywords description and title of this property should override values of the referenced schema, all other keywords of this property should be ignored. If a schema reference is not used for a property all keywords of the referenced schema should be used.

Thanks for considering this change request.

Note on using description and title in a JSON Schema:
I've read #175. The IPTC experience of the use of JSON or XML Schemas by developers is that they look at what an JSON or XML editor provides as inline documentation and it is very hard to convince them having to look at an external documentation. To protect a key feature of a standard, the same use by all parties using it, it is a must for a standardisation body to include at least a basic definition of properties in any kind of file defining a schema. As W3C's XML Schema supports local descriptions of global elements or data types there is a need to be able to do the same for JSON Schema.

[awwright]

@nitmws Can you describe a little bit how this compares to the canonical solution of using "allOf"? For example:

{ "allOf": [
	{"$ref": "#/definitions/aURI"},
	{ "description": "Local description of the uriProperty1",
	"title": "Local title of uriProperty1" }
] }

[handrews]

@awwright this is almost identical to #98. The only difference is that #98 introduces the "$use" keyword to explicitly indicate that local annotation keywords (in the "with" clause of "$use") should override the "$ref"'d keywords (in the "source" clause), while this proposal just automatically applies that behavior to annotation keywords at the same level as any "$ref".

Honestly, I like this better. But if you want use cases, there is an enormous amount of discussion in #98 already. Including from other people outside of the usual suspects that comment on every issue here.

[nitmws]

Thanks for both suggestions, I'll have a look into #98.
Simple question: where is the use of allOf described? I can only find its definition in the schema validation document and that doesn't help to get the proper use.

[handrews]

@nitmws be warned that #98 is rambling and has a lot of digressions. Maybe skim it but don't slog through it all. I will likely close it in favor of this one.

[handrews]

@nitmws "allOf" is not described to do what you want. If you have "description" and "title" both inside and outside of your referenced schema (the one at "#/definitions/aURI" in @awwright's example), the JSON Schema spec provides no guidance on whether one take precedence over the other, or whether both should be used. I don't know why @awwright considers this a solution to the problem.

Of course a documentation-generating application could just pick one of those behaviors and go with it, but that takes the expression of intent out of the hands of schema authors which I believe is where it belongs.

[awwright]

@nitmws It's been our solution before, and it follows from the definition of the keywords. Schemas always add restrictions to an instance, and keywords in a schema are typically independent of each other.

Suppose we have a record that specifies the outcome of a game and which user won against who. A schema "identifier.json" specifies what a database identifier must look like. Suppose "identifier.json" looks like { type: "string", maxLength: 20 }.

And suppose the match record looks like:

{ type: "object",
properties: {
   winner:
      { allOf: [
         { $ref: "identifier.json" },
         { description: "The winner of the match", pattern: "^USER-" }
      ] },
   loser:
      { allOf: [
         { $ref: "identifier.json" },
         { description: "The loser of the match", pattern: "^USER-" }
      ] },
   }
}

Which is the same as this:

{ type: "object",
properties: {
   winner: {  type: "string", maxLength: 20, description: "The winner of the match", pattern: "^USER-" }
   loser: {  type: "string", maxLength: 20, description: "The loser of the match", pattern: "^USER-" }
   }
}

The "description" is describing the semantics of the instance in prose -- and specifically saying it's not just a string, it's a string that identifies a winner (or a loser). You can't put a winner ID in the loser instance or vice-versa. So I think the use of "allOf" is warranted here.

This doesn't change if we add a "description" keyword to "identifier.json" too, both of those keywords will still apply to describe the instance. It's just messy to describe because it can only be done with "allOf" (or with a hypothetical superset of JSON/JSON Schema where repeated keywords are allowed).

[handrews]

@awwright

It's been our solution before

who is this "our" of which you speak?

both of those keywords will still apply to describe the instance

that is often undesirable, as is discussed at length in #98 and is illustrated in @nitmws's example in the initial comment.

[nitmws]

I try to match comments above against the JSON Schema draft 5:

• I can't find a definition for allOf which merges keywords of two schema objects as outlined by @awwright - the only allOf definition I can find is in the validation document: http://json-schema.org/latest/json-schema-validation.html#rfc.section.5.22
• Assuming that this allOf kind of merge means for a property using allOf its keywords are a union of the keywords of the objects in the allOf array: what is the rule if two (or three or more) objects in the array have the same keyword, e.g. description ? ...
• ... "both of those keywords will still apply" would cause confusion about the semantics.

A quick practical check: I've created a JSON Schema with this allOf design with my JSON Schema editor: it shows both objects as separated objects, with their individual descriptions. A user of our JSON Schema might scratch his head what description is the right one.

In general I feel this $use of #98 comes very close to my raised need - without having studied in detail the long list of pros and cons. In other words: if changing the rule for processing the keywords of $ref is not supported then it seems reasonable to create a new kind of schema object reference covering the need for "local annotations".

[Relequestual]

I think it's clear that we are getting multiple people calling for the same requirements, with several having identical use cases. I think that's a good call. The need has presented itself. As such, I'm going to increase priority of #98 to high.

[Relequestual]

@nitmws I'm going to close this issue in favour of #98. Super thanks for clearly providing your use case and giving us reason to bump priority of #98. Suggest you subscribe to that issue to be kept up to date on any progress =]

[awwright]

@handrews I'm not sure how I see this behavior is undesirable.

@nitmws The definition for "allOf" should say everything you need to know:

An instance validates successfully against this keyword if it validates successfully against all schemas defined by this keyword's value.

So if multiple schemas under "allOf" have a "description" keyword, all of them apply. It is not a validation keyword, so it won't ever cause validation to fail, however it should be saying something true about the instance nonetheless.

A user of our JSON Schema might scratch his head what description is the right one.

Both should accurately describe what the instance is doing (one in general terms, one more specifically). Although it will pass validation, two "description" keywords shouldn't provide contradictory definitions.

Which implementation are you testing this against?

[awwright]

@Relequestual I know we're really eager to get these issues closed, but this is a design issue so we either need consensus, or the author needs to be satisfied that the issue is resolved in some fashion (i.e. covered under another issue).

@nitmws Feel free to let us know and/or close the issue if your questions are answered, or otherwise resolved by another issue.

[handrews]

@awwright it was closed because it's a duplicate of #98, which discusses a superset of this.

The fact that you don't see why anyone else wants this is much less important than the numerous people (not even counting the usual suspects) who keep asking for this. Not just individuals on their own but individuals representing larger projects (public or private) as well.

"allOf" does not do what any of us are asking for. As you say, it means all of the constituent parts apply. That is not the goal. The goal is to selectively replace the value of non-validation keywords.

I agree with @Relequestual that this should be consolidated with #98. @nitmws if you feel that #98 does not cover the necessary possibilities could you either add to that and close this one, or explain here why you feel they need to stay separate? (don't worry about examining every comment- if you end up duplicating something in a new comment that's fine).

[nitmws]

As commented earlier: $use in #98 covers the need raised in this issue.

• it is possible to reference a schema object by source
• it is possible to override the title and description properties of the referenced schema object by "local" properties (and their values). To be precise: the title and description of the referenced schema object should not belong to, should not be visible with a property referencing it.

That's what I requested and this is covered.

(... and I have to add that I'll be away from the internet communication in the next days ...)

[dskvr]

I'm sorry to revive this conversation, that surely everyone here is tired of and would like to resolve. After going down the rabbit hole of the very specific problem detailed in the title of this issue, I ended up running in circles through countless issues and then ended at a conclusion that doesn't seem to actually solve the issue referenced in the title.

This is an extremely common documentation problem, and what I've walked away with is this "Don't reuse schemas if you want them to have local documentation."

Can anybody point me somewhere specific where the issue of overriding/merging/using a simplefield (like description) is discussed, or something I can continually reference to monitor it's progress?

Are any of the below true-ish?

1. Is possible with an existing draft.
2. Is in a proposed draft specification
3. Is in a draft published somewhere by somebody

Please note: I am pretty sure I've read every issue on this topic between OAS and JSON-Schema discussions

[Makumasa]

I'm sorry to revive this conversation, that surely everyone here is tired of and would like to resolve. After going down the rabbit hole of the very specific problem detailed in the title of this issue, I ended up running in circles through countless issues and then ended at a conclusion that doesn't seem to actually solve the issue referenced in the title.

This is an extremely common documentation problem, and what I've walked away with is this "Don't reuse schemas if you want them to have local documentation."

Can anybody point me somewhere specific where the issue of overriding/merging/using a simplefield (like description) is discussed, or something I can continually reference to monitor it's progress?

Are any of the below true-ish?

1. Is possible with an existing draft.

2. Is in a proposed draft specification

3. Is in a draft published somewhere by somebody

Please note: I am pretty sure I've read every issue on this topic between OAS and JSON-Schema discussions

I'm in the exact same boat, haha