Output Formatting #679

gregsdennis · 2018-11-18T22:46:02Z

Here is my first go at the output formatting. Happy to update as necessary. It was quite difficult coming up with examples that showed the flexibility of the output while also being concise.

~~The .md file should be removed before merging. It's just a composition file.~~

This PR resolves #643 and relates to #396.

jsonschema-core.xml

handrews

This looks great!

I added some tiny comments about clarity, or about wording so that it is a bit more consistent with other parts of the spec. But they are just tiny tweaks, not major changes.

jsonschema-core.xml

handrews · 2018-12-05T23:18:33Z

jsonschema-core.xml

+                            structure that matches the exact structure of the schema.
+                        </t>
+                    </list>
+                    Implementations SHOULD specify which levels of validation they support.


Do you need to implement all levels up to and including the most complex/verbose one? Or could an implementation support only the Verbose option? We should be clear about that in this section.

I don't think that an "up-to" requirement is necessary. Implementations should be free to support any of the formats independently. Maybe "levels" is wrong (I don't want to over-use "format," though)?

Personally, I found it easier to simply do the verbose format then mutate the output to get the others.

Okay, updated. Please let me know if that works for you.

As I originally described it an updated version would say that an implementation:
MUST provide flag
SHOULD provide basic
RECOMMENDED to provide detailed and
OPTIONALLY provide verbose

If the verbose format is likely to be easier to implement I would prefer to allow any subset of formats be implemented. @Anthropic do you feel strongly about this?

Relequestual

Super thanks for all your hard work on this, and sorry it's taken such an age for me to get to review it.
I think one or two of my comments are asking if changes are required, while the other are questions or comments to make sure I understand this right.

standardized-output-composition.md

Relequestual · 2018-12-06T11:41:22Z

standardized-output-composition.md

+
+- `valid` - a boolean value indicating the overall validation success or failure
+- `errors` - the collection of errors or annotations produced by a failed validation
+- `annotations` - the collection of errors or annotations produced by a successful validation


In addition, why would errors include annotations (and the reverse)?

Only the verbose output would do that. The primary cases to consider are the applicator keywords (especially not). For example, an allOf could contain 1 failed subschema, yielding an overall failure, but the verbose output would list all of the results. This would need to be inside errors since it failed.

I agree it's a little counterintuitive, but I think it works better for automated processing of all the results are together rather than having both errors and annotations.

OK, so my follow up question here is:
If a subschema of not fails, obviously the not means the parent schema asserts valid is true. My question is then what happens to the annotations in the subschema? They only trickle up for schemas for which validation is a positive assertion, no? /ccing @handrews on this one =]

Correct, you only return annotations if the assertion result is valid. Otherwise they are dropped from the final result set (although showing what they were before they were dropped in the more complex output format is fine and useful.

standardized-output-composition.md

gregsdennis · 2018-12-06T19:12:36Z

Some of this has changed in the XML version based on @handrews's comments. It's definitely worth reading that over this file. I can probably go ahead and delete it.

gregsdennis · 2018-12-06T20:41:18Z

jsonschema-core.xml

+                    <t>
+                        Because this output structure can be quite large, a smaller example is given
+                        here for brevity.  The full output structure of the example above can be found
+                        [here](standardized-output-verbose.json)


@handrews how do I do external links in this XML file? Probably not with Markdown as I have here.

@gregsdennis I think you want <eref>. The full XML RFC reference is https://tools.ietf.org/html/rfc7749 (yes, that RFC is marked obsolete, but it's the right one- the xml2rfc tool still doesn't support the new RFC for xml2rfc, ironically).

gregsdennis · 2018-12-06T20:41:29Z

jsonschema-core.xml

+                <section title="Output validation schema">
+                    <t>
+                        For convenience, a JSON Schema has been provided to validate output generated
+                        by implementations.  It can be found [here](schema-output.json).


Another external link.

jsonschema-core.xml

standardized-output-composition.md

Relequestual

Happy with the changes. Just one outstanding question regarding the JSON Pointer for paths is unresolveable when including by-reference applicators. At least, by means of standard resolvers.

gregsdennis · 2018-12-10T20:29:40Z

@handrews @Relequestual I've updated per all of your comments (thank you). I think the only thing that remains is to ensure that the two new files are deployed with the HTML and TXT files. I looked at the makefile, but couldn't understand it enough to be confident in editing it.

I figure it'd be the same as how the meta-schemas are deployed, but I didn't see where that was done.

handrews · 2018-12-11T07:10:33Z

Thanks @gregsdennis I will get to this in the next day or two. Really :-P

The meta-schema (and XML) deployment process is manual and somewhat absurd. I don't even know what the makefile does other than generate the .html and .txt. I think we've gotten a PR or two around it but it was always someone wanting to do something else with them instead / in addition. Given that they get deployed roughly twice a year, by me, it's incredibly low on my priority list to change.

handrews · 2018-12-14T08:19:49Z

This is looking great! I think all of my concerns have been addressed.

gregsdennis · 2018-12-18T20:13:49Z

@Anthropic I've updated the language around format support requirements. Please have a look.

handrews

Sorry for the delay, this is the first day in nearly a week that I've felt well enough to think clearly.

handrews · 2018-12-24T00:59:28Z

jsonschema-core.xml

-                    to support the "detailed" format, it is not also required to support the "basic"
-                    or "verbose" formats, and vice versa.  Implementations SHOULD specify in their
-                    documentation which formats of validation they support.
+                    Implementations MUST provide support for the "flag" format and SHOULD provide the


Early on, we were going to avoid MUST requirements, on the grounds that implementations can take many forms, including those with extreme memory constraints, or streaming implementations, or other things we can't anticipate where there may be a pretty compelling reason for just a boolean rather than an object. I'm a little hesitant to use MUST here, although I understand the desire for interoperability

I could support an approach where we say that implementations SHOULD support one or more of these formats, and if they do, then they MUST support flag, SHOULD support basic, etc.

That leaves some wiggle room for implementations that may be working under constraints that we cannot anticipate.

handrews · 2018-12-24T00:59:39Z

jsonschema-core.xml

-                    or "verbose" formats, and vice versa.  Implementations SHOULD specify in their
-                    documentation which formats of validation they support.
+                    Implementations MUST provide support for the "flag" format and SHOULD provide the
+                    "basic" and "detailed" formats.  The "verbose" format is OPTIONAL.  Implementations


I'd use

"flag" format, SHOULD provide the "basic" and "detailed" formats, and MAY provide the "verbose" format.

because we use MAY a lot in these documents, but rarely if ever use OPTIONAL. I don't think there's any reason, it just seems to be the convention that has developed organically.

I don't feel strongly about this, though.

Thanks. I was sure there was a way to make it smoother.

handrews

Thanks for all of the hard work @gregsdennis and everyone who participated in the discussion. This is now ready to merge!

gregsdennis added 3 commits November 19, 2018 11:28

Add 'output formatting' markdown composition as working medium

cc8957f

Add exceedingly large verbose output as separate file

2a4b28b

Add composition to spec XML

6571b06

Relequestual self-requested a review November 21, 2018 09:27

Anthropic reviewed Nov 24, 2018

View reviewed changes

jsonschema-core.xml Outdated Show resolved Hide resolved

Anthropic reviewed Nov 24, 2018

View reviewed changes

jsonschema-core.xml Outdated Show resolved Hide resolved

handrews added Type: Enhancement Priority: High core output labels Nov 26, 2018

handrews added this to the draft-08 milestone Nov 26, 2018

handrews mentioned this pull request Nov 27, 2018

Need more details of annotation collection #530

Closed

updated per @Anthropic's suggestions 👍

59f9ff0

gregsdennis mentioned this pull request Nov 30, 2018

JSON Schema 8 gregsdennis/Manatee.Json#175

Closed

handrews reviewed Dec 5, 2018

View reviewed changes

addressed most of @handrews's comments

cf7a6e0

Relequestual requested changes Dec 6, 2018

View reviewed changes

removed composition file; added output schema

fd1b00d

gregsdennis commented Dec 6, 2018

View reviewed changes

gregsdennis added 2 commits December 7, 2018 13:25

clarified requirements around implementation support

df63510

addressed some wording concerns

06cd268

Relequestual reviewed Dec 10, 2018

View reviewed changes

jsonschema-core.xml Show resolved Hide resolved

standardized-output-composition.md Outdated Show resolved Hide resolved

Relequestual reviewed Dec 10, 2018

View reviewed changes

added comment for relative pointers in output formatting

5504ffc

Relequestual approved these changes Dec 17, 2018

View reviewed changes

Updated language around format support requirements

32c624b

gregsdennis mentioned this pull request Dec 18, 2018

Define Validation spec vocabularies #697

Closed

Set support requirements per conversation in slack

fb312cc

handrews reviewed Dec 24, 2018

View reviewed changes

gregsdennis added 2 commits December 25, 2018 15:52

better wording for output

02295dc

a bit more wordsmithing around output requirements

063fbe2

handrews approved these changes Dec 28, 2018

View reviewed changes

Merge branch 'master' into draft-08-output-formatting

235c54a

handrews merged commit ba2c0b6 into json-schema-org:master Dec 28, 2018

gregsdennis mentioned this pull request Jan 7, 2019

Add output formatting tests json-schema-org/JSON-Schema-Test-Suite#247

Closed

davishmcclurg mentioned this pull request Jan 18, 2019

More Precise Error Validation for allOf, anyOf, oneOf davishmcclurg/json_schemer#1

Closed

This was referenced Mar 10, 2019

Is the ReadMe upto date ? davishmcclurg/json_schemer#24

Closed

Output formatting davishmcclurg/json_schemer#27

Closed

vearutop mentioned this pull request Apr 25, 2019

Error messages opis/json-schema#16

Closed

adjenks mentioned this pull request Mar 24, 2020

Looking at the future json-schema drafts opis/json-schema#3

Closed

karenetheridge mentioned this pull request May 26, 2020

resolve ambiguity regarding annotation inclusion in failed validation results #939

Closed

gregsdennis mentioned this pull request Oct 5, 2021

Restructuring of output formatting #973

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Output Formatting #679

Output Formatting #679

gregsdennis commented Nov 18, 2018 •

edited

Loading

handrews left a comment

handrews Dec 5, 2018

gregsdennis Dec 5, 2018 •

edited

Loading

gregsdennis Dec 7, 2018

Anthropic Dec 12, 2018

handrews Dec 14, 2018

Relequestual left a comment

Relequestual Dec 6, 2018

gregsdennis Dec 6, 2018

Relequestual Dec 10, 2018

handrews Dec 10, 2018

gregsdennis commented Dec 6, 2018

gregsdennis Dec 6, 2018

handrews Dec 7, 2018

gregsdennis Dec 6, 2018

Relequestual left a comment

gregsdennis commented Dec 10, 2018

handrews commented Dec 11, 2018

handrews commented Dec 14, 2018

gregsdennis commented Dec 18, 2018

handrews left a comment

handrews Dec 24, 2018

handrews Dec 26, 2018

handrews Dec 24, 2018

gregsdennis Dec 25, 2018

handrews left a comment

Output Formatting #679

Output Formatting #679

Conversation

gregsdennis commented Nov 18, 2018 • edited Loading

handrews left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gregsdennis Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Relequestual left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gregsdennis commented Dec 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Relequestual left a comment

Choose a reason for hiding this comment

gregsdennis commented Dec 10, 2018

handrews commented Dec 11, 2018

handrews commented Dec 14, 2018

gregsdennis commented Dec 18, 2018

handrews left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

handrews left a comment

Choose a reason for hiding this comment

gregsdennis commented Nov 18, 2018 •

edited

Loading

gregsdennis Dec 5, 2018 •

edited

Loading