json-schema: initial pass at schema and validator #313

vbatts · 2016-01-19T17:43:27Z

Conforming to https://tools.ietf.org/html/draft-zyp-json-schema-03
and http://json-schema.org/latest/json-schema-core.html

This work would ideally pair with separating the current reference source effort #276

provide validation tool to validate a configuration file
full structures
define basic primitives
account for all the pointer usage
discuss versioned schema-$ver.json

The gist of the config.json being tested with (may change periodically): https://gist.github.com/vbatts/5ac2723c9598875ffb0e

Signed-off-by: Vincent Batts vbatts@hashbangbash.com

vbatts · 2016-01-19T17:44:12Z

I am working through making usable definitions like has been done for devices

philips · 2016-01-19T20:12:31Z

looks good but makes my head hurt, oy.

vbatts · 2016-01-19T21:27:46Z

It is something to behold, but it is allowing decently flexible validation.

kofalt · 2016-01-19T23:20:57Z

Food for thought: I see you're already using schema references ("$ref"); perhaps if you broke this up into several files and combined them at the end? For example, the cpu key could be its own schema, etc.

vbatts · 2016-01-20T02:53:50Z

I wondered about that, but didn't see a way to "source" or "include", and
really don't think that just concatenated them together felt smooth

On Tue, Jan 19, 2016 at 6:20 PM, Nathaniel Kofalt notifications@github.com
wrote:

Food for thought: I see you're already using schema references ("$ref");
perhaps if you broke this up into several files and combined them at the
end? For example, the cpu key could be its own schema, etc.

—
Reply to this email directly or view it on GitHub
#313 (comment).

wking · 2016-01-20T05:04:49Z

On Tue, Jan 19, 2016 at 06:53:53PM -0800, Vincent Batts wrote:

On Tue, Jan 19, 2016 at 6:20 PM, Nathaniel Kofalt:
Food for thought: I see you're already using schema references
("$ref"); perhaps if you broke this up into several files and
combined them at the end? For example, the cpu key could be its
own schema, etc.

I wondered about that, but didn't see a way to "source" or
"include", and really don't think that just concatenated them
together felt smooth

$ref is the way to source/include other content 1.

vbatts · 2016-01-20T15:14:52Z

how quaint. let me revisit this.

wking · 2016-01-27T06:42:45Z

schema/defs.json

+      "type": "array",
+      "items": {
+        "oneOf": [ { "$ref": "#/definitions/GID" } ]
+      }


I don't think you need oneOf for arrays. Is there a reason:

"items": { "$ref": "/definitions/GID" }

doesn't work?

it is a typed array. Otherwise it would allow ["foo", 1, false]

On Thu, Mar 03, 2016 at 12:27:46PM -0800, Vincent Batts wrote:

"items": {

"oneOf": [ { "$ref": "#/definitions/GID" } ]

}

it is a typed array. Otherwise it would allow ["foo", 1, false]

I'm pretty sure that's what the $ref-only phrasing 1 means.

tetron · 2016-01-27T21:34:47Z

This is just a drive-by suggestion, but you might want to look at https://github.com/common-workflow-language/schema_salad as a possible alternative to JSON Schema.

vbatts · 2016-02-10T17:42:39Z

@tetron thanks, but that seems too niche and less of a standard even than JSON-schema is.

vbatts · 2016-03-03T20:04:13Z

rebased again

vbatts · 2016-03-03T20:05:52Z

I feel this is about ready for review too.

wking · 2016-03-03T20:12:09Z

On Thu, Mar 03, 2016 at 12:05:54PM -0800, Vincent Batts wrote:

I feel this is about ready for review too.

Any comments on my three still-applicable comments:

Unnecessary oneOf 1.
Missing indent (and a suggestion to wash through jq) 2.
Avoid the generic ‘schema/schema.json’ in favor of
‘schema/config.json’, so the schema/*.json filename matches the
config file it covers 3.

vbatts · 2016-03-03T22:06:55Z

Done. PTAL

wking · 2016-03-03T22:15:52Z

On Thu, Mar 03, 2016 at 02:06:57PM -0800, Vincent Batts wrote:

Done. PTAL

Indent and oneOfs look good with 5e656f4. I'd still like
schema/schema.json renamed to schema/config.json [1,2].

wking · 2016-03-03T22:18:32Z

schema/defs-linux.json

+                            {
+                                "$ref": "#/definitions/SyscallArg"
+                            }
+                        ]


Remaining unnecessary oneOf (see 5e656f4).

wking · 2016-03-09T16:23:22Z

schema/defs-linux.json

+            }
+        },
+        "CapabilityString": {
+            "description": "File permissions mode (typically an octal value)",


This is not the right description for CapabilityString.

And we can probably use Capability instead of CapabilityString (like we don't use MajorUInt); I don't foresee other singular capability types.

vbatts · 2016-03-09T18:24:24Z

rebased and updated

crosbymichael · 2016-03-09T19:01:36Z

LGTM

mrunalp · 2016-03-09T19:14:52Z

Can we squash or re-order the commits?

vbatts · 2016-03-09T19:29:42Z

@mrunalp ah yeah. the commits were just as I made progress. Let me clean them up.

Conforming to https://tools.ietf.org/html/draft-zyp-json-schema-03 and http://json-schema.org/latest/json-schema-core.html * Utilizes a number of JSON schema features, including 'pattern' * Defined primitives, like integers, that we'll use * Split out definitions for primitives and platform-specific * Provide a Makefile for: - "fmt" target for *.json - "validate" target for building the validation tool Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>

vbatts · 2016-03-09T19:38:28Z

@mrunalp fully squashed

vbatts · 2016-03-09T21:58:35Z

@philips @mrunalp @vishh @hqhq @tianon PTAL

mrunalp · 2016-03-10T18:26:11Z

LGTM

json-schema: initial pass at schema and validator

Also: * Update the link to Go bindings after 7bf06d5 (source and schema: differentiate with examples, 2015-12-18, opencontainers#276). * Add a reference to the JSON Schema after cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313). It's pretty clear that the Go bindings cannot be canonical on their own, because they do not define limits (e.g. the 0 through 512 range for FileMode). The JSON Schema is closer, but still does not cover everything (e.g. "a directory must exist at root.path"). Both the Go bindings and the JSON Schema could grow to cover the full spec by adding that sort of thing to comments and descriptions, but that's not how things seem to be working now. Signed-off-by: W. Trevor King <wking@tremily.us>

Validation is partly covered by cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313) and subequent JSON Schema work. The remainder of these targets are handled by ocitools [1]. [1]: https://github.com/opencontainers/ocitools Signed-off-by: W. Trevor King <wking@tremily.us>

# digest/hashing target Most of this has spun off with [1], and I haven't heard of anyone talking about verifying the on-disk filesystem in a while. My personal take is on-disk verification doesn't add much over serialized verification unless you have a local attacker (or unreliable disk), and you'll need some careful threat modeling if you want to do anything productive about the local attacker case. For some more on-disk verification discussion, see the thread starting with [2]. # distributable-format target This spun off with [1]. # lifecycle target I think this is resolved since 7713efc (Add lifecycle for containers, 2015-10-22, opencontainers#231), which was committed on the same day as the ROADMAP entry (4859f6d, Add initial roadmap, 2015-10-22, opencontainers#230). # container-action target Addressed by 7117ede (Expand on the definition of our ops, 2015-10-13, opencontainers#225), although there has been additional discussion in a7a366b (Remove exec from required runtime functionalities, 2016-04-19, opencontainers#388) and 0430aaf (Split create and start, 2016-04-01, opencontainers#384). # validation and testing targets Validation is partly covered by cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313) and subequent JSON Schema work. The remainder of these targets are handled by ocitools [3]. # printable/compiled-spec target The bulk of this was addressed by 4ee036f (*: printable documents, 2015-12-09, opencontainers#263). Any remaining polishing of that workflow seems like a GitHub-issue thing and not a ROADMAP thing. And publishing these to opencontainers.org certainly seems like it's outside the scope of this repository (although I think that such publishing is a good idea). [1]: https://github.com/opencontainers/image-spec [2]: https://groups.google.com/a/opencontainers.org/d/msg/dev/xo4SQ92aWJ8/NHpSQ19KCAAJ Subject: OCI Bundle Digests Summary Date: Wed, 14 Oct 2015 17:09:15 +0000 Message-ID: <CAD2oYtN-9yLLhG_STO3F1h58Bn5QovK+u3wOBa=t+7TQi-hP1Q@mail.gmail.com> [3]: https://github.com/opencontainers/ocitools Signed-off-by: W. Trevor King <wking@tremily.us>

The JSON Schema requirement dates back to cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313), but the property has been explicitly optional in the Markdown spec since 7ac41c6 (config.md: reformat into a standard style, 2015-06-30). Signed-off-by: W. Trevor King <wking@tremily.us>

Also: * Update the link to Go bindings after 7bf06d5 (source and schema: differentiate with examples, 2015-12-18, opencontainers#276). * Add a reference to the JSON Schema after cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313). It's pretty clear that the Go bindings cannot be canonical on their own, because they do not define limits (e.g. the 0 through 512 range for FileMode). The JSON Schema is closer, but still does not cover everything (e.g. "a directory must exist at root.path"). Both the Go bindings and the JSON Schema could grow to cover the full spec by adding that sort of thing to comments and descriptions, but that's not how things seem to be working now. Signed-off-by: W. Trevor King <wking@tremily.us>

# digest/hashing target Most of this has spun off with [1], and I haven't heard of anyone talking about verifying the on-disk filesystem in a while. My personal take is on-disk verification doesn't add much over serialized verification unless you have a local attacker (or unreliable disk), and you'll need some careful threat modeling if you want to do anything productive about the local attacker case. For some more on-disk verification discussion, see the thread starting with [2]. # distributable-format target This spun off with [1]. # lifecycle target I think this is resolved since 7713efc (Add lifecycle for containers, 2015-10-22, opencontainers#231), which was committed on the same day as the ROADMAP entry (4859f6d, Add initial roadmap, 2015-10-22, opencontainers#230). # container-action target Addressed by 7117ede (Expand on the definition of our ops, 2015-10-13, opencontainers#225), although there has been additional discussion in a7a366b (Remove exec from required runtime functionalities, 2016-04-19, opencontainers#388) and 0430aaf (Split create and start, 2016-04-01, opencontainers#384). # validation and testing targets Validation is partly covered by cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313) and subequent JSON Schema work. The remainder of these targets are handled by ocitools [3]. # printable/compiled-spec target The bulk of this was addressed by 4ee036f (*: printable documents, 2015-12-09, opencontainers#263). Any remaining polishing of that workflow seems like a GitHub-issue thing and not a ROADMAP thing. And publishing these to opencontainers.org certainly seems like it's outside the scope of this repository (although I think that such publishing is a good idea). [1]: https://github.com/opencontainers/image-spec [2]: https://groups.google.com/a/opencontainers.org/d/msg/dev/xo4SQ92aWJ8/NHpSQ19KCAAJ Subject: OCI Bundle Digests Summary Date: Wed, 14 Oct 2015 17:09:15 +0000 Message-ID: <CAD2oYtN-9yLLhG_STO3F1h58Bn5QovK+u3wOBa=t+7TQi-hP1Q@mail.gmail.com> [3]: https://github.com/opencontainers/ocitools Signed-off-by: W. Trevor King <wking@tremily.us>

The JSON Schema requirement dates back to cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313), but the property has been explicitly optional in the Markdown spec since 7ac41c6 (config.md: reformat into a standard style, 2015-06-30). Signed-off-by: W. Trevor King <wking@tremily.us>

vbatts mentioned this pull request Jan 19, 2016

[Tracker] Platform/Compiler agnostic schema #166

Closed

vbatts force-pushed the json-schema branch 3 times, most recently from bda07e0 to c5521a5 Compare January 19, 2016 21:26

vbatts changed the title ~~json-schema: initial pass at schema and validator~~ [WIP] json-schema: initial pass at schema and validator Jan 19, 2016

vbatts force-pushed the json-schema branch from 15f8917 to 449073a Compare January 25, 2016 19:12

vbatts mentioned this pull request Jan 26, 2016

config: Single, unified config file #284

Merged

wking reviewed Jan 27, 2016
View reviewed changes

vbatts force-pushed the json-schema branch from 3d58754 to 8daa770 Compare March 3, 2016 20:01

wking reviewed Mar 3, 2016
View reviewed changes

schema/defs-linux.json

{

"$ref": "#/definitions/SyscallArg"

}

]

Copy link

Contributor

wking Mar 3, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining unnecessary oneOf (see 5e656f4).

wking reviewed Mar 9, 2016
View reviewed changes

vbatts force-pushed the json-schema branch from 0bc8e19 to e974600 Compare March 9, 2016 18:22

vbatts changed the title ~~[WIP] json-schema: initial pass at schema and validator~~ json-schema: initial pass at schema and validator Mar 9, 2016

crosbymichael added this to the v0.4.0 milestone Mar 9, 2016

vbatts force-pushed the json-schema branch from e974600 to 084ff67 Compare March 9, 2016 19:22

vbatts force-pushed the json-schema branch from 084ff67 to cdcabde Compare March 9, 2016 19:38

vbatts mentioned this pull request Mar 9, 2016

[Tracker] Support standard validation and defaulting #219

Closed

mrunalp pushed a commit that referenced this pull request Mar 10, 2016

Merge pull request #313 from vbatts/json-schema

9b19e76

json-schema: initial pass at schema and validator

mrunalp merged commit 9b19e76 into opencontainers:master Mar 10, 2016

wking mentioned this pull request Mar 10, 2016

config: Make Markdown canonical #342

Merged

vbatts deleted the json-schema branch April 12, 2016 03:50

wking mentioned this pull request Jun 2, 2016

schema/state.json: Add a JSON Schema for the state JSON #481

Merged

wking mentioned this pull request Jun 6, 2016

schema/schema.json: Make 'hostname' optional #491

Merged

thaJeztah mentioned this pull request Jul 10, 2020

JSON Schema definitions of 'int64' and 'uint64' types have ranges not representable in 64 bits #1051

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

json-schema: initial pass at schema and validator #313

json-schema: initial pass at schema and validator #313

vbatts commented Jan 19, 2016

vbatts commented Jan 19, 2016

philips commented Jan 19, 2016

vbatts commented Jan 19, 2016

kofalt commented Jan 19, 2016

vbatts commented Jan 20, 2016

wking commented Jan 20, 2016

vbatts commented Jan 20, 2016

wking Jan 27, 2016

vbatts Mar 3, 2016

wking Mar 3, 2016

tetron commented Jan 27, 2016

vbatts commented Feb 10, 2016

vbatts commented Mar 3, 2016

vbatts commented Mar 3, 2016

wking commented Mar 3, 2016

vbatts commented Mar 3, 2016

wking commented Mar 3, 2016

wking Mar 3, 2016

wking Mar 9, 2016

vbatts Mar 9, 2016

vbatts commented Mar 9, 2016

crosbymichael commented Mar 9, 2016

mrunalp commented Mar 9, 2016

vbatts commented Mar 9, 2016

vbatts commented Mar 9, 2016

vbatts commented Mar 9, 2016

mrunalp commented Mar 10, 2016

json-schema: initial pass at schema and validator #313

json-schema: initial pass at schema and validator #313

Conversation

vbatts commented Jan 19, 2016

vbatts commented Jan 19, 2016

philips commented Jan 19, 2016

vbatts commented Jan 19, 2016

kofalt commented Jan 19, 2016

vbatts commented Jan 20, 2016

wking commented Jan 20, 2016

vbatts commented Jan 20, 2016

wking Jan 27, 2016

Choose a reason for hiding this comment

vbatts Mar 3, 2016

Choose a reason for hiding this comment

wking Mar 3, 2016

Choose a reason for hiding this comment

tetron commented Jan 27, 2016

vbatts commented Feb 10, 2016

vbatts commented Mar 3, 2016

vbatts commented Mar 3, 2016

wking commented Mar 3, 2016

vbatts commented Mar 3, 2016

wking commented Mar 3, 2016

wking Mar 3, 2016

Choose a reason for hiding this comment

wking Mar 9, 2016

Choose a reason for hiding this comment

vbatts Mar 9, 2016

Choose a reason for hiding this comment

vbatts commented Mar 9, 2016

crosbymichael commented Mar 9, 2016

mrunalp commented Mar 9, 2016

vbatts commented Mar 9, 2016

vbatts commented Mar 9, 2016

vbatts commented Mar 9, 2016

mrunalp commented Mar 10, 2016