Create Image Tools Interface

[RobDolinMS]

Describes an interface that the image-tools project could provide for image validation

Signed-off-by: Rob Dolin robdolin@microsoft.com

[wking]

On Mon, Mar 20, 2017 at 02:48:12PM -0700, Rob Dolin (MSFT) wrote: Describes an interface that the image-tools project could provide for image validation

Maybe this should be an image-tools issue/PR? I expect a certifier to be using image-tools directly, and don't see a need to specify its API independent of the implementation. For comparison, the runtime command line API in opencontainers/runtime-tools#321 is aimed at the *runtime* API, not at the runtime-tools API. For your UI with number-of-cases, failed cases, verbose listing, one approach would be to make image-tools's image validator a TAP producer. More on this in the image-tools context in opencontainers/image-tools#60. [edit: removed TAP plug in favor of a link to the existing image-tools TAP issue]

[caniszczyk]

cc: @opencontainers/image-tools-maintainers maintainers and @opencontainers/image-spec-maintainers for RFC

[caniszczyk]

@RobDolinMS is it possible for you to bring this up at the OCI Dev meeting and gather feedback from the OCI TDC?

[caniszczyk]

@RobDolinMS also any thoughts about just merging this into the README instead of a separate doc? (just keeps it easier to track)

[RobDolinMS]

@RobDolinMS is it possible for you to bring this up at the OCI Dev meeting and gather feedback from the OCI TDC?

@caniszczyk Yes, I mentioned this PR on the OCI Dev ConCall a few hours ago.

I'm also tagging @opencontainers/image-spec-maintainers and @opencontainers/image-tools-maintainers to please take a look at this DRAFT (PR #26) requested interface from the @opencontainers/certification-maintainers: https://github.com/opencontainers/certification/pull/26/files

[RobDolinMS]

@opencontainers/image-tools-maintainers - Does this proposal roughly align to how the code runs?

[wking]

On Mon, Apr 03, 2017 at 04:45:04PM -0700, Rob Dolin (MSFT) wrote: Does this proposal roughly align to how the code runs?

Current invocation is documented here [1]. [1]: https://github.com/opencontainers/image-tools/blob/master/man/oci-image-tool-validate.1.md#examples

[vbatts]

this tab has been open for a few days. I like the direction, just want to ensure it plumbs-up to expected behavior

[vbatts]

and maybe git commit of the tool?
and version of the OCI spec tested against.
And the timestamp the validation just ran.
Also, presumably this would need to output the digests of the objects tested.

With this data points in a structured output (i.e. xml, json, etc) then it could be more deterministically used for badges, etc.

[wking]

and maybe git commit of the tool?

I like this, but note that in a similar context the OCI consensus (as voiced by @duglin here) was that a consistent API for extracting the implementation version was not required for interop.

And the timestamp the validation just ran.

Does this matter? Hopefully the version of the test suite/tools and the digest(s) of the tested object would be sufficient without recourse to the validation timestamp. Higher-level tooling can always add a timestamp if it feels it would be useful.

Also, presumably this would need to output the digests of the objects tested.

This is tricky for stuff outside of CAS. For example, if validation was pointed at an image-layout via a HTTP URL, or at a directory, what digest hash do you show for complaining about “I couldn't find index.json there”?

[RobDolinMS]

Replying to comments from @vbatts and @wking:

and maybe git commit of the tool?

I was thinking that 'version of the OCI Image Tools' would be sufficient since only released versions of the tools will be relevant for image validation.

and version of the OCI spec tested against.

I didn't think of this, but it makes sense. I'll update the PR.

And the timestamp the validation just ran.
I'm not sure this is needed since a released version of the OCI Image Tool, a released version of the OCI Image Spec, and the target image should have the same output every time.

Also, presumably this would need to output the digests of the objects tested.

@vbatts Would you mind suggesting what this might look like?

[RobDolinMS]

I didn't think of [version of the OCI spec tested against], but it makes sense. I'll update the PR.

PR is updated at lines 11, 12, 20, 30, and 40. /cc @vbatts @wking

[vbatts]

for a deterministic badge of validation, I think those three things are needed/suitable to triangulate precisely _what_ was tested on _which_ artifacts, and a timestamp for freshness check.

…

On Wed, Apr 5, 2017 at 3:44 PM W. Trevor King ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In image-tools-interface.md <#26 (comment)> : > + +The below document describes an interface for the OCI Image Tools project as desired by the OCI Certification Program working group (Cert WG.) + +Goals: +* An engineer can verify a container image complies with the OCI Image Spec (https://github.com/opencontainers/image-spec/releases) + +Inputs: +* (REQUIRED): Path or reference to a container image +* (OPTIONAL): If the user would like verbose output + +Outputs: +* Number of cases run +* Number of cases passed +* Number of cases failed +* List of cases failed +* Version of OCI Image Tools and maybe git commit of the tool? *I* like this, but note that in a similar context the OCI consensus (as voiced by @duglin <https://github.com/duglin> here <http://ircbot.wl.linuxfoundation.org/meetings/opencontainers/2016/opencontainers.2016-07-20-21.03.log.html#l-75>) was that a consistent API for extracting the implementation version was not required for interop. And the timestamp the validation just ran. Does this matter? Hopefully the version of the test suite/tools and the digest(s) of the tested object would be sufficient without recourse to the validation timestamp. Higher-level tooling can always add a timestamp if it feels it would be useful. Also, presumably this would need to output the digests of the objects tested. This is tricky for stuff outside of CAS. For example, if validation was pointed at an image-layout via a HTTP URL, or at a directory, what digest hash do you show for complaining about “I couldn't find index.json there”? — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#26 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEF6XvUoWB2S2WRVy5BiA9e0dQsBijJks5rs-8ygaJpZM4MjDtp> .

[wking]

On Wed, Apr 05, 2017 at 12:48:00PM -0700, Vincent Batts wrote: … and a timestamp for freshness check.

But what is the purpose of the freshness check? I think the tooling for deprecating old versions [1] (and their certification results) should depend on the tested OCI version, not on the timestamp of the testing. [1]: https://github.com/opencontainers/certification/blame/76149742708a6eedef64c683bfb5f889d15e4b11/README.md#L89-L90

[vbatts]

Perhaps timestamp isn't a hard requirement, but rather a nice-to-have.

…

On Wed, Apr 5, 2017, 16:00 W. Trevor King ***@***.***> wrote: On Wed, Apr 05, 2017 at 12:48:00PM -0700, Vincent Batts wrote: > … and a timestamp for freshness check. But what is the purpose of the freshness check? I think the tooling for deprecating old versions [1] (and their certification results) should depend on the tested OCI version, not on the timestamp of the testing. [1]: https://github.com/opencontainers/certification/blame/76149742708a6eedef64c683bfb5f889d15e4b11/README.md#L89-L90 — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub <#26 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEF6T90CyU5Tsyxgnx3UXxg3eWWN2ACks5rs_LpgaJpZM4MjDtp> .

[wking]

Trailing whitespace on this line (and a few others):

$ git diff --check origin/master..origin/pr/26
image-tools-interface.md:6: trailing whitespace.
+* An engineer can verify a container image complies with the OCI Image Spec (https://github.com/opencontainers/image-spec/releases) 
image-tools-interface.md:11: trailing whitespace.
+* (OPTIONAL): Which version of the OCI Image Spec to validate against 
image-tools-interface.md:25: trailing whitespace.
+INPUT: 
image-tools-interface.md:28: trailing whitespace.
+OUTPUT (default): 
image-tools-interface.md:31: trailing whitespace.
+Case 12: FAILED: Image using reserved field key: https://github.com/opencontainers/image-spec/blob/master/descriptor.md#reserved 
image-tools-interface.md:32: trailing whitespace.
+Results: 
image-tools-interface.md:48: trailing whitespace.
+Results: 

[wking]

“IF” → “If”?

[stevvooe]

It might be good to look at rust's compiler error documentation for the items that might be required here for each error. As this is, it is a little hard to parse.

Let's also take a peak at Go's output:

=== RUN   TestCopyDirectory
--- PASS: TestCopyDirectory (0.00s)
=== RUN   TestSimpleDiff
--- PASS: TestSimpleDiff (0.00s)
=== RUN   TestDirectoryReplace
--- PASS: TestDirectoryReplace (0.00s)
=== RUN   TestRemoveDirectoryTree
--- PASS: TestRemoveDirectoryTree (0.00s)
=== RUN   TestFileReplace
--- PASS: TestFileReplace (0.00s)
=== RUN   TestUpdateWithSameTime
--- PASS: TestUpdateWithSameTime (0.00s)
=== RUN   TestBaseDirectoryChanges
--- PASS: TestBaseDirectoryChanges (0.00s)
PASS
ok  	github.com/containerd/containerd/fs	0.007s

Each line is marked as "starting test" and "ending test" (=== and --- respectively), the we get a parseable package name at the end. Timing is less important for validation, so we don't need both, but we need a way to differentiate a result from other information.

Something like this might work:

* TestName1: PASSED
* TestNameFailure: FAILED https://github.com/opencontainers/image-spec/blob/master/descriptor.md#reserved
... Information about failure.
* TestName2: PASSED
# Results: 23 / 24 (96%) passed

Notice that the status and result lines are marked with a leading character, leaving the other output to be expressive. There may be better characters than * for marking these lines, but we can see how we mark what is consumable by a leading character.

[cyphar]

We could also use TAP rather than making our own output format.

[caniszczyk]

Any comments here @stephenrwalli

[alban]

If distribution is out of scope of image-spec, what would a "reference" be? A https:// URL to the OCI Image Layout directory? And the software would download ${URL}/index.json and other files as required by the image layout content?