layer: Require ustar (originally from IEEE Std 1003.1-1988 but linking IEEE Std 1003.1-2013)

[wking]

The idea with a spec like this is to define behavior so that different implementations can interoperate reliably. When there is an interop problem betweem implementations A and B, it should be clear from the spec whether implementation A is broken, implementation B is broken, or the spec is insufficiently clear.

For example, pax defines g and x typeflags that aren't part of the older ustar (IEEE Std 1003.1-1988). Before this commit, if implementation A produced a layer with a g or x typeflag and implementation B died unpacking it, it was unclear whose fault it was. With this commit, it is clearly B's fault (because it does not understand the pax typeflag).

If implementation A had produces a layer with an S typeflag (which GNU uses for sparse files) and implementation B died unpacking it, it is neither party's fault, because pax explicitly makes those values implementation-defined. Interop around them is up to out of band communication between the layer author and layer consumer, and is not covered by this spec.

The previous “File Types” section listed sockets, but the pax spec has:

Attempts to archive a socket using ustar interchange format shall produce a diagnostic message.

And I see no socket entry in Go's set of typeflag constants, so I'm not sure how they were supported before.

It looks like Go has supported pax since v1.1.

Spun off from discussion in #336. I don't really care what spec we require here. Requiring ustar (IEEE Std 1003.1-1988) support would be fine (a lower bar to implement, but does not support path names with more than 100 characters, etc.). Or requiring support for a number of specifications (e.g. you have to understand both pax and GNU records). But I think we should be requiring something, otherwise interop relies on unspecified, out-of-band agreements on layer format.

[vbatts]

Trevor, this is a bit excessive. Honestly i'm inclined to just close this.
@jonboulle @stevvooe thoughts?

[wking]

On Sat, Sep 24, 2016 at 05:44:38AM -0700, Vincent Batts wrote:

Trevor, this is a bit excessive.

Would it be less excessive if I moved the bar from pax (IEEE Std
1003.1-2013) to ustar [1](IEEE Std 1003.1-1988)? Or is it the idea of
requiring compatibility with a tar-ish standard of any kind that you
find excessive?

[jonboulle]

I do like the spirit of this because going down the path of defining this all in-spec (e.g. #317, #336) is a an awful rabbit hole (inventing our own archive format), and I'd rather defer to external standards wherever possible. Unfortunately pax is not super well known or popular so I fear we risk a lot of confusion by changing the references everywhere like this. Can we still generally talk about tar but then just define more specifically what we mean by it? (most modern tar implementations support pax right?)

[wking]

On Mon, Sep 26, 2016 at 02:09:44AM -0700, Jonathan Boulle wrote:

Unfortunately pax is not super well known or popular so I fear we
risk a lot of confusion by changing the references everywhere like
this.

If we want to require pax support, I can put in links from anywhere
1833fd1 said “pax archive”. Or just drop the archive information from
most places outside of layer.md, because the fact that layers are
using tar or pax or whatever does not matter outside of generating and
unpacking layers.

Can we still generally talk about tar but then just define more
specifically what we mean by it?

If we're requiring pax support, I'd rather call it pax. It seems like
it would be more confusing to say “tar” when we meant “at least pax”.
But I agree that there is less name-recognition for pax. I've pushed
1833fd1 → 93b5819 shifting the bar from pax to ustar. I also
tightened the wording to make it clear that ustar is the minimum
required bar for unpackers, and that layer generators are free to use
other formats if they are comfortable with the posibility of an
OCI-compliant unpacker choking on them.

(most modern tar implementations support pax right?)

I don't know how many tar implementations there are, but libarchive
(obviously?) supports pax 1. Go supports pax (links in my topic
post here). Python supports pax 2. Perl's Archive::Tar supports
pax [3](and defines a socket typeflag as 8, which I haven't been able
to turn up a spec for). GNU tar has supported pax since v1.14 4.
If there's anyone you're curious about, I'm happy to look into their
pax support.

[stevvooe]

This is way out in left field. No one know what any of this language means and it makes the specification impossible to read.

What this is called is affectation:

a behavior, speech, or writing that is artificial and designed to impress.

[wking]

On Mon, Sep 26, 2016 at 11:41:45AM -0700, Stephen Day wrote:

This is way out in left field. No one know what any of this language
means and it makes the specification impossible to read.

Can you point out a particular bit of the language I'm adding which
needs clarification?

I agree with @jonboulle that when external specs defining ustar (or
whatever tar flavor we want to require in unpackers) and gzip exist,
we want to punt to them instead of defining our own tar flavor 1.
We already to some punting to other standards, e.g. to RFC 2119 2
and RFC 4634 3. I don't see anything especially confusing about
linking to an external spec for our chosen tar flavor and gzip. But
that doesn't mean that the wording I'm using to do so couldn't be
improved, and I welcome suggestions to improve that wording.

[runcom]

I never heard about ustar in my entire life

[stevvooe]

Can you point out a particular bit of the language I'm adding which
needs clarification?

The entire thing is an example of affectation, even the comment that you responded with. The links you keep adding everywhere don't help to provide context or understanding. They are just a way to make your points look intelligent "'cause references".

The fact is, no one knows what pax or ustar is (nor are these very well written specs). They know what tar is and there are several implementations. The stuff you are linking to are attempts at standards but most people defer to the implementations in GNU and BSD.

Writing a specification like this doesn't add clarity, it just adds volume. Help your reader, don't drown them.

[wking]

On Mon, Sep 26, 2016 at 12:04:41PM -0700, Antonio Murdaca wrote:

I never heard about ustar in my entire life

Have you ever written a tar specification, validator, or unpacker?

[wking]

On Mon, Sep 26, 2016 at 12:05:41PM -0700, Stephen Day wrote:

The fact is, no one knows what pax or ustar is (nor are these very
well written specs). They know what tar is and there are several
implementations.

The stuff you are linking to are attempts at standards but most
people defer to the implementations in GNU and BSD.

I think “the current GNU (or BSD) implementation” or even “the v1.14
GNU tar implementation” are much worse specs from a validation
standpoint than the ustar spec in 1. The point of a spec is to
define these things to promote interop. If someone hands you a
tarball using an S typeflag, does “everyone know” how to unpack that?
@vbatts seems to think not [2](and I agree that folks outside of GNU
tar are less likely to have implemented a GNU-tar extension like
sparse files). If we set a minimum bar for unpackers, then layer
authors will know how far they can push things before they risk
portability issues.

[stevvooe]

@wking

Have you ever written a tar specification, validator, or unpacker?

I know @runcom's credentials. Have you?

[wking]

On Mon, Sep 26, 2016 at 12:35:11PM -0700, Stephen Day wrote:

Have you ever written a tar specification, validator, or unpacker?

I know @runcom's credentials. Have you?

Nope. Which is why I want to stay out of that business ;). But given
that GNU's tar(1), libarchive's tar(5), the POSIX pax spec, etc. all
reference ustar, it's hard to imagine someone who has done any of
those things not having stumbled across “ustar” before. I'd never
heard of it before image-spec either, but I haven't been involved in
writing a spec that needs tar before. I think this sort of prior-art
research is an important part of writing such a spec, and where
someone else has already done all the work writing a tar standard, and
that standard has been widely supported for almost 30 years, I think
it is wise to lean on that standard instead of waving the “everybody
already knows this” wand around it.

[vbatts]

On 26/09/16 12:55 -0700, W. Trevor King wrote:

On Mon, Sep 26, 2016 at 12:35:11PM -0700, Stephen Day wrote:

Have you ever written a tar specification, validator, or unpacker?

I know @runcom's credentials. Have you?

Nope. Which is why I want to stay out of that business ;). But given
that GNU's tar(1), libarchive's tar(5), the POSIX pax spec, etc. all
reference ustar, it's hard to imagine someone who has done any of
those things not having stumbled across “ustar” before. I'd never
heard of it before image-spec either, but I haven't been involved in
writing a spec that needs tar before. I think this sort of prior-art
research is an important part of writing such a spec, and where
someone else has already done all the work writing a tar standard, and
that standard has been widely supported for almost 30 years, I think
it is wise to lean on that standard instead of waving the “everybody
already knows this” wand around it.

I've heard of it. And used it. As well know that many features that
folks rely on for distributing container images is not possible with
ustar and pax.
This whole web of references is not accomplishing the task at hand.

[wking]

On Mon, Sep 26, 2016 at 01:08:19PM -0700, Vincent Batts wrote:

I've heard of it. And used it. As well know that many features that
folks rely on for distributing container images is not possible with
ustar and pax.

This is useful information to put in the spec. Which features are you
looking for that aren't covered by ustar? Link names over 100 chars?
Something else? Which features aren't covered by pax? Sparse files?
Something else? Are these features used commonly enough that
unpackers MUST handle them? Or are they rare enough that we want to
leave a low bar like ustar and leave interop for these advanced
features up to out-of-band communication between the layer author and
unpacker?

[stevvooe]

@wking Linking out to a large document can often add more confusion than it provides.

We need to be clear, in context, which portion of those specifications matter. For example, handling of sockets or pipes is probably not in scope and that needs to be clear this specification.

[wking]

On Mon, Sep 26, 2016 at 02:43:13PM -0700, Stephen Day wrote:

We need to be clear, in context, which portion of those
specifications matter.

Do we? I expect all consumers will be using an off-the-shelf tar
library which supports all of ustar (or all of pax, or whatever).
Requiring part of ustar (or part of pax, or whatever) seems like more
complication than it's worth.

For example, handling of sockets or pipes is probably not in scope
and that needs to be clear this specification.

I think handling FIFOs (aka “named pipes”) is important, and ustar
supports them.

Our current spec also claims (ish) support for sockets 1, but I
don't see the point to that (whoever is listening should be able to
create their own socket when they start up), have not found any spec
covering it, and have not found support for storing them outside of
Perl's Archive::Tar 2. So ustar, pax, and I agree with you that
sockets are not in scope.

[wking]

Rebased around #316 and #336 with 93b5819 → 37ec118. There are no changes in the lines I'm adding, but with #316 in place I'm no longer adding the gzip requirement. I'm dropping the hardlink discussion from #336, because the ustar spec covers hardlinks without us having to explain them locally. The discussion in #336 is more detailed than the ustar equivalent, but folks have been able to write interoperable tar implementations sharing hardlinks since 1988 based on the ustar spec, so I doubt we need to add the additional background and implementation hints here. On the other hand, I'm not strongly against keeping some of these if folks do want to keep some or all of the hardlink tutorial, I'm fine restoring it.

[stevvooe]

I'm dropping the hardlink discussion from #336, because the ustar spec
covers hardlinks without us having to explain them locally. The
discussion in #336 is more detailed than the ustar equivalent, but
folks have been able to write interoperable tar implementations
sharing hardlinks since 1988 based on the ustar spec, so I doubt we
need to add the additional background and implementation hints here.

From experience, this is hardly the case. This spec needs to be clear about hardlinks in the context of layer files, which aren't covered in the 1988 specification.

Remember, while these are indeed tar files, from the point of view of the specification, they represent the semantic concept of a filesystem changeset. One cannot simply unpack these on top of each other and get a useable result. The tar entries need to be interpreted and this specification instructs that interpretation.

[wking]

On Tue, Sep 27, 2016 at 01:32:32PM -0700, Stephen Day wrote:

I'm dropping the hardlink discussion from #336, because the ustar
spec covers hardlinks without us having to explain them locally.
The discussion in #336 is more detailed than the ustar equivalent,
but folks have been able to write interoperable tar
implementations sharing hardlinks since 1988 based on the ustar
spec, so I doubt we need to add the additional background and
implementation hints here.

From experience, this is hardly the case. This spec needs to be
clear about hardlinks in the context of layer files, which aren't
covered in the 1988 specification.

I don't see anything that is layer-specific in the hardlink section
except the psuedocode and following line. I'm fine keeping that if
you like.

Remember, while these are indeed tar files, from the point of view
of the specification, they represent the semantic concept of a
filesystem changeset. One cannot simply unpack these on top of each
other and get a useable result.

I think that's only because of whiteouts, which are orthogonal to
hardlinks. In the absence of whiteouts, you can unpack them on top
of each other and get the correct result, regardless of hardlink
presence.

[runcom]

On Mon, Sep 26, 2016 at 12:04:41PM -0700, Antonio Murdaca wrote:
I never heard about ustar in my entire life
Have you ever written a tar specification, validator, or unpacker?

@wking I'm going to document myself if I don't know something. But, my comment was referring to the fact that in my opinion you're trying to put stuff into this spec which aren't so common overall. The beauty of this spec (as well as the Docker spec) is its simplicity. By requiring all this other stuff you're trying to add everywhere you make that simplicity simply go away. This is just my opinion of course.

[wking]

On Wed, Sep 28, 2016 at 08:07:08AM -0700, Antonio Murdaca wrote:

… my comment was referring to the fact that in my opinion you're
trying to put stuff into this spec which aren't so common overall.

I think ustar is common. Are you aware of a tar tool or library
that has seen any development since 2000 which does not support it?

The beauty of this spec (as well as the Docker spec) is its
simplicity. By requiring all this other stuff you're trying to
add everywhere you make that simplicity simply go away.

The Wikipedia entry currently linked from the spec has ~3k words. The
ustar section I'm linking instead has ~2k words, which have been
carefully honed since 1988 as a normative spec. If we're looking for
a normative link, I think the latter is simpler, and it makes it very
clear what the OCI requires unpackers to handle.

[runcom]

On Wed, Sep 28, 2016 at 08:07:08AM -0700, Antonio Murdaca wrote:
… my comment was referring to the fact that in my opinion you're
trying to put stuff into this spec which aren't so common overall.
I think ustar is common. Are you aware of a tar tool or library
that has seen any development since 2000 which does not support it?
The beauty of this spec (as well as the Docker spec) is its
simplicity. By requiring all this other stuff you're trying to
add everywhere you make that simplicity simply go away.
The Wikipedia entry currently linked from the spec has ~3k words. The
ustar section I'm linking instead has ~2k words, which have been
carefully honed since 1988 as a normative spec. If we're looking for
a normative link, I think the latter is simpler, and it makes it very
clear what the OCI requires unpackers to handle.

you probably didn't get what I said if you are still quoting stuff which I understand - so, fair enough, I'm out of this conversation :)

[wking]

On Wed, Sep 28, 2016 at 08:22:26AM -0700, Antonio Murdaca wrote:

you probably didn't get what I said if you are still quoting stuff
which I understand…

I was confused, but @runcom clarified on IRC that he considers the
ustar requirement to be 2k words of new complexity 1. I, on the
other hand, think we were already suggesting ustar support, because
the Wikipedia article we previously linked had 2:

The file structure to store this information was later standardized
in POSIX.1-1988 and later POSIX.1-2001.

And I think this PR simplifies things by cutting straight to that spec
(currently the ustar stuff originally from the 1988 version) and
bypassing the rest of the stuff in the Wikipedia article.

It also makes it clear that ustar comatibility is only a requirement
for unpackers, and that layer authors are free to generate non-ustar
entries if they're comfortable with the portability risk, while
previously we made no clear requirements of either the layer author or
the unpacker.

[wking]

As an example of the sort of thing you can do after this change, I've
stubbed in a callback-based validator with a stict mode 1. In
strict mode it errors out for typeflags not defined by ustar, but in
permissive mode it does not.

However, without this change there are no grounds in the spec for
making that distinction around ustar typeflags. There's only the
warning about ‘S’ (GNU's sparse), but no warnings for the other GNU
extensions like ‘K’ or ‘L’ or pax extensions like ‘g’ and ‘x’.

[runcom]

However, without this change there are no grounds in the spec for
making that distinction around ustar typeflags. There's only the
warning about ‘S’ (GNU's sparse), but no warnings for the other GNU
extensions like ‘K’ or ‘L’ or pax extensions like ‘g’ and ‘x’.

and maybe it's not the right time to do this? or it's not necessary altogether? you can see it yourself, that change isn't required anywhere, the spec is still working even w/o it. it's you adding it and then making that change.

[wking]

On Thu, Sep 29, 2016 at 03:33:34AM -0700, Antonio Murdaca wrote:

However, without this change there are no grounds in the spec for
making that distinction around ustar typeflags. There's only the
warning about ‘S’ (GNU's sparse), but no warnings for the other
GNU extensions like ‘K’ or ‘L’ or pax extensions like ‘g’ and ‘x’.

and maybe it's not the right time to do this?

It's hard for me to imagine a modern tar implementation not even
supporting ustar (but this isn't my field, maybe they're out there).
If we don't land a requirement like this before 1.0, the risk is that
layer authors publish layers that use more advanced tar features and
then have their assumed compatibility rug pulled out from under them
when we later land something like this to say “actually, when we said
‘tar’ before, we're only requiring ustar support. Sorry”.
I'd rather avoid that by setting our low bar before 1.0.

or it's not necessary altogether? you can see it yourself, that
change isn't required anywhere, the spec is still working even w/o
it. it's you adding it and then making that change.

The current layer-publisher ecosystem is (as I understand it, again,
not my field) a monoculture around Go's archive/tar library. The idea
behind the OCI is to pin down standards to facilitate interop between
multiple implementations. If I create a tar file using
SCHILY.acl.default, will all OCI implementations be able to unpack it?
Libarchive 1, star 2, GNU tar 3, and tar-split 4 understand
it. But I can't find code in Docker's unpacker to support it 5.
With this PR, it's clear that OCI-compliant unpackers are not
required to support SCHILY.acl.default (although they can support it
if they want), and layer authors can push layers using
SCHILY.acl.default if they're comfortable figuring out that
compatibility on their own. And layer authors can push layers that
only use ustar without having to worry about compatibility with
OCI-compliant unpackers.

[vbatts]

I'm still calling this ridiculous and excessive. At best linking to some IEEE pay-walled doc. But for many, that is a loose link that they will not research the validity of. I'm likely going to close this issue.

[wking]

On Thu, Oct 06, 2016 at 10:48:24AM -0700, Vincent Batts wrote:

I'm still calling this ridiculous and excessive. At best linking to
some IEEE pay-walled doc.

It currently links to a non-paywalled 2013 version of the ustar spec.

[aecolley]

This discussion is exactly what the Unix world used to be like before Posix: we just wanted our programs to run without having to make conditionals for every different distribution's way of doing every single trivial thing. Posix didn't standardize the tar command, so it's still a mess. Now it's an anachronistic mess.

Why not just say GNU tar? Meaning that layers in conforming images will be extractable by GNU tar without unusual options. If you really wanted to tie it down, you could specify a tar version (1.29 is current). GNU is already central to Linux and tolerated on Windows, so it's a good-enough choice.

[wking]

On Mon, Oct 17, 2016 at 04:08:12PM -0700, Adrian Colley wrote:

Why not just say GNU tar?
Meaning that conforming images will be extractable by GNU tar
without unusual options. If you really wanted to tie it down, you
could specify a tar version (1.29 is current). GNU is already
central to Linux and tolerated on Windows, so it's a good-enough
choice.

I'm ok with requiring GNU tar, although I'd definitely want to pin to
a version, and it would be nice to have a clear definition of what
that spec is. Linking to [1](or docs built from it [2], like we
currently do [3]) seems fairly loose. For example:

In addition to entries describing archive members, an archive may
contain entries which tar itself uses to store information. See
section Including a Label in the Archive, for an example of such an
archive entry.

is pretty open-ended. The questions are “are the GNU docs
sufficiently spec-like?” and “do we actually want to require
implementations to support everything in the GNU spec?”. I'm not sure
about either, but would be happier linking to a spec of any stripe
than leaving the required tar support unspecified.

[vbatts]

I'm closing this as docs currently reference GNU tar "standard" format, as well as having requirements for xattrs. This PR goes far enough that would be immediately incompatible with what folks are using and expecting.