Reconsider "Sparse files SHOULD NOT be used" restriction?

[AkihiroSuda]

f5a91ff added the following sentence to layer.md:

[Sparse files](https://en.wikipedia.org/wiki/Sparse_file) SHOULD NOT be used because they lack consistent support across tar implementations.

GIven that most implementations use Go and now Go supports sparse files (golang/go#13548), can we reconsider this?

@vbatts @stevvooe

[wking]

In #255, @vbatts made this an implementation-support issue instead of a filesystem-support issue, because he wasn't aware of many filesystems that didn't support sparse files. The recent Go support for sparse files has turned up a number of cases that don't support sparse files (e.g. see golang/go@718d9de giving up on trying to parameterize this, with the previous parameterization suggesting FFS and HFS+ lack support, and AUFS has only partial support). However, if that's the argument we want to make, we'd still want to adjust the spec to mention inconsistent OS/FS support instead of inconsistent tool support.

[vbatts]

I'm still keen to keep that sparse files SHOULD not be used. If an implementation uses them, they are accepting the risk of checksums not validating or similar.

…

On Thu, Sep 21, 2017, 18:03 W. Trevor King ***@***.***> wrote: In #255 <#255>, @vbatts <https://github.com/vbatts> made this an implementation-support issue instead of a filesystem-support issue, because he wasn't aware of many filesystems that didn't support sparse files <#255 (comment)>. The recent Go support for sparse files has turned up a number of cases that don't support sparse files (e.g. see ***@***.*** <golang/go@718d9de> giving up on trying to parameterize this, with the previous parameterization suggesting FFS and HFS+ lack support, and AUFS has only partial support). However, if that's the argument we want to make, we'd still want to adjust the spec to mention inconsistent OS/FS support instead of inconsistent tool support. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#733 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEF6VUyQrWoZ94HtDkqaSHYDnnsEjlxks5skt01gaJpZM4Pe286> .

[wking]

On Thu, Sep 21, 2017 at 11:31:03PM +0000, Vincent Batts wrote: If an implementation uses them, they are accepting the risk of checksums not validating or similar.

That's only a problem for unpack/repack, though right? That seems like less of a “passing image data between parties” issue and more of a “some consumers may create sub-optimal images if they use an image containing sparse files as a base” issue. It's a risk, and I'm fine calling it out (which we don't do at the moment), but I'm not sure it's a SHOULD-worthy risk. Folks will still be able to unpack the image (assuming their local OS/FS is capable of supporting sparse files or they have a big enough disk to hold the unsparsed file) and run containers from it, which seems like the primary use-case image-spec is enabling. And if folks occasionally pack a new image based on the content and happen to not recycle the same tar blob, it still seems like a CAS win to not be holding unsparsed copies.

[stevvooe]

In general, sparse files should not be used. You're still welcome to use them but I don't see the benefit, especially with compression.

@AkihiroSuda Is there are particular use case that you had in mind?

[AkihiroSuda]

• adduser (with large uid value), which seeks /var/log/lastlog depending on uid value: docker build hangs/crashes when useradd with large UID moby/moby#5419

[stevvooe]

@AkihiroSuda What if we added a section that says implementations should be able to consume them?

Considering the sparse support of sparse files, I'm a little reticent to remove this from the specification. It doesn't say you can't use them, but generating them may come with consequences.

[AkihiroSuda]

OK, closing this issue.

However, if we encounter real compatibility problem across different implementations, I'd like to reopen this issue with some new annotation convention (org.opencontainers.image.has-sparse-tar-layer-blah-blah-blah)

[vbatts]

Understandable

…

On Mon, Sep 25, 2017, 22:49 Akihiro Suda ***@***.***> wrote: OK, closing this issue. However, if we encounter real compatibility problem across different implementations, I'd like to reopen this issue with some new annotation convention (org.opencontainers.image.has-sparse-tar-layer-blah-blah-blah) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#733 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEF6dbyhCxzRsYkLeLJqzXWThaY0ctIks5smGYfgaJpZM4Pe286> .