v1.Layer doesn't abstract away tarball

[jonjohnsonjr]

There's not a way to access files in a v1.Layer other than through Uncompressed + tar.NewReader. Our current abstraction is fine for moving images around, but not for accessing the actual filesystem. If we exposed a way to directly access files, we could be resilient to changes to the format, but we probably don't have enough information to make good decisions for the actual abstraction. Opening this as a nice place to dump ideas.

[clrprod]

I like FileMaps as an export, since they're easy to generate. They don't work well for larger containers. The classic way to represent files is a http.FileServer. We could also implement a Read / ReadDir method. which is cheaper. I know there is some work on making this really cheap (from Brad Fitzpatrick), so maybe we should copy the api they are setting up?

[jonjohnsonjr]

For reference: https://github.com/google/crfs (godoc).

There might be a small interface we could pull out of here.

[jonjohnsonjr]

We should implement the same interface as http.FileSystem:

type FileSystem interface {
        Open(name string) (File, error)
}

The File has Readdir:

type File interface {
        io.Closer
        io.Reader
        io.Seeker
        Readdir(count int) ([]os.FileInfo, error)
        Stat() (os.FileInfo, error)
}

So we could call layer.Open("/").Readdir(-1) and traverse the filesystem, but it would be really nice to have a way to efficiently get an index of the entire filesystem (not just one directory), something like Brad's TOC footer for stargz.

[dprotaso]

somewhat related: https://github.com/dprotaso/license-collector/blob/master/layers.go I added a walk and you could pass in your own 'struct' instead of the 'tar.Header'

…

On Wed, Jul 24, 2019 at 4:34 PM jonjohnsonjr ***@***.***> wrote: We should implement the same interface as http.FileSystem <https://golang.org/pkg/net/http/#FileSystem>: type FileSystem interface { Open(name string) (File, error) } The File has Readdir: type File interface { io.Closer io.Reader io.Seeker Readdir(count int) ([]os.FileInfo, error) Stat() (os.FileInfo, error) } So we could call layer.Open("/").Readdir(-1) and traverse the filesystem, but it would be really nice to have a way to efficiently get an index of the entire filesystem (not just one directory), something like Brad's TOC footer <https://github.com/google/crfs/blob/a3f395beac6199531833859cfb72ae5f440d066f/stargz/stargz.go#L557-L575> for stargz. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#487?email_source=notifications&email_token=AAAERAUPPJ4P2F7XCHIFBH3QBC4ELA5CNFSM4IGT4X2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2XRAQQ#issuecomment-514789442>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAERAVEGC3X55SXGVUYPP3QBC4ELANCNFSM4IGT4X2A> .

[jonjohnsonjr]

Related: #531

Compressed presumes gzip.

[micahyoung]

Just wanted to add that this would be very nice for abstracting some Windows/Linux layer content differences (as required by Docker WCOW, K8s on Windows, ContainerD for Windows, etc):

When Reading:

• File entries in the tar are prefixed with Files/ instead of /

If the abstraction went so far as to support writing to layers:

• Each Windows layers require Files/ and Hives/ directory entries
• Each parent directory for a nested path must have an prior, explicit tar entry

[jonjohnsonjr]

I'd be interested in learning more about that. I haven't looked too much into it, but the windows container stuff seems remarkably bad, and there are a ton of random caveats that aren't really documented anywhere.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[jonjohnsonjr]

https://tip.golang.org/doc/go1.16#fs 👀