Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support tar-split style storage #305

Open
cgwalters opened this issue Jun 8, 2022 · 2 comments
Open

Support tar-split style storage #305

cgwalters opened this issue Jun 8, 2022 · 2 comments

Comments

@cgwalters
Copy link
Member

Right now when we fetch container images we parse them and:

  • Drop on the floor things like files in /var
  • Rename /etc
  • Pass the tar stream to ostree commit, which will also discard e.g. mtimes

And other bits.

This breaks the ability to re-synthesize the tar stream; basically what I think there are some use cases for is ostree container push. This also relates to #273

I think the solution to this is to take a similar approach as https://github.com/vbatts/tar-split
Although, I think what I'd propose is an algorithm like this:

let mut tar_meta_entries = Vec::new();
for entry in tar {
    let (metadata, raw_metadata_bytes) = entry.metadata();
    let checksum = write_ostree_object(entry);
    tar_meta_entries.push((raw_metadata_bytes, checksum));
}

Then we serialize tar_meta_entries...somewhere. In the commit metadata is an obvious place, but would need some size analysis. We could store it in another ref as data, but just mildly ugly to have two refs per layer.

To reliably reconstitute the bit for bit uncompressed tar stream, we just need to load tar_meta_entries, write the metadata bytes, then load the corresponding ostree object and append its raw data.

@cgwalters
Copy link
Member Author

A key feature this would enable is ostree container push from the running OS. (Obviously, it also kind of scope creeps this whole thing into an ostree-based container storage backend)

@cgwalters
Copy link
Member Author

Was briefly looking at this today, and it looks like the current tar API may already provide exactly what we need in e.g. https://docs.rs/tar/latest/tar/struct.Entry.html#method.raw_header_position

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant