Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFE: support pushing images using zstd:chunked compression #112

Open
miabbott opened this issue Mar 2, 2024 · 8 comments · Fixed by #134
Open

RFE: support pushing images using zstd:chunked compression #112

miabbott opened this issue Mar 2, 2024 · 8 comments · Fixed by #134
Assignees
Labels
type: feature Brand new functionality, features, pages, workflows, endpoints, etc.
Milestone

Comments

@miabbott
Copy link

miabbott commented Mar 2, 2024

The use of zstd:chunked compression can dramatically decrease the amount of data that needs to be pulled over the wire when the client is able to to do partial pulls. See https://www.redhat.com/sysadmin/faster-container-image-pulls for details and an asciinema demo.

I believe support for pushing container images using zstd:chunked compression exists in newer versions of buildah, docker, and podman.

However, I think the big issue is that the ubuntu-latest GH runners have very old versions of all that software, so newer versions of the software would have to be installed into the runner before this option could be used/supported.

@xynydev
Copy link
Member

xynydev commented Mar 2, 2024

Hadn't heard of this, sounds cool. The actual pushing logic is in the CLI, so @gmpinder might want to take a look at this.

However, I think the big issue is that the ubuntu-latest GH runners have very old versions of all that software, so newer versions of the software would have to be installed into the runner before this option could be used/supported.

The action actually currently runs the build (and push) from inside a container with podman, so this wouldn't be a problem. Except that building inside a container is probably the biggest cause of blue-build/github-action#16, so we're currently working towards supporting building with docker directly instead...

Should probably investigate, though, if the GH runners have a newer version of docker that supports this feature (podman is on a very old version, that's why we can't use it directly here).
Edit: v23 of Docker adds this feature. GitHub Actions is currently at 22.04. Docker version there is currently at v20 (I think). GitHub runner images use Docker from their own repository, though, but I'm not sure what the latest available Docker version there is.
Edit2: Docker in GitHub runners is on v24, this feature should be supported.

Also need to investigate whether rpm-ostree supports these new compression types yet.

@miabbott
Copy link
Author

miabbott commented Mar 2, 2024

Also need to investigate whether rpm-ostree supports these new compression types yet.

I'm fairly certain the container handling code in rpm-osteee is using skopeo or the containers/image library, so I believe it will support the zstd compression as it is the same library that buildah and podman use.

@gmpinder gmpinder transferred this issue from blue-build/github-action Mar 2, 2024
@gmpinder gmpinder added the type: feature Brand new functionality, features, pages, workflows, endpoints, etc. label Mar 2, 2024
@gmpinder
Copy link
Member

I've been looking into the docs for docker and saw some options for changing the compression. There are some structural code changes I'm working through before I can get to this, but I'll definitely work on trying this out. I recently put in a PR #121 to check for minumum versions of the various build tools we support.

@gmpinder gmpinder added this to the v0.8.3 milestone Mar 17, 2024
@gmpinder gmpinder self-assigned this Mar 17, 2024
@gmpinder gmpinder linked a pull request Mar 23, 2024 that will close this issue
@fiftydinar
Copy link
Contributor

I would re-open this issue as upstream-related, as ZSTD:chunked compression is not yet supported in rpm-ostree as linked in issue above.

@gmpinder
Copy link
Member

I would re-open this issue as upstream-related, as ZSTD:chunked compression is not yet supported in rpm-ostree as linked in issue above.

Good idea

@gmpinder gmpinder reopened this Mar 25, 2024
@gmpinder gmpinder removed this from the v0.8.3 milestone Mar 25, 2024
@gmpinder gmpinder added the state: blocked Something is blocking action. label Mar 25, 2024
@fiftydinar
Copy link
Contributor

fiftydinar commented May 22, 2024

ZSTD:chunked support got merged in ostree-rs-ext (reposting from the linked issue, since this one is open for tracking)

ostreedev/ostree-rs-ext#622

@gmpinder
Copy link
Member

gmpinder commented Jul 9, 2024

It's looking like we can go ahead and move forward with creating a multi-manifest image that will have both gzip and zstd. Then the pulling runtime will determine which to use.

Thanks to Robert (p5) for all the work figuring this out: https://discord.com/channels/1072614816579063828/1242839114735423488/1257480020260159549

@gmpinder gmpinder removed the state: blocked Something is blocking action. label Jul 9, 2024
@gmpinder gmpinder added this to the v0.10.0 milestone Jul 9, 2024
@gmpinder
Copy link
Member

gmpinder commented Jul 9, 2024

Also tracking this (containers/bootc#215) issue since this will give bootc support which we will also have via #200 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: feature Brand new functionality, features, pages, workflows, endpoints, etc.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants