Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Userspace Convertor] Overlaybd Layer Versioning for Deduplication #250

Open
1 task done
estebanreyl opened this issue Dec 13, 2023 · 2 comments · May be fixed by #271
Open
1 task done

[Userspace Convertor] Overlaybd Layer Versioning for Deduplication #250

estebanreyl opened this issue Dec 13, 2023 · 2 comments · May be fixed by #271
Assignees
Labels
enhancement New feature or request

Comments

@estebanreyl
Copy link
Member

What is the version of your Accelerated Container Image

v1.0.2

What would you like to be added?

Layer deduplication needs to account for the underlying overlaybd-tool versions to avoid conversion mismatches.

Why is this needed for Accelerated Container Image?

As layer deduplication is currently unaware of any versioning for the underlying overlaybd tools, whenever overlaybd tools are upgraded, this can result in a situation where deduplication will try to reuse layers that are no longer compatible during conversion. This is also relevant for #249 and the non-user-space convertor, I think. We have been tracking this manually to prevent issues on our side, but a more programmatic approach is probably necessary in light of a few format issues we have discovered recently like containerd/overlaybd#301 and running with root vs not doing so. Overall, we want to discuss how to obtain versioning from overlaybd-tools to account for breaking changes and take this into account for the layer deduplication.

Are you willing to submit PRs to contribute to this feature?

  • Yes, I am willing to implement it.
@estebanreyl estebanreyl added the enhancement New feature or request label Dec 13, 2023
@estebanreyl estebanreyl self-assigned this Dec 13, 2023
@yuchen0cc
Copy link
Contributor

It's reasonable and important to aware changes of reproducible conversion.

First, we add a CI for userspace convertor to avoid unexpected inconsistency.
https://github.com/containerd/accelerated-container-image/blob/main/.github/workflows/ci-userspace-convertor.yml (in #253)

Second, we make sure root & non-root will produce the same product.
containerd/overlaybd#303

For versioning for convertor, we are planning to use an option to get consistency version. Maybe --version or --consistency-version.
The version should be a combined version of convertor and underlay overlaybd tools.
If it's an unavoidable change (like #251), we update the version.
@estebanreyl what do you think of this ?

@estebanreyl
Copy link
Member Author

Sorry for my delay with replying, I completely missed your reply and forgot to check. The versioning suggestion and addition of the CI seem completely reasonable to me and great additions. As for what flag to use I am inclined to use the --version and internally have a consistency-version group to which a version belongs which is actually used for the deduplication. This way we can update the version independent of the deduplication logic and still maintain the deduplication. As for how to combine the overlaybd version into this that's where I am less clear. Is there a way today to determine the version of the tools once they are binaries? If so, we can note what --version goes with a specific overlaybd version, when possible, to manage the deduplication. I'd be happy to create a PR for expressing this initially over the next few days to have a more precise discussion if one is not already in the works on your end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
2 participants