chore(oci): migrate to ORAS Golang library v2 #12310

zregvart · 2023-08-16T16:11:23Z

What this PR does / why we need it:

Updates from oras.land/oras-go to oras.land/oras-go/v2. The main user facing change is that the --plain-http parameter is now required even when accessing OCI registries running on localhost.

Fixes #11821

Special notes for your reviewer:

I'm new to this codebase and some tests needed to be adjusted to match the new dependency, please look over the test changes to see if there are any issues with those.

If applicable:

this PR contains documentation
this PR contains unit tests
this PR has been tested for backwards compatibility

pkg/registry/client.go

shizhMSFT

LGTM but I am not a maintainer.

pkg/registry/client.go

zregvart · 2023-08-23T08:42:49Z

Thank you @Wwwsylvia for excellent comments and suggestions.
@sabre1041 this has gotten an review and suggestions from two ORAS contributors, can you have a look now?

sabre1041 · 2023-08-24T14:33:17Z

Spent some time testing out the integration of the new ORAS v2 library. Below are the results

Test	Result	Comment
Push Chart with Registry	✅
Pull + show values from Registry	✅
Push Chart with Registry with self signed certificates	✅
Pull + show values from Registry with self signed certificates	✅
Push Chart with Registry with plain http	✅
Pull + show values from Registry with plain http	✅
Login to registry	✅
Logoff from registry	✅
Login to registry with self signed certificates	✅
Logoff from registry with self signed certificates	✅
Login to registry with plain http	❌	Fails. No option exists in Helm to explicitly specify plain http. Existing functionality automatically falls back to plain HTTP and succeeds

The migration looks good, but there does need to be additional work into the one validation failure scenario as described above in addition to the now conflicting go.mod file

zregvart · 2023-08-24T15:35:37Z

@sabre1041 how would you like to handle the plain HTTP issue? With a if localhost assume --plain-http? Or should I add a --plain-http flag to helm, the helm push and helm pull already have it.

sabre1041 · 2023-08-24T17:32:14Z

@zregvart Personally, I would like to have feature parity with --plain-http in the login subcommand so that it aligns with the rest of capabilities surronding communicating

zregvart · 2023-08-25T12:11:56Z

@sabre1041 added support --plain-http to helm login the diff is here.

sabre1041

LGTM

sabre1041 · 2023-08-26T10:11:12Z

@zregvart This looks good. Can you resolve the conflict and we should be in good shape

Wwwsylvia · 2023-08-30T03:41:29Z

Hi @zregvart @sabre1041, FYI, oras-go v2.3.0 is just out, would you like to update the go mod in this PR?

sabre1041 · 2023-08-30T06:37:01Z

Hi @zregvart @sabre1041, FYI, oras-go v2.3.0 is just out, would you like to update the go mod in this PR?

@zregvart if you want to update, I can retest again

banjoh · 2023-12-15T20:09:16Z

Spent some time testing out the integration of the new ORAS v2 library. Below are the results

Test Result Comment
Push Chart with Registry ✅
Pull + show values from Registry ✅
Push Chart with Registry with self signed certificates ✅
Pull + show values from Registry with self signed certificates ✅
Push Chart with Registry with plain http ✅
Pull + show values from Registry with plain http ✅
Login to registry ✅
Logoff from registry ✅
Login to registry with self signed certificates ✅
Logoff from registry with self signed certificates ✅
Login to registry with plain http ❌ Fails. No option exists in Helm to explicitly specify plain http. Existing functionality automatically falls back to plain HTTP and succeeds
The migration looks good, but there does need to be additional work into the one validation failure scenario as described above in addition to the now conflicting go.mod file

I ran these tests as well and they seem to pass. One test that is missing is the mTLS use case. I configured zot to enforce mTLS and managed to push and pull charts. I'm however not convinced that zot is enforcing mTLS fully cause I also managed to push charts without a client cert passed using --cert-file flag. Omitting this flag ought to fail tests. I'll try with another registry

banjoh · 2023-12-19T01:00:41Z

I managed to run an mTLS test using this setup of nginx + zot. This PR changes work as expected. helm successfully gets authenticated using a client cert and pushes the chart to the registry

helm push ../playground/my-chart-1.0.0.tgz oci://127.0.0.1 --ca-file ca.crt --cert-file client.crt --key-file client.key
Pushed: 127.0.0.1/my-chart:1.0.0
Digest: sha256:e0cdbbf0ddd269b74abf748fba3e6ae80b7e5fe6d5c9285ad31c239bbd354b0a

Strangely enough, the current version of helm cli did not work as I expected it to. I might be missing something here though. By looking at the output, it seems to be defaulting to insecure connections even if certs have been provided. It might just be the PR fixed an existing issue.

helm push ../playground/my-chart-1.0.0.tgz oci://127.0.0.1 --ca-file ca.crt --cert-file client.crt --key-file client.key
Error: failed to do request: Head "http://127.0.0.1/v2/my-chart/blobs/sha256:0160276299adc66a534585cacf1af08d1dd3ac9c4eb6aa20d82af81deb40e67e": dial tcp 127.0.0.1:80: connect: connection refused

helm version
version.BuildInfo{Version:"v3.13.3", GitCommit:"c8b948945e52abba22ff885446a1486cb5fd3474", GitTreeState:"clean", GoVersion:"go1.21.5"}

zregvart · 2024-01-02T12:53:56Z

I is unclear to me what the expectation here is, i.e. could a release of Helm v4 be made with this or should the backward compatibility be maintained without exceptions. I think communicating the expectations would make this work a lot easier.

I've made a judgement call and restored the backward compatibility in a62313e, now the ClientOptResolver function panics on
invocation and the LoginOption function accepts the loginOperation and does not return error. The loginOperation struct was still modified, it is and it's members are unexported and I consider this not to be braking backward compatibility.

chrisgacsal · 2024-01-19T14:39:48Z

@zregvart Thank you for keeping this PR up-to-date! 🙏🏻

mattfarina · 2024-02-22T19:40:32Z

pkg/registry/client.go

-		}
+func ClientOptResolver(_ remotes.Resolver) ClientOption {
+	return func(c *Client) {
+		c.err = errDeprecatedRemote


This is still a breaking change and requires a Helm major version change. It's about taking functions and making them non-functional with an error.

As previously noted support for containerd was removed from oras-go and there are no plans to reintroduce it. See oras-project/oras-go#64 (comment) for details.

I have yet to see any suggestions on how to get this addressed, no plans on the next major version of Helm either. Please do share any insights that you are privy to.

Frankly I do not see a way forward with this and will limit my effort going forward. Anyone is welcome to pick this up and continue the work.

For the records, this client option:

was recently added in 3.13.0 (under 6 months ago, and only two weeks before this PR was opened)

was added without any real testing

was in beta / RC1 for less than 10 days (3.13.0-rc.1 is identical to 3.13.0)

was implemented by the same person that opened the issue

has what seems like 1 known use across all of github's public codebases, barring matches that seem to be forks or vendored modules

vmware-labs/distribution-tooling-for-helm does use the current implementation, but again, VMware was the team that opened the issue and implemented this option in helm.

d2iq-labs/csi-driver-trusted-ca's usage is of it's own identically named option. This repository depends on helm 3.11.1 and has seemingly implemented #10623 itself

has alternative solutions to what the original issue was asking for -- albeit yes, breaking

In terms of breaking changes, I think this is very manageable and the benefits of moving to oras v2 outweigh the costs of delaying this PR until the theoretical helm v4 release. I'm fairly sure that VMware team members @antgamdia or @alemorcuq would be okay with managing this change, as they are the source of this API in helm and still seem to be the only people affected.

Hi @zevisert, thanks for the analysis on the current usage of this ClientOptResolver we added.
To give you all some context, we required this resolver to be configurable for use cases in Kubeapps (see vmware-tanzu/kubeapps#4194). However... due to different business reasons, we never get the bandwidth to actually make the changes on our side). We will probably bypass the Helm usage and will interact directly with ORAS, I guess.

I have checked with @alemorcuq and I think they can safely remove the usage of this option. In fact, I have just opened vmware-labs/distribution-tooling-for-helm#55 for doing so.
Both ClientOptPlainHTTP and ClientOptHTTPClient options cover their needs without the need of setting the resolver manually.

So, my (personal) two cents here: despite it being a breaking change, migrating to an up-to-date and maintained ORAS version and removing some CVEs is worthfile enough.

Great analysis @zevisert! Just FYI https://github.com/d2iq-labs/csi-driver-trusted-ca is not maintained so does not need to be considered in any discussion.

I'm also very supportive of getting this change in.

Hey @mattfarina, I understand if you're busy with other things, but we'd appreciate if you could come back here reconsider this sometime soon :)

I'd like to add one more thing to the discussion: This is the only remaining reference to containerd v1.x in Helm. Removing this API allows the entire containerd dependency to be dropped. This would allow Helm and the upcoming containerd 1.8 API to be used in the same Go module, avoiding ambiguous import errors (e.g. github.com/containerd/containerd v1.7.16 vs. github.com/containerd/containerd/api v1.8.0-rc.0).

Updates from `oras.land/oras-go` to `oras.land/oras-go/v2`. The main user facing change is that the `--plain-http` parameter is now required even when accessing OCI registries running on `localhost`. I'm new to this codebase and some tests needed to be adjusted to match the new dependency, please look over the test changes to see if there are any issues with those. Fixes helm#11821 Signed-off-by: Zoran Regvart <zoran@regvart.com>

This restores the `ClientOptResolver` function which now panics on invocation and previous signature of the `LoginOption` function. Signed-off-by: Zoran Regvart <zoran@regvart.com>

pull-request-size bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Aug 16, 2023

zregvart mentioned this pull request Aug 17, 2023

Switch to oras.land/oras-go/v2 #11821

Open

sabre1041 self-assigned this Aug 17, 2023

Wwwsylvia reviewed Aug 17, 2023

View reviewed changes

pkg/registry/client.go Outdated Show resolved Hide resolved

pkg/registry/client.go Outdated Show resolved Hide resolved

pkg/registry/client.go Outdated Show resolved Hide resolved

pkg/registry/client.go Outdated Show resolved Hide resolved

zregvart force-pushed the main branch 3 times, most recently from 7c04305 to 4255ba2 Compare August 19, 2023 14:26

Wwwsylvia reviewed Aug 22, 2023

View reviewed changes

pkg/registry/client.go Outdated Show resolved Hide resolved

pkg/registry/client.go Outdated Show resolved Hide resolved

pkg/registry/client.go Outdated Show resolved Hide resolved

pkg/registry/client.go Outdated Show resolved Hide resolved

zregvart force-pushed the main branch from 4255ba2 to 8f876e1 Compare August 22, 2023 11:27

Wwwsylvia reviewed Aug 22, 2023

View reviewed changes

pkg/registry/client.go Outdated Show resolved Hide resolved

zregvart force-pushed the main branch from 8f876e1 to dc0ffa3 Compare August 22, 2023 11:44

shizhMSFT approved these changes Aug 22, 2023

View reviewed changes

joejulian added the oci Related to Helm OCI feature label Aug 22, 2023

Wwwsylvia reviewed Aug 23, 2023

View reviewed changes

pkg/registry/client.go Outdated Show resolved Hide resolved

zregvart force-pushed the main branch from dc0ffa3 to 6efbe85 Compare August 23, 2023 08:41

zregvart force-pushed the main branch from 6efbe85 to 928a201 Compare August 23, 2023 08:43

zregvart force-pushed the main branch from 928a201 to a2dc34a Compare August 24, 2023 15:38

zregvart force-pushed the main branch from a2dc34a to 492f78f Compare August 25, 2023 12:10

sabre1041 approved these changes Aug 26, 2023

View reviewed changes

zregvart force-pushed the main branch 2 times, most recently from a3cd03e to 1ee13f3 Compare August 29, 2023 14:19

zregvart force-pushed the main branch from 1ee13f3 to ebe0e00 Compare August 30, 2023 09:05

ysdongAmazon mentioned this pull request Dec 18, 2023

Bump github.com/docker/docker from 20.10.24+incompatible to 24.0.7+incompatible aws/aws-app-mesh-controller-for-k8s#735

Open

banjoh approved these changes Dec 19, 2023

View reviewed changes

zregvart force-pushed the main branch from 6c0b19a to 1152c3f Compare December 21, 2023 13:42

DmitriyLewen mentioned this pull request Dec 27, 2023

chore(deps): bump github.com/moby/buildkit from 0.11.6 to 0.12.4 aquasecurity/trivy#5708

Closed

chrisgacsal mentioned this pull request Jan 11, 2024

chore(deps): use helm version with ORASv2 openclarity/vmclarity#1075

Merged

7 tasks

zregvart force-pushed the main branch from a62313e to 2b81f88 Compare January 12, 2024 13:31

zregvart force-pushed the main branch from 2b81f88 to 4a92217 Compare January 19, 2024 13:38

zregvart force-pushed the main branch from 4a92217 to 498142f Compare January 30, 2024 12:30

banjoh mentioned this pull request Feb 1, 2024

Helm template doesnt work with credentials for oci registries #12767

Open

zregvart force-pushed the main branch 2 times, most recently from 05fe6fe to f974fac Compare February 12, 2024 10:48

mattfarina requested changes Feb 22, 2024

View reviewed changes

This was referenced Feb 26, 2024

Adapt the current Helm's OCI code to use the upstream version vmware-tanzu/kubeapps#4194

Open

Remove non-required ClientOptResolver option usage vmware-labs/distribution-tooling-for-helm#55

Open

zregvart force-pushed the main branch 2 times, most recently from dda9052 to 2f3bd30 Compare March 22, 2024 09:35

zregvart force-pushed the main branch from 2f3bd30 to 358a649 Compare April 2, 2024 13:05

zregvart force-pushed the main branch from 358a649 to 9b1dba9 Compare April 18, 2024 10:40

zregvart force-pushed the main branch from 9b1dba9 to 29fcbe1 Compare May 15, 2024 10:03

scottrigby mentioned this pull request May 16, 2024

bump oras minor version #12998

Merged

3 tasks

scottrigby added the refactor label May 16, 2024

sunjayBhatia mentioned this pull request May 21, 2024

test/e2e: Updates for k8s 1.30 and CAPI/CAPV bump kubernetes/cloud-provider-vsphere#1034

Closed

zregvart added 2 commits June 10, 2024 11:24

refactor(oci): maintain backward compatibility

9adc7db

This restores the `ClientOptResolver` function which now panics on invocation and previous signature of the `LoginOption` function. Signed-off-by: Zoran Regvart <zoran@regvart.com>

zregvart force-pushed the main branch from 29fcbe1 to 9adc7db Compare June 10, 2024 09:24

twz123 mentioned this pull request Jun 10, 2024

Update to containerd 2.0 k0sproject/k0s#4581

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(oci): migrate to ORAS Golang library v2 #12310

chore(oci): migrate to ORAS Golang library v2 #12310

zregvart commented Aug 16, 2023

shizhMSFT left a comment

zregvart commented Aug 23, 2023

sabre1041 commented Aug 24, 2023

zregvart commented Aug 24, 2023

sabre1041 commented Aug 24, 2023

zregvart commented Aug 25, 2023

sabre1041 left a comment

sabre1041 commented Aug 26, 2023

Wwwsylvia commented Aug 30, 2023

sabre1041 commented Aug 30, 2023

banjoh commented Dec 15, 2023

banjoh commented Dec 19, 2023 •

edited

zregvart commented Jan 2, 2024

chrisgacsal commented Jan 19, 2024

mattfarina Feb 22, 2024

zregvart Feb 23, 2024

zevisert Feb 23, 2024

antgamdia Feb 26, 2024

jimmidyson Feb 26, 2024

zevisert Apr 2, 2024

twz123 Jun 10, 2024

chore(oci): migrate to ORAS Golang library v2 #12310

Are you sure you want to change the base?

chore(oci): migrate to ORAS Golang library v2 #12310

Conversation

zregvart commented Aug 16, 2023

shizhMSFT left a comment

Choose a reason for hiding this comment

zregvart commented Aug 23, 2023

sabre1041 commented Aug 24, 2023

zregvart commented Aug 24, 2023

sabre1041 commented Aug 24, 2023

zregvart commented Aug 25, 2023

sabre1041 left a comment

Choose a reason for hiding this comment

sabre1041 commented Aug 26, 2023

Wwwsylvia commented Aug 30, 2023

sabre1041 commented Aug 30, 2023

banjoh commented Dec 15, 2023

banjoh commented Dec 19, 2023 • edited

zregvart commented Jan 2, 2024

chrisgacsal commented Jan 19, 2024

mattfarina Feb 22, 2024

Choose a reason for hiding this comment

zregvart Feb 23, 2024

Choose a reason for hiding this comment

zevisert Feb 23, 2024

Choose a reason for hiding this comment

antgamdia Feb 26, 2024

Choose a reason for hiding this comment

jimmidyson Feb 26, 2024

Choose a reason for hiding this comment

zevisert Apr 2, 2024

Choose a reason for hiding this comment

twz123 Jun 10, 2024

Choose a reason for hiding this comment

banjoh commented Dec 19, 2023 •

edited