internal/mod/modregistry: some registries do not support arbitrary repo names #2816
Assignees
Labels
modules
Issues related to CUE modules and the experimental implementation
Comments
rogpeppe
added
NeedsInvestigation
Triage
cueckoo
pushed a commit
that referenced
this issue
Feb 16, 2024
In-the-wild registries can have significant restrictions on the names that can be stored. Here we implement a more general resolver that can be configured with a configuration file. The schema for that configuration file is held in internal/mod/modresolve/schema.cue; all the field names and arrangement of that configuration are provisional. The key property here is that a module is always stored with exactly the same blob and manifest files regardless of the registry, but the mapping from (module, version) to (repository, tag) can vary arbitrarily according to the configuration. The existing `modmux` registry multiplexer is now no longer sufficient, because there is no longer always a one-to-one relationship between modules and OCI repositories. So we change the layering so that the modregistry package directly knows about multiple registries, using the new `modresolve.Resolver` interface to tell it where modules live. The `modcache` package now takes a `*modregistry.Client` directly, which doesn't actually affect its core logic much at all because it only ever used that type internally anyway. We don't yet wire up the new configuration file logic inside `cmd/cue`: we'll do that in a subsequent change. For now, we verify that all the old behavior is still maintained despite the internal refactoring. As part of this work, also implement an `AllHosts` method that we can use to address the duplicate registry parsing logic in `cue login` by allowing that command to introspect all the hosts that need to be logged into. For #2816. Signed-off-by: Roger Peppe <rogpeppe@gmail.com> Change-Id: I5a5d7576c237e4844a0e7b3455cbdd781dc7fa19
cueckoo
pushed a commit
that referenced
this issue
Feb 16, 2024
In-the-wild registries can have significant restrictions on the names that can be stored. Here we implement a more general resolver that can be configured with a configuration file. The schema for that configuration file is held in internal/mod/modresolve/schema.cue; all the field names and arrangement of that configuration are provisional. The key property here is that a module is always stored with exactly the same blob and manifest files regardless of the registry, but the mapping from (module, version) to (repository, tag) can vary arbitrarily according to the configuration. The existing `modmux` registry multiplexer is now no longer sufficient, because there is no longer always a one-to-one relationship between modules and OCI repositories. So we change the layering so that the modregistry package directly knows about multiple registries, using the new `modresolve.Resolver` interface to tell it where modules live. The `modcache` package now takes a `*modregistry.Client` directly, which doesn't actually affect its core logic much at all because it only ever used that type internally anyway. We don't yet wire up the new configuration file logic inside `cmd/cue`: we'll do that in a subsequent change. For now, we verify that all the old behavior is still maintained despite the internal refactoring. As part of this work, also implement an `AllHosts` method that we can use to address the duplicate registry parsing logic in `cue login` by allowing that command to introspect all the hosts that need to be logged into. For #2816. Signed-off-by: Roger Peppe <rogpeppe@gmail.com> Change-Id: I5a5d7576c237e4844a0e7b3455cbdd781dc7fa19
cueckoo
pushed a commit
that referenced
this issue
Feb 16, 2024
In-the-wild registries can have significant restrictions on the names that can be stored. Here we implement a more general resolver that can be configured with a configuration file. The schema for that configuration file is held in internal/mod/modresolve/schema.cue; all the field names and arrangement of that configuration are provisional. The key property here is that a module is always stored with exactly the same blob and manifest files regardless of the registry, but the mapping from (module, version) to (repository, tag) can vary arbitrarily according to the configuration. The existing `modmux` registry multiplexer is now no longer sufficient, because there is no longer always a one-to-one relationship between modules and OCI repositories. So we change the layering so that the modregistry package directly knows about multiple registries, using the new `modresolve.Resolver` interface to tell it where modules live. The `modcache` package now takes a `*modregistry.Client` directly, which doesn't actually affect its core logic much at all because it only ever used that type internally anyway. We don't yet wire up the new configuration file logic inside `cmd/cue`: we'll do that in a subsequent change. For now, we verify that all the old behavior is still maintained despite the internal refactoring. As part of this work, also implement an `AllHosts` method that we can use to address the duplicate registry parsing logic in `cue login` by allowing that command to introspect all the hosts that need to be logged into. Also update the `ociregistry` dependency to the latest version so we can use the new `ociref.IsValid*` functions. For #2816. It provides a way to fix (most of) that specific issue by using the `hashAsPath` path encoding, which avoids the restriction on the number of path elements in a repository. It does not address #2817. Signed-off-by: Roger Peppe <rogpeppe@gmail.com> Change-Id: I5a5d7576c237e4844a0e7b3455cbdd781dc7fa19
cueckoo
pushed a commit
that referenced
this issue
Feb 16, 2024
In-the-wild registries can have significant restrictions on the names that can be stored. Here we implement a more general resolver that can be configured with a configuration file. The schema for that configuration file is held in internal/mod/modresolve/schema.cue; all the field names and arrangement of that configuration are provisional. The key property here is that a module is always stored with exactly the same blob and manifest files regardless of the registry, but the mapping from (module, version) to (repository, tag) can vary arbitrarily according to the configuration. The existing `modmux` registry multiplexer is now no longer sufficient, because there is no longer always a one-to-one relationship between modules and OCI repositories. So we change the layering so that the modregistry package directly knows about multiple registries, using the new `modresolve.Resolver` interface to tell it where modules live. The `modcache` package now takes a `*modregistry.Client` directly, which doesn't actually affect its core logic much at all because it only ever used that type internally anyway. We don't yet wire up the new configuration file logic inside `cmd/cue`: we'll do that in a subsequent change. For now, we verify that all the old behavior is still maintained despite the internal refactoring. As part of this work, also implement an `AllHosts` method that we can use to address the duplicate registry parsing logic in `cue login` by allowing that command to introspect all the hosts that need to be logged into. Also update the `ociregistry` dependency to the latest version so we can use the new `ociref.IsValid*` functions. For #2816. It provides a way to fix (most of) that specific issue by using the `hashAsPath` path encoding, which avoids the restriction on the number of path elements in a repository. It does not address #2817. Signed-off-by: Roger Peppe <rogpeppe@gmail.com> Change-Id: I5a5d7576c237e4844a0e7b3455cbdd781dc7fa19
cueckoo
pushed a commit
that referenced
this issue
Feb 19, 2024
In-the-wild registries can have significant restrictions on the names that can be stored. Here we implement a more general resolver that can be configured with a configuration file. The schema for that configuration file is held in internal/mod/modresolve/schema.cue; all the field names and arrangement of that configuration are provisional. The key property here is that a module is always stored with exactly the same blob and manifest files regardless of the registry, but the mapping from (module, version) to (repository, tag) can vary arbitrarily according to the configuration. The existing `modmux` registry multiplexer is now no longer sufficient, because there is no longer always a one-to-one relationship between modules and OCI repositories. So we change the layering so that the modregistry package directly knows about multiple registries, using the new `modresolve.Resolver` interface to tell it where modules live. The `modcache` package now takes a `*modregistry.Client` directly, which doesn't actually affect its core logic much at all because it only ever used that type internally anyway. We don't yet wire up the new configuration file logic inside `cmd/cue`: we'll do that in a subsequent change. For now, we verify that all the old behavior is still maintained despite the internal refactoring. As part of this work, also implement an `AllHosts` method that we can use to address the duplicate registry parsing logic in `cue login` by allowing that command to introspect all the hosts that need to be logged into. Also update the `ociregistry` dependency to the latest version so we can use the new `ociref.IsValid*` functions. For #2816. It provides a way to fix (most of) that specific issue by using the `hashAsPath` path encoding, which avoids the restriction on the number of path elements in a repository. It does not address #2817. Signed-off-by: Roger Peppe <rogpeppe@gmail.com> Change-Id: I5a5d7576c237e4844a0e7b3455cbdd781dc7fa19
cueckoo
pushed a commit
that referenced
this issue
Feb 19, 2024
In-the-wild registries can have significant restrictions on the names that can be stored. Here we implement a more general resolver that can be configured with a configuration file. The schema for that configuration file is held in internal/mod/modresolve/schema.cue; all the field names and arrangement of that configuration are provisional. The key property here is that a module is always stored with exactly the same blob and manifest files regardless of the registry, but the mapping from (module, version) to (repository, tag) can vary arbitrarily according to the configuration. The existing `modmux` registry multiplexer is now no longer sufficient, because there is no longer always a one-to-one relationship between modules and OCI repositories. So we change the layering so that the modregistry package directly knows about multiple registries, using the new `modresolve.Resolver` interface to tell it where modules live. The `modcache` package now takes a `*modregistry.Client` directly, which doesn't actually affect its core logic much at all because it only ever used that type internally anyway. We don't yet wire up the new configuration file logic inside `cmd/cue`: we'll do that in a subsequent change. For now, we verify that all the old behavior is still maintained despite the internal refactoring. As part of this work, also implement an `AllHosts` method that we can use to address the duplicate registry parsing logic in `cue login` by allowing that command to introspect all the hosts that need to be logged into. Also update the `ociregistry` dependency to the latest version so we can use the new `ociref.IsValid*` functions. For #2816. It provides a way to fix (most of) that specific issue by using the `hashAsPath` path encoding, which avoids the restriction on the number of path elements in a repository. It does not address #2817. Signed-off-by: Roger Peppe <rogpeppe@gmail.com> Change-Id: I5a5d7576c237e4844a0e7b3455cbdd781dc7fa19
cueckoo
pushed a commit
that referenced
this issue
Feb 19, 2024
In-the-wild registries can have significant restrictions on the names that can be stored. Here we implement a more general resolver that can be configured with a configuration file. The schema for that configuration file is held in internal/mod/modresolve/schema.cue; all the field names and arrangement of that configuration are provisional. The key property here is that a module is always stored with exactly the same blob and manifest files regardless of the registry, but the mapping from (module, version) to (repository, tag) can vary arbitrarily according to the configuration. The existing `modmux` registry multiplexer is now no longer sufficient, because there is no longer always a one-to-one relationship between modules and OCI repositories. So we change the layering so that the modregistry package directly knows about multiple registries, using the new `modresolve.Resolver` interface to tell it where modules live. The `modcache` package now takes a `*modregistry.Client` directly, which doesn't actually affect its core logic much at all because it only ever used that type internally anyway. We don't yet wire up the new configuration file logic inside `cmd/cue`: we'll do that in a subsequent change. For now, we verify that all the old behavior is still maintained despite the internal refactoring. As part of this work, also implement an `AllHosts` method that we can use to address the duplicate registry parsing logic in `cue login` by allowing that command to introspect all the hosts that need to be logged into. Also update the `ociregistry` dependency to the latest version so we can use the new `ociref.IsValid*` functions. For #2816. It provides a way to fix (most of) that specific issue by using the `hashAsPath` path encoding, which avoids the restriction on the number of path elements in a repository. It does not address #2817. Signed-off-by: Roger Peppe <rogpeppe@gmail.com> Change-Id: I5a5d7576c237e4844a0e7b3455cbdd781dc7fa19 Reviewed-on: https://review.gerrithub.io/c/cue-lang/cue/+/1176978 Reviewed-by: Paul Jolly <paul@myitcv.io> TryBot-Result: CUEcueckoo <cueckoo@gmail.com> Unity-Result: CUE porcuepine <cue.porcuepine@gmail.com> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
cueckoo
pushed a commit
that referenced
this issue
Feb 19, 2024
This hooks up the support for registry configuration added in https://cuelang.org/cl/1176978 to the `cue` command. This allows a user to affect how modules are stored in a custom registry, allowing them to work around some common restrictions such as repository name depth or availability. Fixes #2816. Signed-off-by: Roger Peppe <rogpeppe@gmail.com> Change-Id: Id76337bcdda9103c84ccb23b372161c1f62d6da3
cueckoo
pushed a commit
that referenced
this issue
Feb 19, 2024
This hooks up the support for registry configuration added in https://cuelang.org/cl/1176978 to the `cue` command. This allows a user to affect how modules are stored in a custom registry, allowing them to work around some common restrictions such as repository name depth or availability. Fixes #2816. Signed-off-by: Roger Peppe <rogpeppe@gmail.com> Change-Id: Id76337bcdda9103c84ccb23b372161c1f62d6da3
cueckoo
pushed a commit
that referenced
this issue
Feb 19, 2024
This hooks up the support for registry configuration added in https://cuelang.org/cl/1176978 to the `cue` command. This allows a user to affect how modules are stored in a custom registry, allowing them to work around some common restrictions such as repository name depth or availability. We still need to document this format in the help, but that's substantial enough that we'll leave it for another CL. Fixes #2816. Signed-off-by: Roger Peppe <rogpeppe@gmail.com> Change-Id: Id76337bcdda9103c84ccb23b372161c1f62d6da3
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What version of CUE are you using (
cue version)?Does this issue reproduce with the latest stable release?
No. The feature didn't exist then,
What did you do?
(with "xxxx" replaced with my actual personal access token):
What did you expect to see?
A passing test.
What did you see instead?
Although the response says "unauthorized", the actual reason is that the naming restriction documented here has not been followed.
If I try with a module name with fewer components, it gets further:
This is how it fails then:
It turns out that gitlab does not support unknown artifact types for historical reasons (see https://gitlab.com/gitlab-org/container-registry/-/issues/973). If CUE is changed to avoid the custom artifact type, the test above passes.
The text was updated successfully, but these errors were encountered: