"Input contains a loop" on vendoring new dependencies that depend on staging projects

[dcantah]

What happened:

If you try and vendor in a dependency into kubernetes/kubernetes that depends on a select few of the projects that get published out of /staging (api or apimachinery that also happen to depend on each other), it seems the update-vendor.sh script whines that the input contains a loop:

tsort: /tmp/update-vendor.3JAS/tidy_deps.txt: input contains a loop:
tsort: k8s.io/api
tsort: k8s.io/apimachinery
An error has occurred. Please see more details in /tmp/update-vendor.3JAS/update-vendor.log

The error is that a certain package doesn't exist in the tag I was using but did locally. This is on tag 0.20.6 for k8s.io/api. With tag 0.22.0 the error goes away, but the "input contains a loop" remains.

=== tidying root
go: finding module for package k8s.io/api/discovery/v1
go: finding module for package k8s.io/api/policy/v1
k8s.io/kubernetes/pkg/apis/discovery/v1 imports
        k8s.io/api/discovery/v1: module k8s.io/api@latest found (v0.22.1, replaced by k8s.io/api@v0.20.6), but does not contain package k8s.io/api/discovery/v1
k8s.io/kubernetes/pkg/apis/policy/v1 imports
        k8s.io/api/policy/v1: module k8s.io/api@latest found (v0.22.1, replaced by k8s.io/api@v0.20.6), but does not contain package k8s.io/api/policy/v1

As the staging projects depend on each other at version 0.0.0 and use replace directives to actually depend on the code locally, when vendoring in a project that actually uses a real tag, all of these get updated which is definitely not what is desired regardless.

What you expected to happen:

Truthfully, I'm not sure. The go modules behavior isn't a shocker here, but I would've hoped this would be possible. I wouldn't imagine this scenario to be too outlandish, but maybe this is just a downside of the staging setup here.

How to reproduce it (as minimally and precisely as possible):

Have a project that depends on k8s.io/api (https://github.com/dcantah/deps or https://github.com/dcantah/deps2) for example and run:

hack/pin-dependency.sh github.com/dcantah/deps2 90b0ada1dfdb243bd29f9718f72615517a716c21
+
hack/update-vendor.sh

Anything else we need to know?:

This is mainly for tracking and trying to figure out if this has been a pain point for anything else. The answer may be to just try and trim the deps of the project trying to get vendored into here to not include api/apimachinery etc.

Where'd we'd ran into this is through trying to vendor in a new tag of https://github.com/microsoft/hcsshim that has a https://github.com/containerd/containerd v1.5.x dependency. As containerd depends on a lot of the staging projects, it will try and update all of these tags and then run into the stated error.

For more info on this: microsoft/hcsshim#1148

cc @dims @mikebrow @kevpar @jsturtevant @marosset

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[k8s-ci-robot]

@dcantah: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[palnabarun]

/area code-organization
/sig architecture

[liggitt]

Components that have a transitive dependency to anything in k8s.io/kubernetes, including the staging components, can not be added as a dependency. The detection and prevention of this in the dependency management scripts is intentional.

The reason is that it introduces a gridlock/chicken-and-egg scenario that couples the staging components to their dependencies in a way that makes it impossible to make updates until the dependencies update (which they can't do until the changes are made and released).

[liggitt]

The answer may be to just try and trim the deps of the project trying to get vendored into here to not include api/apimachinery etc.

That is indeed the solution

[dcantah]

@liggitt Ok, I appreciate the prompt response on this! Looks like we'll need to start trimming at the hcsshim/containerd layers.