Support publishing staging directories to subdirectories of external repos

[andrewsykim]

As part of the cloud provider removal efforts, it may be beneficial for us to support publishing staged directories to only subdirectories of external repos. This gives us the flexibility to publish the cloud provider implementations developed in k8s.io/kubernetes into their respective external repos without forcing the entire repository to reflect what is in staging.

An example of this would be to publish k8s.io/kubernetes/staging/src/k8s.io/cloud-provider/gce to k8s.io/cloud-provider-gce/provider/. This would keep only k8s.io/cloud-provider-gce/provider/ in sync with what is staged but allow us to freely update any other directory in that repo.

Happy to pick up any work required for this (will need some guidance though).

cc @nikhita @cheftako @mcrute @dims

[sttts]

@andrewrynhard we cannot change the package hierarchy on the fly when publishing, e.g. have it called k8s.io/cloud-provider/gce in-tree and k8s.io/cloud-provider/provider/gce out-of-tree. We can tough have k8s.io/cloud-provider/provider/gce in-tree, publish it under the same name, but have other non-staging code outside of k8s.io/cloud-provider/provider. The only crucial restriction is that nobody ever changes k8s.io/cloud-provider/provider outside, in the published repository. But changes to k8s.io/cloud-provider/foo/bar are no problem.

Does that help your use-caes?

[andrewsykim]

Yes! That should work. So to be clear, we can publish k8s.io/kubernetes/staging/src/k8s.io/cloud-provider-gce/provider/gcp -> k8s.io/cloud-provider-gcp/provider/gce assuming that no one manually pushes changes to k8s.io/cloud-provider-gcp/provider. But the rest of the repository can be pushed from k8s.io/cloud-provider-gcp directly.

[sttts]

@andrewrynhard yes. The other exception is Godeps. Never touch that directory from outside.

[sttts]

One important point: the bot updates Godeps/Godeps.json (today) and go.mod (tomorrow) at the root of the repository. I.e. this will have to cover the whole repository.

I assume you will put other Go code in other directories, which will have to be covered by go.mod as well.

Go.mod also has to be in the k/k repo. But there it will be only partial. Also hack/update-vendor.sh will call go mod tidy to make it minimal.

How are you planning to solve this?

[andrewsykim]

Sorry in advance, I'm still learning the details around go modules. Since go modules supports nested directories, would having a go.mod file in the legacy provider (staging/src/k8s.io/cloud-provider-*/legacyprovider) address this sufficiently or will this not work since there are conflicting dependencies with the same import paths? Alternatively, if the external repo is using another dependency management tool couldn't Godeps & go modules be ignored in the meantime?

[sttts]

would having a go.mod file in the legacy provider (staging/src/k8s.io/cloud-provider-*/legacyprovider) address this sufficiently

Technically yes. But the bot currently does not support maintaining a subdirectory. It expects go.mod at the top-level.

[sttts]

Alternatively, if the external repo is using another dependency management tool couldn't Godeps & go modules be ignored in the meantime?

Could work, if nobody is confused for Gopkg and go.mod (and Godeps/Godeps.json which we create for compatibility) :-)

[andrewsykim]

Technically yes. But the bot currently does not support maintaining a subdirectory. It expects go.mod at the top-level.

Assuming it's possible, I'm willing to do the work to make this happen with a bit of guidance. Any objections?

[sttts]

@andrewsykim here you have to add a gomod-dir parameter everywhere: https://github.com/kubernetes/publishing-bot/blob/master/artifacts/scripts/util.sh

[andrewsykim]

/close

Don't think we'll need this for now, will re-open if we do

[k8s-ci-robot]

@andrewsykim: Closing this issue.

In response to this:

/close

Don't think we'll need this for now, will re-open if we do

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.