k8s.io/code-generator scripts do not work from repos that use new Go modules

[sgnn7]

When invoking generate-groups.sh from a repo that uses the new Golang v1.11 modules, none of the codegens work as they assume that the repo layout has the code in $GOPATH/src/k8s.io/code-generator and that dependencies have been downloaded for this project (not doable using go mod easily).

Steps to reproduce:

$ docker run -it --rm golang:1.11rc1 /bin/bash
root@5f5497fb59c9:/go# mkdir /foo && cd /foo && go mod init foo
go: creating new go.mod: module foo

root@5f5497fb59c9:/foo# go get -u k8s.io/code-generator
go: finding k8s.io/code-generator latest
go: downloading k8s.io/code-generator v0.0.0-20180806210827-caff7734d7c0

root@5f5497fb59c9:/foo# find / -name 'generate-groups.sh'
/go/pkg/mod/k8s.io/code-generator@v0.0.0-20180806210827-caff7734d7c0/generate-groups.sh

root@5f5497fb59c9:/foo# /go/pkg/mod/k8s.io/code-generator@v0.0.0-20180806210827-caff7734d7c0/generate-groups.sh all foo foo bar
bash: /go/pkg/mod/k8s.io/code-generator@v0.0.0-20180806210827-caff7734d7c0/generate-groups.sh: Permission denied

Changing the executable flag doesn't help as the script goes into that directory and runs go install which in this case cannot complete due to this module not having installable dependencies with go mod:

root@5f5497fb59c9:/foo# chmod +x /go/pkg/mod/k8s.io/code-generator@v0.0.0-20180806210827-caff7734d7c0/generate-groups.sh

root@5f5497fb59c9:/foo# /go/pkg/mod/k8s.io/code-generator@v0.0.0-20180806210827-caff7734d7c0/generate-groups.sh all foo foo bar
cmd/defaulter-gen/main.go:48:2: cannot find package "github.com/golang/glog" in any of:
        /usr/local/go/src/github.com/golang/glog (from $GOROOT)
        /go/src/github.com/golang/glog (from $GOPATH)
cmd/defaulter-gen/main.go:49:2: cannot find package "github.com/spf13/pflag" in any of:
        /usr/local/go/src/github.com/spf13/pflag (from $GOROOT)
        /go/src/github.com/spf13/pflag (from $GOPATH)
cmd/defaulter-gen/main.go:50:2: cannot find package "k8s.io/gengo/args" in any of:
        /usr/local/go/src/k8s.io/gengo/args (from $GOROOT)
        /go/src/k8s.io/gengo/args (from $GOPATH)
cmd/defaulter-gen/main.go:51:2: cannot find package "k8s.io/gengo/examples/defaulter-gen/generators" in any of:
        /usr/local/go/src/k8s.io/gengo/examples/defaulter-gen/generators (from $GOROOT)
        /go/src/k8s.io/gengo/examples/defaulter-gen/generators (from $GOPATH)
cmd/deepcopy-gen/main.go:52:2: cannot find package "k8s.io/gengo/examples/deepcopy-gen/generators" in any of:
        /usr/local/go/src/k8s.io/gengo/examples/deepcopy-gen/generators (from $GOROOT)
        /go/src/k8s.io/gengo/examples/deepcopy-gen/generators (from $GOPATH)

It seems that this can get further (until full go modules support) by adding a go.mod to the base of this repo and making the script executable but even then, the pathing is broken in these:

root@26e1c7ae81c1:/foo# cd ..                                      
root@26e1c7ae81c1:/# git clone https://github.com/kubernetes/code-generator.git
Cloning into 'code-generator'...                                            
...     

root@26e1c7ae81c1:/# cd code-generator/

root@26e1c7ae81c1:/code-generator# go mod init                           
go: creating new go.mod: module k8s.io/code-generator                        
go: copying requirements from Godeps/Godeps.json

root@26e1c7ae81c1:/code-generator# cd /foo                             
root@26e1c7ae81c1:/foo# chmod +x /code-generator/generate-groups.sh

root@26e1c7ae81c1:/foo# /code-generator/generate-groups.sh all foo foo bar
go: finding github.com/PuerkitoBio/urlesc v0.0.0-20160726150825-5bd2802263f2
...
go: downloading golang.org/x/tools v0.0.0-20170428054726-2382e3994d48
Generating deepcopy funcs
F0817 20:44:40.912251     845 main.go:81] Error: Failed making a parser: unable to add directory "-O": unable to import "-O": go/build: importGo -O: exit status 2
flag provided but not defined: -O
usage: go list [-f format] [-json] [-m] [list flags] [build flags] [packages]
Run 'go help list' for details.

/kind bug
/sig api-machinery

[sgnn7]

Another sidenote - seems like output and input prefixes for these generators hardcode <org>/<repo>/<path> after them so if you have the new go modules in use and your repository is outside of $GOPATH, you cannot produce as-is runnable code with them.

[yue9944882]

/remove-kind bug
/kind feature

thx for reporting. at least for now plz manage dependencies by godep.

It seems that this can get further (until full go modules support) by adding a go.mod

i'm afraid release-1.12 won't provide support for golang-1.11 features but it sounds good. hopefully we'll consider it in the next release. 😉

[yue9944882]

/cc @sttts

[lavalamp]

We need to figure out a module strategy for the whole project. See: #63607

I won't mark this a duplicate because it's targeted at a specific subset of the problem.

[perryao]

@sgnn7 is there a workaround you know of for using code-generator tools in a go mod project?

[sgnn7]

@perryao Yes but it requires a few changes and all the changes are out-of-tree. You can see how we use this here.

Sadly my PRs for getting some of this in-tree have either been closed or struck in review. Let me know if you have any questions.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gdoctor]

/reopen

Is this supported yet, and if so, is there any documentation on how to use the new functionality? Linking here before closing would be of great help to the community.

[k8s-ci-robot]

@gdoctor: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Is this supported yet, and if so, is there any documentation on how to use the new functionality? Linking here before closing would be of great help to the community.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kragniz]

/reopen

This still isn't addressed.

[k8s-ci-robot]

@kragniz: Reopened this issue.

In response to this:

/reopen

This still isn't addressed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nikhita]

FYI the way to use code-generator right now is: https://github.com/kubernetes/sample-apiserver/#when-using-go-111-modules

[ialidzhikov]

I have no success with k8s.io/code-generator for go modules projects located outside of $GOPATH.
I described my findings in new issue - ref #86753.

[dvaldivia]

@nikhita has the way to use code-generator changed at all? does it support go modules? I see apimachinery already does