Code-gen: Remove lowercasing for project imports #68484

jsturtevant · 2018-09-10T19:28:30Z

What this PR does / why we need it:
If a project has a uppercase letter in the project import path then the generated paths will end up in different directories when some of the files are lower-cased during generation. This PR fixes the few places where the imports are lower-cased.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes kubernetes/code-generator#20

Special notes for your reviewer:
There was another PR (#63892) that was started but closed with no explanation. It looks like it was missing a few spots area's where the lower casing was missed which are addressed in this PR.

I was not sure where or if there are tests for this section of code. Would be happy to add tests if you can point me in the right direction.

If you run the code-generator/generate-groups.sh against the project here you can see the lower-casing in action. The generated code is split across two directories and the project does not build but with these fixes code-generation completes as expected and the project builds.

Release note:

Code-gen: Remove lowercasing for project imports

jsturtevant · 2018-09-10T19:33:59Z

/assign @caesarxuchao

yue9944882 · 2018-09-11T03:08:21Z

/cc @kubernetes/code-generator-maintainers

caesarxuchao · 2018-09-11T22:41:38Z

/ok-to-test

jsturtevant · 2018-09-13T00:02:04Z

/retest

jsturtevant · 2018-09-13T20:31:22Z

@caesarxuchao I was able to pass the tests with a few updates.

I started to add an example of the Uppercasing in _examples like the previous PR. The generation works as expected (I haven't commited this yet).

But during that further testing I notice that the verify tests don't actually run the smoke tests in the verify script. From the test output of this PR:

I0913 00:03:18.965] Smoke testing _example by compiling...
I0913 00:03:19.008] Building client-gen
W0913 00:03:19.108] warning: "vendor/k8s.io/code-generator/_example/..." matched no packages
W0913 00:03:19.109] warning: "vendor/k8s.io/code-generator/_example/..." matched no packages

There _example folder looks to have been renamed to _examples at some point. I made an update to build the projects by updating staging/src/k8s.io/code-generator/hack/verify-codegen.sh to build the samples for example:

go build ${SCRIPT_ROOT}/_examples/crd/...

but now I get the error:

./hack/verify-codegen.sh
Generating deepcopy funcs
Generating defaulters
Generating conversions
Generating clientset for example:v1 example2:v1 at k8s.io/code-generator/_examples/apiserver/clientset
Generating listers for example:v1 example2:v1 at k8s.io/code-generator/_examples/apiserver/listers
Generating informers for example:v1 example2:v1 at k8s.io/code-generator/_examples/apiserver/informers
Generating deepcopy funcs
Generating clientset for example:v1 example2:v1 at k8s.io/code-generator/_examples/crd/clientset
Generating listers for example:v1 example2:v1 at k8s.io/code-generator/_examples/crd/listers
Generating informers for example:v1 example2:v1 at k8s.io/code-generator/_examples/crd/informers
Generating deepcopy funcs
Generating clientset for example:v1 at k8s.io/code-generator/_examples/UpperCaseProjectName/clientset
Generating listers for example:v1 at k8s.io/code-generator/_examples/UpperCaseProjectName/listers
Generating informers for example:v1 at k8s.io/code-generator/_examples/UpperCaseProjectName/informers
diffing ./hack/../_examples against freshly generated codegen
./hack/../_examples up to date.
Smoke testing _example by compiling...
./hack/..
../../../../vendor/k8s.io/code-generator/_examples/crd/clientset/versioned/typed/example/v1/clustertesttype.go:28:2: cannot find package "k8s.io/kubernetes/pkg/apis/autoscaling" in any of:
	/home/jstur/go/src/github.com/kubernetes/kubernetes/vendor/k8s.io/kubernetes/pkg/apis/autoscaling (vendor tree)
	/usr/local/go/src/k8s.io/kubernetes/pkg/apis/autoscaling (from $GOROOT)
	/home/jstur/go/src/k8s.io/kubernetes/pkg/apis/autoscaling (from $GOPATH)

Since the smoke tests have not been running it seems this has been an issue before this PR as you can see on this file in master. Looking at the vendor folder it seems that the all those packages are symlinked back into the main pkg folder. Although none of them have the prefix kubernetes. It seems that there might be another bug related to how those autoscaling imports are being generated.

Thoughts on how to proceed? I could check in the samples folder with the Uppercase as an example, get this merged and closed, then open a separate bug for the re-enabling the smoke tests and fixing the generating of the autoscaling package? What do you think?

jsturtevant · 2018-09-21T15:33:02Z

@caesarxuchao all the tests pass is this good to merge? I can open issues for the comments above separately.

caesarxuchao

Can you add a test demonstrating the client-gen works for camelcased path now?

staging/src/k8s.io/code-generator/cmd/client-gen/generators/client_generator.go

jsturtevant · 2018-09-24T22:36:35Z

I added an example with MixedCase. Also needed to update the verify script to build the packages correctly as stated in my previous comment.

Once the example projects were building properly, this uncovered a bug in the code generation for the sub resource scale in the existing crd example:

Smoke testing _example by compiling...
# k8s.io/kubernetes/vendor/k8s.io/code-generator/_examples/crd/clientset/versioned/typed/example/v1/fake
vendor/k8s.io/code-generator/_examples/crd/clientset/versioned/typed/example/v1/fake/fake_clustertesttype.go:137:46: not enough arguments in call to testing.NewRootGetSubresourceAction
	have (schema.GroupVersionResource, string)
	want (schema.GroupVersionResource, string, string)

I fixed this in the generator_fake_for_type.go by adding the correct parameters. Everything builds nows. I also tested the updates against my larger project that prompted this bug fix as mentioned in the initial PR.

jsturtevant · 2018-09-25T16:02:22Z

@caesarxuchao the latest commits passed the tests. Ready for review. Thanks!

caesarxuchao · 2018-09-25T22:33:42Z

LGTM. Thanks. Can you squash it to meaningful commits?

This commit provides a fix for the scenario where a project has an uppercase letter in the project import path. Prior to this fix the generated files would end up in different directories with some of the imports being lower-cased during generation. An example of this would be a project such as 'github.com/MixedCase/project' would result in some of the imports with 'github.com/mixedcase/project' causing a broken build.

The smoke tests were not being run for the example projects. Re-enabled the smoke tests by building each of the sample projects.

The code generation for fake types was missing the subresource path parameter in the template which caused a compile error for the sample projects using the scale subresource. Also re-ran the code generation after applying the fix.

jsturtevant · 2018-09-26T21:13:44Z

@caesarxuchao squashed commits and tests passed. Looks ready to go. Thanks again!

caesarxuchao · 2018-09-26T21:45:32Z

/lgtm

caesarxuchao · 2018-09-26T22:08:25Z

/assign @lavalamp

for approval

nikhita · 2018-09-29T21:48:53Z

/assign @sttts
/kind cleanup

sttts · 2018-10-05T09:19:35Z

staging/src/k8s.io/code-generator/hack/verify-codegen.sh

+go build ./${SCRIPT_ROOT}/_examples/crd/...
+go build ./${SCRIPT_ROOT}/_examples/apiserver/...
+go build ./${SCRIPT_ROOT}/_examples/MixedCase/...


sttts · 2018-10-05T09:19:54Z

/lgtm
/approve

k8s-ci-robot · 2018-10-05T09:20:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: caesarxuchao, jsturtevant, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/code-generator/OWNERS~~ [sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

The PR (kubernetes/kubernetes#68484) was merged into upstream kubernetes. This commit removes the temporary fix for lowercasing. The latest version of the code gen has an update to the patch client that use a parameter that is not in current version of Go-Client used by this repo so dropped code gen for patching. Patching is not being used currently so safe to drop. May consider dropping other verbs not used in the client in future.

k8s-ci-robot requested review from caesarxuchao and wojtek-t September 10, 2018 19:28

k8s-ci-robot assigned caesarxuchao Sep 10, 2018

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 11, 2018

caesarxuchao mentioned this pull request Sep 11, 2018

Remove lowercasing the import path for the first part of import. #63892

Closed

jsturtevant force-pushed the fix-lowercased-imports branch from fd7217f to b692073 Compare September 12, 2018 23:21

caesarxuchao reviewed Sep 21, 2018

View reviewed changes

staging/src/k8s.io/code-generator/cmd/client-gen/generators/client_generator.go Outdated Show resolved Hide resolved

jsturtevant force-pushed the fix-lowercased-imports branch from a7da549 to 86ff5f8 Compare September 24, 2018 22:28

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Sep 24, 2018

jsturtevant and others added 2 commits September 26, 2018 10:59

Add MixedCase example project

5d7bd65

jsturtevant added 2 commits September 26, 2018 10:59

Re-enable smoke tests in verify-codegen script

798bd34

The smoke tests were not being run for the example projects. Re-enabled the smoke tests by building each of the sample projects.

Fix code generation by adding missing parameter

93d8a53

The code generation for fake types was missing the subresource path parameter in the template which caused a compile error for the sample projects using the scale subresource. Also re-ran the code generation after applying the fix.

jsturtevant force-pushed the fix-lowercased-imports branch from 86ff5f8 to 93d8a53 Compare September 26, 2018 18:11

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 26, 2018

k8s-ci-robot assigned lavalamp Sep 26, 2018

k8s-ci-robot assigned sttts Sep 29, 2018

k8s-ci-robot added kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Sep 29, 2018

sttts reviewed Oct 5, 2018

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 5, 2018

k8s-ci-robot merged commit 5b9eb1c into kubernetes:master Oct 5, 2018

jsturtevant mentioned this pull request Oct 17, 2018

Remove temporary fix for lower case code gen Azure/azure-k8s-metrics-adapter#41

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Code-gen: Remove lowercasing for project imports #68484

Code-gen: Remove lowercasing for project imports #68484

jsturtevant commented Sep 10, 2018

jsturtevant commented Sep 10, 2018

yue9944882 commented Sep 11, 2018

caesarxuchao commented Sep 11, 2018

jsturtevant commented Sep 13, 2018

jsturtevant commented Sep 13, 2018

jsturtevant commented Sep 21, 2018

caesarxuchao left a comment

jsturtevant commented Sep 24, 2018

jsturtevant commented Sep 25, 2018

caesarxuchao commented Sep 25, 2018

jsturtevant commented Sep 26, 2018

caesarxuchao commented Sep 26, 2018

caesarxuchao commented Sep 26, 2018

nikhita commented Sep 29, 2018

sttts Oct 5, 2018

sttts commented Oct 5, 2018

k8s-ci-robot commented Oct 5, 2018

Code-gen: Remove lowercasing for project imports #68484

Code-gen: Remove lowercasing for project imports #68484

Conversation

jsturtevant commented Sep 10, 2018

jsturtevant commented Sep 10, 2018

yue9944882 commented Sep 11, 2018

caesarxuchao commented Sep 11, 2018

jsturtevant commented Sep 13, 2018

jsturtevant commented Sep 13, 2018

jsturtevant commented Sep 21, 2018

caesarxuchao left a comment

Choose a reason for hiding this comment

jsturtevant commented Sep 24, 2018

jsturtevant commented Sep 25, 2018

caesarxuchao commented Sep 25, 2018

jsturtevant commented Sep 26, 2018

caesarxuchao commented Sep 26, 2018

caesarxuchao commented Sep 26, 2018

nikhita commented Sep 29, 2018

sttts Oct 5, 2018

Choose a reason for hiding this comment

sttts commented Oct 5, 2018

k8s-ci-robot commented Oct 5, 2018