Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix defaulter-gen for recursive types #61

Merged
merged 1 commit into from Jul 6, 2017

Conversation

@nikhita
Copy link
Member

commented Jun 9, 2017

For recursive types, the defaulter currently goes into an infinite recursion.
Fix to support recursive types.

/cc @sttts

@k8s-ci-robot k8s-ci-robot requested a review from sttts Jun 9, 2017

@k8s-reviewable

This comment has been minimized.

Copy link

commented Jun 9, 2017

This change is Reviewable

@@ -320,7 +320,8 @@ func Packages(context *generator.Context, arguments *args.GeneratorArgs) generat
if d.object != nil {
continue
}
if buildCallTreeForType(t, true, existingDefaulters, newDefaulters) != nil {
existingTypes := make(map[string]bool)

This comment has been minimized.

Copy link
@nikhita

nikhita Jun 9, 2017

Author Member

PTAL if this is the right way to do it.

@@ -396,7 +397,7 @@ func Packages(context *generator.Context, arguments *args.GeneratorArgs) generat
// that could be or will be generated. If newDefaulters has an entry for a type, but the 'object' field is nil,
// this function skips adding that defaulter - this allows us to avoid generating object defaulter functions for
// list types that call empty defaulters.
func buildCallTreeForType(t *types.Type, root bool, existingDefaulters, newDefaulters defaulterFuncMap) *callNode {
func buildCallTreeForType(t *types.Type, root bool, existingDefaulters, newDefaulters defaulterFuncMap, existingTypes map[string]bool) *callNode {

This comment has been minimized.

Copy link
@sttts

sttts Jun 9, 2017

Contributor

Can you document existingTypes? Is it feasible to have a recursive type in some test? Are there tests here in the generator?

This comment has been minimized.

Copy link
@nikhita

nikhita Jun 9, 2017

Author Member

Will document it.
Currently there are no tests. But I'll add one.

return parent
}
// if type doesn't exist, mark it as existing
existingTypes[t.Name.Name] = true

This comment has been minimized.

Copy link
@sttts

sttts Jun 9, 2017

Contributor

shouldn't we check that Name is actually files, i.e. not an anonymous type? Otherwise, we stop too early for those, don't we?

This comment has been minimized.

Copy link
@nikhita

nikhita Jun 9, 2017

Author Member

So I should check for t.Name?

This comment has been minimized.

Copy link
@sttts

sttts Jun 9, 2017

Contributor

If I read the type godoc correctly, Name can be empty.

@sttts

This comment has been minimized.

Copy link
Contributor

commented Jun 9, 2017

Can you link a commit for the Kubernetes defaulters? Do they change after this PR?

@nikhita nikhita referenced this pull request Jun 9, 2017
13 of 13 tasks complete

@nikhita nikhita changed the title Fix defaulter for recursive types [WIP] Fix defaulter for recursive types Jun 9, 2017

@thockin

This comment has been minimized.

Copy link
Member

commented Jun 10, 2017

@nikhita nikhita force-pushed the nikhita:fix-recursive-types branch from 331e754 to ba7ac3b Jun 11, 2017

@nikhita

This comment has been minimized.

Copy link
Member Author

commented Jun 11, 2017

Updated the defaulter code. PTAL.

I have added some tests. However, I come across the following error when I run make: https://gist.github.com/nikhita/379c3789b8049038109b1ea1509b483b. What could be the possible reason for this error?

It works when there is no defaults.go file though. I have also pushed the file generated in this case.

After making the changes in this PR and fixing the conversion in kubernetes/kubernetes#47263, nothing breaks/changes in Kubernetes (tested against HEAD). This PR shows the cumulative changes in both the defaulter and converter: nikhita/kubernetes#3.

@@ -0,0 +1,36 @@
/*
Copyright 2016 The Kubernetes Authors.

This comment has been minimized.

Copy link
@nikhita

nikhita Jun 11, 2017

Author Member

Oops, will fix the year after the review.

@nikhita nikhita force-pushed the nikhita:fix-recursive-types branch from 9929cff to ba7ac3b Jun 12, 2017

@nikhita

This comment has been minimized.

Copy link
Member Author

commented Jun 12, 2017

The tests are being added in #55. Will create a PR to add a test for recursive types on top of #55.

@nikhita nikhita changed the title [WIP] Fix defaulter for recursive types Fix defaulter-gen for recursive types Jun 12, 2017

@nikhita

This comment has been minimized.

Copy link
Member Author

commented Jun 12, 2017

Added tests on top of #55 here: caesarxuchao#1.

@sttts

This comment has been minimized.

Copy link
Contributor

commented Jun 12, 2017

if buildCallTreeForType(t, true, existingDefaulters, newDefaulters) != nil {
// existingTypes keeps track of types that have already been visited in the tree.
// This is used to avoid recursion for recursive types.
existingTypes := make(map[*types.Type]bool)

This comment has been minimized.

Copy link
@sttts

sttts Jul 3, 2017

Contributor

s/existingTypes/alreadyVisitedTypes/

This comment has been minimized.

Copy link
@nikhita

nikhita Jul 3, 2017

Author Member

Done.

@sttts

This comment has been minimized.

Copy link
Contributor

commented Jul 3, 2017

@wojtek-t can this go in? (you are in the OWNERs files, together with @lavalamp. Though the merge+review policy here is unclear to me looking at previous PRs).

@nikhita nikhita force-pushed the nikhita:fix-recursive-types branch from ba7ac3b to 5f0312b Jul 3, 2017


// The type now acts as a parent, not a nested recursive type.
// We can now build the tree for it safely.
alreadyVisitedTypes[t] = false

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 5, 2017

Member

The name is wrong if you set it to false after you are done. Should be "currentlyBuildingTypes" or "typesInProgress" or something.

@@ -396,7 +399,7 @@ func Packages(context *generator.Context, arguments *args.GeneratorArgs) generat
// that could be or will be generated. If newDefaulters has an entry for a type, but the 'object' field is nil,
// this function skips adding that defaulter - this allows us to avoid generating object defaulter functions for
// list types that call empty defaulters.
func buildCallTreeForType(t *types.Type, root bool, existingDefaulters, newDefaulters defaulterFuncMap) *callNode {
func buildCallTreeForType(t *types.Type, root bool, existingDefaulters, newDefaulters defaulterFuncMap, alreadyVisitedTypes map[*types.Type]bool) *callNode {

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 5, 2017

Member

Consider making a struct to hold all of the parameters that don't change, and make this function a method on that struct. Will make the code a lot easier to read.

@lavalamp

This comment has been minimized.

Copy link
Member

commented Jul 5, 2017

(happy to lgtm & merge after those comments are addressed)

@nikhita nikhita force-pushed the nikhita:fix-recursive-types branch from 5f0312b to ce68359 Jul 6, 2017

@nikhita

This comment has been minimized.

Copy link
Member Author

commented Jul 6, 2017

@lavalamp Updated.

@lavalamp
Copy link
Member

left a comment

Sorry for not being clear but that wasn't quite what I had in mind.

@@ -387,6 +391,25 @@ func Packages(context *generator.Context, arguments *args.GeneratorArgs) generat
return packages
}

// callTreeForType contains fields necessary to call buildCallTreeForType.
type callTreeForType struct {
t *types.Type

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 6, 2017

Member

remove; this struct should have only the immutable parameters.

// callTreeForType contains fields necessary to call buildCallTreeForType.
type callTreeForType struct {
t *types.Type
root bool

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 6, 2017

Member

remove; this struct should have only the immutable parameters.

currentlyBuildingTypes map[*types.Type]bool
}

func NewCallTreeForType(t *types.Type, root bool, existingDefaulters, newDefaulters defaulterFuncMap, currentlyBuildingTypes map[*types.Type]bool) callTreeForType {

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 6, 2017

Member

Delete this or change to:

func NewCallTreeForType(existingDefaulters, newDefaulters defaulterFuncMap) *callTreeForType {
  return &callTreeForType{
    existingDefaulters: existingDefaulters,
    newDefaulters: newDefaulters,
    currentlyBuildingTypes: make(map[*types.Type]bool),
  }
}

This comment has been minimized.

Copy link
@nikhita

nikhita Jul 6, 2017

Author Member

We can't initialize currentlyBuildingTypes in newCallTreeForType because when this is encountered:

r := newCallTreeForType(c.existingDefaulters, c.newDefaulters)
	if child := r.build(t.Elem, false); child != nil

it will create a new map for currentlyBuildingTypes - destroying our purpose for storing the types that are currently building. :)

case types.Pointer:
if child := buildCallTreeForType(t.Elem, false, existingDefaulters, newDefaulters); child != nil {
r := NewCallTreeForType(c.t.Elem, false, c.existingDefaulters, c.newDefaulters, c.currentlyBuildingTypes)
if child := r.buildCallTreeForType(); child != nil {

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 6, 2017

Member

All of these should look like if child := r.Build(c.t.Elem, false); child != nil {

@@ -396,22 +419,22 @@ func Packages(context *generator.Context, arguments *args.GeneratorArgs) generat
// that could be or will be generated. If newDefaulters has an entry for a type, but the 'object' field is nil,
// this function skips adding that defaulter - this allows us to avoid generating object defaulter functions for
// list types that call empty defaulters.
func buildCallTreeForType(t *types.Type, root bool, existingDefaulters, newDefaulters defaulterFuncMap) *callNode {
func (c callTreeForType) buildCallTreeForType() *callNode {

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 6, 2017

Member

func (c *callTreeForType) Build(t *types.Type, root bool) *callNode {

// This is used to avoid recursion for recursive types.
currentlyBuildingTypes := make(map[*types.Type]bool)
c := NewCallTreeForType(t, true, existingDefaulters, newDefaulters, currentlyBuildingTypes)
if c.buildCallTreeForType() != nil {

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 6, 2017

Member

This should be a one-liner:

if NewCallTreeForType(existingDefaulters, newDefaulters).Build(t, true) != nil {

 

currentlyBuildingTypes map[*types.Type]bool
}

func NewCallTreeForType(t *types.Type, root bool, existingDefaulters, newDefaulters defaulterFuncMap, currentlyBuildingTypes map[*types.Type]bool) callTreeForType {

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 6, 2017

Member

Actually both this and Build should be private.

@nikhita nikhita force-pushed the nikhita:fix-recursive-types branch from ce68359 to 644e318 Jul 6, 2017

@nikhita

This comment has been minimized.

Copy link
Member Author

commented Jul 6, 2017

@lavalamp Updated. PTAL, thanks.

switch t.Kind {
case types.Pointer:
if child := buildCallTreeForType(t.Elem, false, existingDefaulters, newDefaulters); child != nil {
r := newCallTreeForType(c.existingDefaulters, c.newDefaulters, c.currentlyBuildingTypes)

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 6, 2017

Member

What I was trying to communicate is that you don't call newCallTreeForType here, you reuse c.

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 6, 2017

Member

I think I wrote an r where I meant a c--sorry about that.

This comment has been minimized.

Copy link
@nikhita

nikhita Jul 6, 2017

Author Member

Right, sorry - should have done this in the first place.

Updated. Please check if this is better.

@nikhita nikhita force-pushed the nikhita:fix-recursive-types branch from 644e318 to 9a0932b Jul 6, 2017


// The type now acts as a parent, not a nested recursive type.
// We can now build the tree for it safely.
c.currentlyBuildingTypes[t] = false

This comment has been minimized.

Copy link
@lavalamp

lavalamp Jul 6, 2017

Member

Looking good--last question, should this be in a defer block? The return on line 496 skips it, for example.

This comment has been minimized.

Copy link
@nikhita

nikhita Jul 6, 2017

Author Member

Yes. Done.

Fix defaulter for nested recursive types
Rename

Address comments

Address further comments

Use defer

@nikhita nikhita force-pushed the nikhita:fix-recursive-types branch from 9a0932b to 96fe5d0 Jul 6, 2017

@lavalamp

This comment has been minimized.

Copy link
Member

commented Jul 6, 2017

LGTM, thanks for bearing with me :)

@lavalamp lavalamp merged commit d354881 into kubernetes:master Jul 6, 2017

2 checks passed

cla/linuxfoundation nikhita authorized
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@nikhita nikhita deleted the nikhita:fix-recursive-types branch Jul 6, 2017

@nikhita nikhita restored the nikhita:fix-recursive-types branch Jul 7, 2017

@nikhita nikhita referenced this pull request Jul 7, 2017
k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this pull request Aug 30, 2017
Kubernetes Submit Queue
Merge pull request #47263 from nikhita/crd-01-validation-types
Automatic merge from submit-queue

apiextensions: validation for customresources

- [x] Add types for validation of CustomResources
- [x] Fix conversion-gen: #49747
- [x] Fix defaulter-gen: kubernetes/gengo#61
- [x] Convert to OpenAPI types
- [x] Validate CR using go-openapi
- [x] Validate CRD Schema
- [x] Add integration tests
- [x] Fix round trip tests: #51204 
- [x] Add custom fuzzer functions
- [x] Add custom conversion functions
- [x] Fix data race while updating CRD: #50098 
- [x] Add feature gate for CustomResourceValidation
- [x] Fix protobuf generation

Proposal: kubernetes/community#708
Additional discussion: #49879, #50625

**Release note**:

```release-note
Add validation for CustomResources via JSON Schema.
```

/cc @sttts @deads2k
sttts pushed a commit to sttts/apiextensions-apiserver that referenced this pull request Sep 22, 2017
Merge pull request #47263 from nikhita/crd-01-validation-types
Automatic merge from submit-queue

apiextensions: validation for customresources

- [x] Add types for validation of CustomResources
- [x] Fix conversion-gen: #49747
- [x] Fix defaulter-gen: kubernetes/gengo#61
- [x] Convert to OpenAPI types
- [x] Validate CR using go-openapi
- [x] Validate CRD Schema
- [x] Add integration tests
- [x] Fix round trip tests: #51204
- [x] Add custom fuzzer functions
- [x] Add custom conversion functions
- [x] Fix data race while updating CRD: #50098
- [x] Add feature gate for CustomResourceValidation
- [x] Fix protobuf generation

Proposal: kubernetes/community#708
Additional discussion: kubernetes/kubernetes#49879, kubernetes/kubernetes#50625

**Release note**:

```release-note
Add validation for CustomResources via JSON Schema.
```

/cc @sttts @deads2k

Kubernetes-commit: 4457e43e7b789586096bfb564330295cf0438e70
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.