Code to automatically generate conversion methods #7107

wojtek-t · 2015-04-21T13:59:31Z

Part of #6800

There are two commits in this PR:

the code that I've written
the code that was generated - to show what code is generated.

This code is in a very early stage - the goal is to get the initial feedback, so any comments [including - this approach is bad, we should use a different one :)] are welcome.

The initial benchmarks show that conversion (without parsing jsons) is ~2x faster.

cc @lavalamp @smarterclayton @fgrzadkowski @davidopp

smarterclayton · 2015-04-21T14:50:50Z

What about memory use

On Apr 21, 2015, at 9:59 AM, Wojciech Tyczynski notifications@github.com wrote:

Part of #6800

There are two commits in this PR:

the code that I've written

the code that was generated (with some small manual fixes to namespaces) - to show what code is generated.

This code is in a very early stage - the goal is to get the initial feedback, so any comments [including - this approach is bad, we should use a different one :)] are welcome.

The initial benchmarks show that conversion (without parsing jsons) is ~2x faster.

cc @lavalamp @smarterclayton @fgrzadkowski @davidopp

You can view, comment on, or merge this pull request online at:

#7107

Commit Summary

Generating conversion methods.
Generated code
File Changes

A cmd/kube-conversion/conversion.go (51)
M pkg/api/register.go (4)
M pkg/api/serialization_test.go (12)
M pkg/api/v1beta3/conversion.go (431)
A pkg/conversion/generator.go (377)
M pkg/runtime/scheme.go (4)
Patch Links:

https://github.com/GoogleCloudPlatform/kubernetes/pull/7107.patch
https://github.com/GoogleCloudPlatform/kubernetes/pull/7107.diff
—
Reply to this email directly or view it on GitHub.

wojtek-t · 2015-04-21T14:55:28Z

@smarterclayton I didn't look at the memory usage yet. Do you have any specific tests on you mind? How would you like me to measure it?

smarterclayton · 2015-04-21T15:59:13Z

If you just set up a benchmark you can add --memprofile mem.out to your bench test run and then run go tool pprof ./testbinary em.out --alloc_space

----- Original Message -----

@smarterclayton I didn't look at the memory usage yet. Do you have any
specific tests on you mind? How would you like me to measure it?

Reply to this email directly or view it on GitHub:
#7107 (comment)

smarterclayton · 2015-04-21T16:02:07Z

I suspect we're still getting hit by the cost to generate the conversion stack and manipulate it, as well as some reflects. Can you add a few benchmark tests to this so I can look at some profiles and we can have a standard metric?

smarterclayton · 2015-04-21T16:03:04Z

Example of benchmark: https://github.com/GoogleCloudPlatform/kubernetes/blob/98e61e252d1087648321a59f515ce0ff95f57f84/pkg/conversion/deep_copy_test.go#L113

wojtek-t · 2015-04-21T16:42:52Z

If you just set up a benchmark you can add --memprofile mem.out to your bench test run and then run go tool pprof ./testbinary em.out --alloc_space

Thanks - will run it tomorrow.

I suspect we're still getting hit by the cost to generate the conversion stack and manipulate it, as well as some reflects.

I'm pretty sure that's true. But I think that we should start with something (the changes I've made are already giving huge improvement) and we can continue improvements after. I would like to make one step at a time.

Can you add a few benchmark tests to this so I can look at some profiles and we can have a standard metric?

So far I was using pkg/api/serialization_test.go. But I can add more.

smarterclayton · 2015-04-21T17:05:00Z

Having a few specific ones is good because we can specifically profile those and get apples-to-apples comparisons.

----- Original Message -----

If you just set up a benchmark you can add --memprofile mem.out to your bench
test run and then run go tool pprof ./testbinary em.out --alloc_space

Thanks - will run it tomorrow.

I suspect we're still getting hit by the cost to generate the conversion
stack and manipulate it, as well as some reflects.

I'm pretty sure that's true. But I think that we should start with something
(the changes I've made are already giving huge improvement) and we can
continue improvements after. I would like to make one step at a time.

Can you add a few benchmark tests to this so I can look at some profiles and
we can have a standard metric?

So far I was using pkg/api/serialization_test.go. But I can add more.

Reply to this email directly or view it on GitHub:
#7107 (comment)

wojtek-t · 2015-04-21T17:21:15Z

Sure - that's why I was so far using always pkg/api/serialization_test.go. But I agree that having more than one is good.

smarterclayton · 2015-04-21T17:37:21Z

That one will give different results every time (subtly) because, while the rand has a consistent seed, the iteration order of the runtime.Scheme map is different. So each run is going to be generating different fuzz for different objects.

----- Original Message -----

Sure - that's why I was so far using always pkg/api/serialization_test.go.
But I agree that having more than one is good.

Reply to this email directly or view it on GitHub:
#7107 (comment)

lavalamp · 2015-04-21T20:38:20Z

pkg/conversion/generator.go

+	return nil
+}
+
+func (g *generator) writeConversionForSlice(f *os.File, inField, outField reflect.StructField, indent int) error {


golang comment: use io.Reader/io.Writer, not os.File as parameters-- helps with testing among other things.

lavalamp · 2015-04-21T23:30:12Z

Hm, I was kinda thinking more along the lines of something that totally replaced the conversion functions with one big conversion function, like maybe it can take the existing conversion functions as input and glue them all together into one big function... it's interesting that this already gets a 2x speed improvement. We really want to track allocations. I think we can get this to more like 10x improvement if we cut out even more of the conversion package.

smarterclayton · 2015-04-21T23:50:40Z

Yeah, good first step though. The nice thing is you can potentially chain these and turn them into static functions with names, then have them cal each other (if necessary).

On Apr 21, 2015, at 7:30 PM, Daniel Smith notifications@github.com wrote:

Hm, I was kinda thinking more along the lines of something that totally replaced the conversion functions with one big conversion function, like maybe it can take the existing conversion functions as input and glue them all together into one big function... it's interesting that this already gets a 2x speed improvement. We really want to track allocations. I think we can get this to more like 10x improvement if we cut out even more of the conversion package.

—
Reply to this email directly or view it on GitHub.

wojtek-t · 2015-04-22T06:24:26Z

Hm, I was kinda thinking more along the lines of something that totally replaced the conversion functions with one big conversion function, like maybe it can take the existing conversion functions as input and glue them all together into one big function... it's interesting that this already gets a 2x speed improvement. We really want to track allocations. I think we can get this to more like 10x improvement if we cut out even more of the conversion package.

Absolutely - see my TODO:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7107/files#diff-e30c44aa3c643af118dbe436900e56ccR357
I just didn't want to do everything in one step.
But I completely agree that this is not the end state.

wojtek-t · 2015-04-22T06:27:10Z

Yeah, good first step though. The nice thing is you can potentially chain these and turn them into static functions with names, then have them cal each other (if necessary).

Yes I completely agree - this was also my plan once this is done.

wojtek-t · 2015-04-22T14:34:06Z

In #7163 I've created some dedicated benchmarks.
[result before the change vs after the change]
Pod: 384897 vs 146677
Node: 196462 vs 59733
RC: 400900 vs 133508
So the gain is more than 2.5x (on average).

wojtek-t · 2015-04-22T14:36:36Z

@smarterclayton
I've tried to profile the memory for the benchmarks mentioned above, but either I was doing something wrong or they were completely useless [there were two boxes on the profile]. Can you please write what exactly I should do?

smarterclayton · 2015-04-22T14:41:00Z

GOPATH=$(pwd)/Godeps/_workspace:$GOPATH go test ./pkg/conversion -test.run=XXXXX -bench=BenchmarkDeepCopy -benchmem -memprofile mem.out -memprofilerate=1
go tool pprof --alloc_space ./conversion.test  mem.out

is what I use.

smarterclayton · 2015-04-22T14:44:44Z

Also, use https://github.com/golang/tools/tree/master/cmd/benchcmp to generate a diff of your benchmark results.

smarterclayton · 2015-04-22T14:45:53Z

Having incremental steps is good - I'd rather have a couple of related optimization passes than one giant step. That potentially allows us to test independently as well as isolate bugs.

----- Original Message -----

Hm, I was kinda thinking more along the lines of something that totally
replaced the conversion functions with one big conversion function, like
maybe it can take the existing conversion functions as input and glue them
all together into one big function... it's interesting that this already
gets a 2x speed improvement. We really want to track allocations. I think we
can get this to more like 10x improvement if we cut out even more of the
conversion package.

Absolutely - see my TODO:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7107/files#diff-e30c44aa3c643af118dbe436900e56ccR357
I just didn't want to do everything in one step.
But I completely agree that this is not the end state.

Reply to this email directly or view it on GitHub:
#7107 (comment)

wojtek-t · 2015-04-22T14:47:53Z

Having incremental steps is good - I'd rather have a couple of related optimization passes than one giant step. That potentially allows us to test independently as well as isolate bugs.

Thanks - I completely agree that's why I decided to send it out and not wait e.g. another 2 weeks with more advanced changes.

wojtek-t · 2015-04-22T15:05:26Z

Regarding memory - the experiments are very promising too (there are ~3x less memory allocations)
I'm attaching two graphs (first before the optimizations, the second after).

smarterclayton · 2015-04-22T15:07:07Z

That's very good that reflection is still our biggest cost. That means every reflect we remove is dramatically increasing performance.

wojtek-t · 2015-04-22T15:08:41Z

I agree - there is still a lot of place for improvements here.

wojtek-t · 2015-04-22T15:11:47Z

BTW - I don't understand the shippable failure - from the shippable page it seems that all tests passed :)

wojtek-t · 2015-04-23T07:56:14Z

Benchmark comparison

benchmark                                    old ns/op     new ns/op     delta
BenchmarkPodConversion                       388502        149268        -61.58%
BenchmarkNodeConversion                      192973        60733         -68.53%
BenchmarkReplicationControllerConversion     385409        135528        -64.84%

The memory allocation graphs above also show ~66% improvement.

The next step I'm going to work on is to change those conversion functions to be named and call each other without going through reflections wherever it's possible.

@smarterclayton @lavalamp
This PR is ready for merging (although it's just the first step of improvement - more will come) [tests are passing]. Can you please merge it or do you have any additional comments now?

wojtek-t · 2015-04-23T08:35:45Z

e2e test are also passing:
Ran 35 of 40 Specs in 1386.554 seconds
SUCCESS! -- 35 Passed | 0 Failed | 1 Pending | 4 Skipped I0423 10:35:22.431741 17688 driver.go:96] All tests pass

smarterclayton · 2015-04-23T14:01:17Z

Can you add something to the upgrading the API doc so that people know that when they add things to conversions that the code is generated? We need to have a process as well for when someone manually adds a conversion working around a bug in the generator, how does the next person know to go fix the generator? Do we force people to use the generator?

wojtek-t · 2015-04-23T14:09:33Z

Currently the generator is still in kind of "experimental" mode so I wouldn't expect people to use it. I'm going to improve it (e.g. named conversion functions calling each other, some separation between user-provided methods and those automatically generated, etc.) Also I'm going to add a test checking whether methods in conversion.go is what we would generate with the generator.
Once all of these is done, I would like people to use it, but we're not at this point yet.
Does it sound reasonable?

Where exactly do you want me to put it?

smarterclayton · 2015-04-23T14:12:25Z

----- Original Message -----

Currently the generator is still in kind of "experimental" mode so I wouldn't
expect people to use it. I'm going to improve it (e.g. named conversion
functions calling each other, some separation between user-provided methods
and those automatically generated, etc.) Also I'm going to add a test
checking whether methods in conversion.go is what we would generate with the
generator.
Once all of these is done, I would like people to use it, but we're not at
this point yet.
Does it sound reasonable?

More just concerned that someone won't know. We can punt on it until you've advanced further, that's fine with me.

Where exactly do you want me to put it?

Reply to this email directly or view it on GitHub:
#7107 (comment)

wojtek-t · 2015-04-23T14:16:33Z

So if it's ok for you, I would wait with adding any documentation etc. until it's in more advanced stage. Once we have tests (consistency between what generator is generating and what already exist) etc. I will add necessary documentation to.

Do you have any other comments blocking the merge?

smarterclayton · 2015-04-23T15:59:25Z

Some code style fixes, no issues from me on structure

wojtek-t · 2015-04-23T16:19:20Z

Comments applied - PTAL

smarterclayton · 2015-04-23T16:37:30Z

LGTM, will merge on green.

lavalamp · 2015-04-23T17:43:10Z

Hold on this please, I would like to look at it first.

lavalamp · 2015-04-23T17:50:57Z

pkg/api/v1beta3/conversion.go

 )

 func init() {
+	err := newer.Scheme.AddConversionFuncs(


It is correct that these are auto-generated, yes? In that case I do NOT think we should commit in this way. The reason is that the other conversion functions are manually written, and if we do this we'll have a mix of manually and automatically written functions.

So, my recommendation is that you separate this in some way. E.g., autogenerate a func AddGeneratedConversionFunctions(), and call that first in init (and then it can be overridden with any manually generate functions.

I think this is very important because eventually the flow should go the other way-- you should produce the auto-generated stuff by gluing together the manually written stuff. So it's important to make the difference between the auto-generated and manually written code very obvious.

Yes - those are autogenerated functions.
I agree that eventually this should be done how you described it (or similarly). But I don't think it's not a big problem now, because I'm focused now on v1beta3 where we don't have manually written conversion functions. But I'm going to work more on it next week.

I've added AddGeneratedConversionFunctions() method to scheme and I'm using it - I hope that would be enough for you for now.
BTW - I would like it to be merged soon (I will be working on it more right after this PR) to unblock other experiments that Filip is also doing in the performance area.

wojtek-t · 2015-04-24T07:18:42Z

PTAL

smarterclayton · 2015-04-24T12:39:10Z

Comments addressed, will expect author to handle more of dan's feedback later :)

Code to automatically generate conversion methods

wojtek-t · 2015-04-24T12:41:30Z

@smarterclayton Thanks! Sure - I will address any additional comments in subsequent PR.

lavalamp · 2015-04-25T00:07:37Z

pkg/api/v1beta3/conversion.go

 )

 func init() {
+	err := newer.Scheme.AddGeneratedConversionFuncs(


In a followup PR can you add a test that these haven't been manually edited? Thanks :)

googlebot added the cla: yes label Apr 21, 2015

lavalamp reviewed Apr 21, 2015
View reviewed changes

wojtek-t force-pushed the conversion_generator branch from f6a677c to d503049 Compare April 22, 2015 10:12

wojtek-t changed the title ~~[WIP][Do NOT merge] Code to automatically generate conversion methods~~ Code to automatically generate conversion methods Apr 22, 2015

wojtek-t mentioned this pull request Apr 22, 2015

Create benchmarks for conversions. #7163

Merged

fabioy assigned smarterclayton Apr 22, 2015

wojtek-t force-pushed the conversion_generator branch from d503049 to 43d78e7 Compare April 23, 2015 07:35

wojtek-t force-pushed the conversion_generator branch from 43d78e7 to 0b383a3 Compare April 23, 2015 16:16

Generating conversion methods.

c4a27f5

wojtek-t force-pushed the conversion_generator branch from 0b383a3 to 7ed9b28 Compare April 23, 2015 16:27

smarterclayton added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 23, 2015

lavalamp removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 23, 2015

lavalamp reviewed Apr 23, 2015
View reviewed changes

Generate conversions for v1beta3 package.

7f919a4

wojtek-t force-pushed the conversion_generator branch from 7ed9b28 to 7f919a4 Compare April 24, 2015 07:18

smarterclayton added a commit that referenced this pull request Apr 24, 2015

Merge pull request #7107 from wojtek-t/conversion_generator

ed00dae

Code to automatically generate conversion methods

smarterclayton merged commit ed00dae into kubernetes:master Apr 24, 2015

lavalamp reviewed Apr 25, 2015
View reviewed changes

fgrzadkowski mentioned this pull request Apr 27, 2015

Get pods for many pods is very heavy #6514

Closed

wojtek-t deleted the conversion_generator branch April 30, 2015 12:14

Code to automatically generate conversion methods #7107

Code to automatically generate conversion methods #7107

Conversation

wojtek-t commented Apr 21, 2015

smarterclayton commented Apr 21, 2015

wojtek-t commented Apr 21, 2015

smarterclayton commented Apr 21, 2015

smarterclayton commented Apr 21, 2015

smarterclayton commented Apr 21, 2015

wojtek-t commented Apr 21, 2015

smarterclayton commented Apr 21, 2015

wojtek-t commented Apr 21, 2015

smarterclayton commented Apr 21, 2015

lavalamp Apr 21, 2015

Choose a reason for hiding this comment

wojtek-t Apr 22, 2015

Choose a reason for hiding this comment

lavalamp commented Apr 21, 2015

smarterclayton commented Apr 21, 2015

wojtek-t commented Apr 22, 2015

wojtek-t commented Apr 22, 2015

wojtek-t commented Apr 22, 2015

wojtek-t commented Apr 22, 2015

smarterclayton commented Apr 22, 2015

smarterclayton commented Apr 22, 2015

smarterclayton commented Apr 22, 2015

wojtek-t commented Apr 22, 2015

wojtek-t commented Apr 22, 2015

smarterclayton commented Apr 22, 2015

wojtek-t commented Apr 22, 2015

wojtek-t commented Apr 22, 2015

wojtek-t commented Apr 23, 2015

wojtek-t commented Apr 23, 2015

smarterclayton commented Apr 23, 2015

wojtek-t commented Apr 23, 2015

smarterclayton commented Apr 23, 2015

wojtek-t commented Apr 23, 2015

smarterclayton commented Apr 23, 2015

wojtek-t commented Apr 23, 2015

smarterclayton commented Apr 23, 2015

lavalamp commented Apr 23, 2015

lavalamp Apr 23, 2015

Choose a reason for hiding this comment

wojtek-t Apr 24, 2015

Choose a reason for hiding this comment

wojtek-t commented Apr 24, 2015

smarterclayton commented Apr 24, 2015

wojtek-t commented Apr 24, 2015

lavalamp Apr 25, 2015

Choose a reason for hiding this comment