Improve CRD openapi serving speed

[roycaihw]

CustomResourcePublishOpenAPI e2e tests are flaky because the time for a given CRD change to get reflected in the openapi spec exceeds the 30s timeout.

Here is a analysis for the worst case-- assuming three CRD changes were made at the same time (change A&B by tests running in parallel, and change C by the current test), it may take up to three 200OK downloads before the current e2e test sees change C in the openapi spec. The 30s timeout can be reached before the test starts the third 200OK download.

Things that we can improve:

1. openapi service update (for both kube-aggregator and CRD) is most expensive. Adding log shows both the first marshaling and ToProtoBinary can cost more that 5s for kube-aggregator. The time spent also varies a lot even though the spec size should be rather stable (due to CPU pressure?). More efficient ways to do the marshaling and generate proto binary would help.
2. although the CRD openapi spec is much smaller compared to kube-aggregator, openapi service update still costs 0.5~1.5s to apiextensions-apiserver for each CRD change. Since openapi service update requires lock, the time queues up when there are lots of CRD changes at the same time (e.g. three CRD changes in the image above). Solving the previous point also helps here
3. apiextensions-apiserver supports serving openapi spec in PB and gzip, but kube-aggregator downloads and deserializes the spec through JSON (which makes sense for aggregated apiservers). Potentially kube-aggregator can use PB when it's talking to apiextensions-apiserver.

cc @liggitt @sttts @wojtek-t

[roycaihw]

/sig api-machinery
/area custom-resources

[wojtek-t]

Adding log shows both the first marshaling and ToProtoBinary can cost more that 5s for kube-aggregator.

Re ToProtoBinary - did you split that into: NewDocument + jsonToYAML and proto.Marshal()?
I guess that the victim may be the first part, not necessary proto.Marshal...

[roycaihw]

Re ToProtoBinary - did you split that into: NewDocument + jsonToYAML and proto.Marshal()?

Not I didn't. I will add logging to break that down

[tedyu]

Looking in the kube-openapi repo, I only found one benchmark under pkg:

func BenchmarkMergeSpecsIgnorePathConflictsWithKubeSpec(b *testing.B) {
pkg//aggregator/aggregator_test.go

Wondering if more benchmark should be added.

[fedebongio]

/assign @roycaihw @sttts

[annajung]

Hello @roycaihw @sttts 👋 I'm one of the Bug Triage shadows for the 1.18 release. Since this issue is tagged for 1.18, I'd like to check its status. Is this issue intended for milestone 1.18?

Friendly reminder: Code freeze is scheduled for March 5th.

[annajung]

Hello @roycaihw @sttts 👋 Checking in again. Do you think this will make it in to 1.18 release?

[roycaihw]

@annajung I don't have the cycle to work on it this release. Let's move it to milestone 1.19. Thank you

[annajung]

/milestone v1.19

[bai]

👋 Hello from Bug Triage team! Wanted to follow up on this issue with a friendly reminder that the code freeze for 1.19 is starting June 25th (about 4 weeks from now).

As this issue is tagged for 1.19, is it still planned for this release?