Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

openapi-aggregation: speed up merging from 1 sec to 50-100 ms #71223

Merged
merged 8 commits into from Feb 11, 2019

Conversation

@sttts
Copy link
Contributor

sttts commented Nov 19, 2018

This PR avoids swagger spec cloning by non-source-mutating merging. It uses a mutating walker that locally clones minimal substructures.

Speedup: >>10

Considerably reduced the CPU load in kube-apiserver while aggregating OpenAPI specifications from aggregated API servers.
@roycaihw

This comment has been minimized.

Copy link
Member

roycaihw commented Nov 19, 2018

Great! The idea looks good to me. Spec cloning makes up ~98% of the time when we do spec merging.

We need to unit test the cloning. I see kube-openapi has a decent amount of unit test covering spec merging. Maybe that's enough

@sttts sttts changed the title openapi-aggreation: speed merging from 1 sec to 50-100 ms openapi-aggreation: speed up merging from 1 sec to 50-100 ms Nov 22, 2018

@sttts sttts changed the title openapi-aggreation: speed up merging from 1 sec to 50-100 ms openapi-aggregation: speed up merging from 1 sec to 50-100 ms Dec 9, 2018

@sttts sttts force-pushed the sttts:sttts-openapi-aggreation-without-clone branch 2 times, most recently from 3a0132a to 9c7b6cb Dec 11, 2018

@mbohlool

This comment has been minimized.

Copy link
Member

mbohlool commented Dec 20, 2018

This is a change in vendor folder, please close this in favor of the other PR for less confusion. I was looking for my comments on the other PR here :)

@sttts sttts force-pushed the sttts:sttts-openapi-aggreation-without-clone branch 2 times, most recently from c66f1c8 to 079c1b0 Feb 1, 2019

@k8s-ci-robot k8s-ci-robot removed the needs-kind label Feb 1, 2019

@sttts sttts force-pushed the sttts:sttts-openapi-aggreation-without-clone branch 2 times, most recently from 7618d5c to 7f9bb8e Feb 8, 2019

@jennybuckley jennybuckley referenced this pull request Feb 8, 2019

Open

Umbrella Issue for Server Side Apply #73723

22 of 45 tasks complete

@sttts sttts force-pushed the sttts:sttts-openapi-aggreation-without-clone branch from 7f9bb8e to 19506e4 Feb 8, 2019

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Feb 8, 2019

something is up with the godeps changes:

Your Godeps.json is different:
--- Godeps/Godeps.json	2019-02-08 20:33:07.190360977 +0000
+++ /tmp/gopath.tGCkZA/src/k8s.io/kubernetes/Godeps/Godeps.json	2019-02-08 20:47:27.121379369 +0000
@@ -1679,7 +1679,7 @@
 		},
 		{
 			"ImportPath": "github.com/go-sql-driver/mysql",
-			"Comment": "v1.3",
+			"Comment": "v1.3.0",
 			"Rev": "a0583e0143b1624142adab07e0e97fe106d99561"
 		},
 		{
@@ -2348,6 +2348,7 @@
 		},
 		{
 			"ImportPath": "github.com/inconshreveable/mousetrap",
+			"Comment": "v1.0",
 			"Rev": "76626ae9c91c4f2a10f34cad8ce83ea42c93bb75"
 		},
 		{
@@ -2429,12 +2430,10 @@
 		},
 		{
 			"ImportPath": "github.com/lib/pq",
-			"Comment": "go1.0-cutoff-203-g88edab0803230a",
 			"Rev": "88edab0803230a3898347e77b474f8c1820a1f20"
 		},
 		{
 			"ImportPath": "github.com/lib/pq/oid",
-			"Comment": "go1.0-cutoff-203-g88edab0803230a",
 			"Rev": "88edab0803230a3898347e77b474f8c1820a1f20"
 		},
 		{
@@ -2508,7 +2507,7 @@
 		},
 		{
 			"ImportPath": "github.com/matttproud/golang_protobuf_extensions/pbutil",
-			"Comment": "v1.0.0-2-gc12348ce28de40",
+			"Comment": "v1.0.1",
 			"Rev": "c12348ce28de40eed0136aa2b644d0ee0650e56c"
 		},

@sttts sttts force-pushed the sttts:sttts-openapi-aggreation-without-clone branch from 19506e4 to ff2547c Feb 8, 2019

Show resolved Hide resolved Godeps/Godeps.json Outdated

@sttts sttts force-pushed the sttts:sttts-openapi-aggreation-without-clone branch 3 times, most recently from c702b4d to 608ca9c Feb 11, 2019

@sttts sttts force-pushed the sttts:sttts-openapi-aggreation-without-clone branch from 608ca9c to 233178e Feb 11, 2019

@sttts

This comment has been minimized.

Copy link
Contributor Author

sttts commented Feb 11, 2019

/retest

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Feb 11, 2019

/retest

verified the only differences in the swagger.json file are object key order and json string escaping (no structural or value changes that would affect anything consuming the published schemas)

diff \
  <(curl -s https://raw.githubusercontent.com/sttts/kubernetes/sttts-openapi-aggreation-without-clone/api/openapi-spec/swagger.json | jq -S .) \
  <(curl -s https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json | jq -S .)

<no diff>

/lgtm
/approve
/retest

@k8s-ci-robot k8s-ci-robot added the lgtm label Feb 11, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Feb 11, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 6912bbb into kubernetes:master Feb 11, 2019

17 checks passed

cla/linuxfoundation sttts authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-godeps Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped
tide In merge pool.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.