openapi-aggregation: speed up merging from 1 sec to 50-100 ms

[sttts]

This PR avoids swagger spec cloning by non-source-mutating merging. It uses a mutating walker that locally clones minimal substructures.

Speedup: >>10

Considerably reduced the CPU load in kube-apiserver while aggregating OpenAPI specifications from aggregated API servers.

[roycaihw]

Great! The idea looks good to me. Spec cloning makes up ~98% of the time when we do spec merging.

We need to unit test the cloning. I see kube-openapi has a decent amount of unit test covering spec merging. Maybe that's enough

[mbohlool]

This is a change in vendor folder, please close this in favor of the other PR for less confusion. I was looking for my comments on the other PR here :)

[liggitt]

something is up with the godeps changes:

Your Godeps.json is different:
--- Godeps/Godeps.json	2019-02-08 20:33:07.190360977 +0000
+++ /tmp/gopath.tGCkZA/src/k8s.io/kubernetes/Godeps/Godeps.json	2019-02-08 20:47:27.121379369 +0000
@@ -1679,7 +1679,7 @@
 		},
 		{
 			"ImportPath": "github.com/go-sql-driver/mysql",
-			"Comment": "v1.3",
+			"Comment": "v1.3.0",
 			"Rev": "a0583e0143b1624142adab07e0e97fe106d99561"
 		},
 		{
@@ -2348,6 +2348,7 @@
 		},
 		{
 			"ImportPath": "github.com/inconshreveable/mousetrap",
+			"Comment": "v1.0",
 			"Rev": "76626ae9c91c4f2a10f34cad8ce83ea42c93bb75"
 		},
 		{
@@ -2429,12 +2430,10 @@
 		},
 		{
 			"ImportPath": "github.com/lib/pq",
-			"Comment": "go1.0-cutoff-203-g88edab0803230a",
 			"Rev": "88edab0803230a3898347e77b474f8c1820a1f20"
 		},
 		{
 			"ImportPath": "github.com/lib/pq/oid",
-			"Comment": "go1.0-cutoff-203-g88edab0803230a",
 			"Rev": "88edab0803230a3898347e77b474f8c1820a1f20"
 		},
 		{
@@ -2508,7 +2507,7 @@
 		},
 		{
 			"ImportPath": "github.com/matttproud/golang_protobuf_extensions/pbutil",
-			"Comment": "v1.0.0-2-gc12348ce28de40",
+			"Comment": "v1.0.1",
 			"Rev": "c12348ce28de40eed0136aa2b644d0ee0650e56c"
 		},

[sttts]

/retest

[liggitt]

/retest

verified the only differences in the swagger.json file are object key order and json string escaping (no structural or value changes that would affect anything consuming the published schemas)

diff \
  <(curl -s https://raw.githubusercontent.com/sttts/kubernetes/sttts-openapi-aggreation-without-clone/api/openapi-spec/swagger.json | jq -S .) \
  <(curl -s https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json | jq -S .)

<no diff>

/lgtm
/approve
/retest

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• Godeps/OWNERS [liggitt,sttts]
• api/OWNERS [liggitt]
• hack/OWNERS [liggitt,sttts]
• pkg/OWNERS [liggitt]
• staging/src/k8s.io/apiextensions-apiserver/Godeps/OWNERS [liggitt,sttts]
• staging/src/k8s.io/apiextensions-apiserver/pkg/apis/OWNERS [liggitt]
• staging/src/k8s.io/apimachinery/Godeps/OWNERS [liggitt,sttts]
• staging/src/k8s.io/apiserver/OWNERS [liggitt,sttts]
• staging/src/k8s.io/client-go/Godeps/OWNERS [liggitt,sttts]
• staging/src/k8s.io/cloud-provider/Godeps/OWNERS [liggitt,sttts]
• staging/src/k8s.io/csi-api/Godeps/OWNERS [liggitt,sttts]
• staging/src/k8s.io/kube-aggregator/OWNERS [liggitt]
• staging/src/k8s.io/metrics/Godeps/OWNERS [liggitt,sttts]
• staging/src/k8s.io/node-api/Godeps/OWNERS [liggitt,sttts]
• staging/src/k8s.io/sample-apiserver/Godeps/OWNERS [liggitt,sttts]
• staging/src/k8s.io/sample-controller/Godeps/OWNERS [liggitt,sttts]
• vendor/OWNERS [liggitt,sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment