Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aggregator/apiextensions: logs & metrics why OpenAPI spec is regenerated #81786

Merged
merged 3 commits into from Aug 27, 2019

Conversation

@sttts
Copy link
Contributor

commented Aug 22, 2019

Every OpenAPI spec regeneration is heavy on the CPU, a multi-second one-core operation. In a normal cluster this should happen not so often. If it does, the admin has to know. This PR will add the log output to make the situation actionable.

/kind feature

Log when kube-apiserver regenerates the OpenAPI spec and why. OpenAPI spec generation is a very CPU-heavy process that is sensitive to continuous updates of CRDs and APIServices.

Added metrics aggregator_openapi_v2_regeneration_count, aggregator_openapi_v2_regeneration_gauge and apiextension_openapi_v2_regeneration_count metrics counting the triggering APIService and CRDs and the reason (add, update, delete).
@deads2k

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2019

Do we have a metric gathering how often we do this, maybe sliced by why?

@deads2k

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2019

I'm fine with it in concept, but I'd like a metric attached.

@fedebongio

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2019

/assign @roycaihw

@roycaihw

This comment has been minimized.

Copy link
Member

commented Aug 22, 2019

agree having metricses for aggregator and apiextensions is better

we have workqueue metricses for crd_openapi_controller and APIServiceOpenAPIAggregationConttollerQueue1, but those aren't exact the same as how many times spec regeneration happened

@sttts

This comment has been minimized.

Copy link
Contributor Author

commented Aug 24, 2019

@roycaihw this is less about how often, but more about why.

@xmudrii

This comment has been minimized.

Copy link
Member

commented Aug 25, 2019

@sttts @deads2k @roycaihw Hello! I'm the bug triage lead for the 1.16 release cycle and I'd just like to remind that the code freeze is starting on August 29th EOD PST (about 5 days from now). Once the code freeze starts, only release-blocking PRs will be considered for merging.

@sttts sttts force-pushed the sttts:sttts-openapi-log-why branch from 37b66b7 to bd9b4f5 Aug 27, 2019
@k8s-ci-robot k8s-ci-robot added size/L and removed size/S labels Aug 27, 2019
@sttts sttts changed the title aggregator/apiextensions: v(2) log why OpenAPI spec is regenerated aggregator/apiextensions: logs & metrics why OpenAPI spec is regenerated Aug 27, 2019
@sttts sttts force-pushed the sttts:sttts-openapi-log-why branch from bd9b4f5 to 8b9e3fa Aug 27, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@deads2k

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2019

thanks
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Aug 27, 2019
@sttts sttts force-pushed the sttts:sttts-openapi-log-why branch from 8b9e3fa to 7fc61eb Aug 27, 2019
@k8s-ci-robot k8s-ci-robot removed the lgtm label Aug 27, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2019

New changes are detected. LGTM label has been removed.

@sttts

This comment has been minimized.

Copy link
Contributor Author

commented Aug 27, 2019

Rebased.

@sttts sttts added the lgtm label Aug 27, 2019
@sttts sttts force-pushed the sttts:sttts-openapi-log-why branch from 7fc61eb to beee72e Aug 27, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Aug 27, 2019

New changes are detected. LGTM label has been removed.

@k8s-ci-robot k8s-ci-robot removed the lgtm label Aug 27, 2019
@sttts sttts added the lgtm label Aug 27, 2019
@sttts

This comment has been minimized.

Copy link
Contributor Author

commented Aug 27, 2019

/retest

@k8s-ci-robot k8s-ci-robot merged commit bfd8610 into kubernetes:master Aug 27, 2019
24 checks passed
24 checks passed
cla/linuxfoundation sttts authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-conformance-kind-ipv6 Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-csi-serial Skipped.
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-iscsi Skipped.
pull-kubernetes-e2e-gce-iscsi-serial Skipped.
pull-kubernetes-e2e-gce-storage-slow Skipped.
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details
@roycaihw roycaihw referenced this pull request Sep 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.