Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug 2044622: staticpod pruner: check if the cert directory exists to avoid panic #1297

Conversation

ingvagabund
Copy link
Member

@ingvagabund ingvagabund commented Jan 25, 2022

@openshift-ci openshift-ci bot added bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Jan 25, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 25, 2022

@ingvagabund: This pull request references Bugzilla bug 2044622, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.z) matches configured target release for branch (4.9.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 2026352 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
  • dependent Bugzilla bug 2026352 targets the "4.10.0" release, which is one of the valid target releases: 4.10.0
  • bug has dependents

Requesting review from QA contact:
/cc @kasturinarra

In response to this:

bug 2044622: Pruner check if dir exists 4.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@@ -305,6 +305,7 @@ func RunOperator(ctx context.Context, controllerContext *controllercmd.Controlle
controllerContext.EventRecorder.WithComponentSuffix("invalid-webhook-certs"),
"/var/run/configmaps/service-ca-bundle/service-ca.crt",
metricscontroller.NewLegacyCNCertsMetricsSyncFunc(
"",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@s-urbaniak you didn't leave instructions on how to update after the changes introduced in openshift/library-go#1289 :). Is it okay to pass an empty prefix?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@p0lyn0mial my bad, i was expecting to do this bump myself :-)
Two comments:

  1. Yes, it can be empty string. In this case, conditions will not be prefixed for status reporting. This is only important if you have multiple metrics controllers in the same operator or you want to give more context. here, empty string is fine.
  2. we're a bit smarter now with the detection what certificates are faulty. For this the query can return labels to differentiate. I gave a suggestion below.

@@ -14,7 +14,7 @@ require (
github.com/openshift/api v0.0.0-20210831091943-07e756545ac1
github.com/openshift/build-machinery-go v0.0.0-20211221164819-4024613928f1
github.com/openshift/client-go v0.0.0-20210831095141-e19a065e79f7
github.com/openshift/library-go v0.0.0-20220113084649-1a3940c8c9da
github.com/openshift/library-go v0.0.0-20220125122342-ff51c8a74c7b
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, this points to the 4.9 branch

@ingvagabund
Copy link
Member Author

/retest-required

@@ -305,6 +305,7 @@ func RunOperator(ctx context.Context, controllerContext *controllercmd.Controlle
controllerContext.EventRecorder.WithComponentSuffix("invalid-webhook-certs"),
"/var/run/configmaps/service-ca-bundle/service-ca.crt",
metricscontroller.NewLegacyCNCertsMetricsSyncFunc(
"",
`sum(apiserver_kube_aggregator_x509_missing_san_total{job="apiserver",namespace="default"}) + sum(apiserver_webhooks_x509_missing_san_total{job="apiserver",namespace="default"})`,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To give the controller clues about what certificates are faulty let's modify the query to be:

sum by (type) (
  label_replace(apiserver_webhooks_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "webhooks", "", "")
  or
  label_replace(apiserver_kube_aggregator_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "aggregation", "", ""))

This will yield results like:

{type="webhooks"}: 123
{type="aggregation"}: 456

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as discussed OOB let's include a promtool unit test file for this query as well:

# This file declares a unit test for the cert detection query. Execute with:
# $ /bin/prometheus-2.x.y/promtool test rules missing_san_total.yaml
# Unit Testing:  missing_san_total.yaml
#  SUCCESS
#
# TODO(sur): add support for promtool in api-build-machinery.

evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      #
      # api-server #1 metrics
      #
      - # apiserver #1 started observing invalid SANs at minute 4
        series: 'apiserver_kube_aggregator_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.1:6443", job="apiserver", namespace="default", service="kubernetes"}'
        values: '0 0 0 0 1'
      - series: 'apiserver_webhooks_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.1:6443", job="apiserver", namespace="default", service="kubernetes"}'
        values: '0 0 0 0 0'

      #
      # api-server #2 metrics
      #
      - series: 'apiserver_kube_aggregator_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.2:6443", job="apiserver", namespace="default", service="kubernetes"}'
        values: '0 0 0 0 0'
      - # apiserver #2 started observing invalid SANs at minute 3
        series: 'apiserver_webhooks_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.2:6443", job="apiserver", namespace="default", service="kubernetes"}'
        values: '0 0 0 1 1'

      #
      # api-server #3 metrics
      #
      - series: 'apiserver_kube_aggregator_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.3:6443", job="apiserver", namespace="default", service="kubernetes"}'
        values: '0 0 0 0 0'
      - # apiserver #3 started observing invalid SANs at minute 4
        series: 'apiserver_webhooks_x509_missing_san_total{apiserver="kube-apiserver", endpoint="https", instance="10.0.0.3:6443", job="apiserver", namespace="default", service="kubernetes"}'
        values: '0 0 0 0 2'

      #
      # rogue job emitting this metrics, it must be ignored
      #
      - series: 'apiserver_webhooks_x509_missing_san_total{endpoint="https-metrics", instance="10.0.128.4:10250", job="unknonw", metrics_path="/metrics", namespace="foobar", node="ci-op-5fhw80bs-465b4-tfs28-worker-b-db2vd", service="foo"}'
        values: '0 0 0 1 1'

    promql_expr_test:
      - eval_time: '0m'
        expr: 'sum by (type) (
  label_replace(apiserver_webhooks_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "webhooks", "", "")
  or
  label_replace(apiserver_kube_aggregator_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "aggregation", "", ""))'
        exp_samples:
          - labels: '{type="webhooks"}'
            value: 0
          - labels: '{type="aggregation"}'
            value: 0

      - eval_time: '3m'
        expr: 'sum by (type) (
  label_replace(apiserver_webhooks_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "webhooks", "", "")
  or
  label_replace(apiserver_kube_aggregator_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "aggregation", "", ""))'
        exp_samples:
          - labels: '{type="webhooks"}'
            value: 1
          - labels: '{type="aggregation"}'
            value: 0

      - eval_time: '4m'
        expr: 'sum by (type) (
  label_replace(apiserver_webhooks_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "webhooks", "", "")
  or
  label_replace(apiserver_kube_aggregator_x509_missing_san_total{job="apiserver", namespace="default"}, "type", "aggregation", "", ""))'
        exp_samples:
          - labels: '{type="webhooks"}'
            value: 3
          - labels: '{type="aggregation"}'
            value: 1

Modify the query to give the controller clues about what certificates are faulty
@p0lyn0mial
Copy link
Contributor

/lgtm
/approve

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 1, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 1, 2022

@ingvagabund: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-operator-single-node dd71f12 link false /test e2e-gcp-operator-single-node

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ingvagabund ingvagabund changed the title bug 2044622: Pruner check if dir exists 4.9 bug 2044622: Pruner check if dir exists to avoid panic Feb 2, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 2, 2022

@ingvagabund: This pull request references Bugzilla bug 2044622, which is valid.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.z) matches configured target release for branch (4.9.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 2026352 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
  • dependent Bugzilla bug 2026352 targets the "4.10.0" release, which is one of the valid target releases: 4.10.0
  • bug has dependents

Requesting review from QA contact:
/cc @kasturinarra

In response to this:

bug 2044622: Pruner check if dir exists 4.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 2, 2022

@ingvagabund: This pull request references Bugzilla bug 2044622, which is valid.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.z) matches configured target release for branch (4.9.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 2026352 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
  • dependent Bugzilla bug 2026352 targets the "4.10.0" release, which is one of the valid target releases: 4.10.0
  • bug has dependents

Requesting review from QA contact:
/cc @kasturinarra

In response to this:

bug 2044622: Pruner check if dir exists to avoid panic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ingvagabund ingvagabund changed the title bug 2044622: Pruner check if dir exists to avoid panic bug 2044622: staticpod pruner: check if the cert directory exists to avoid panic Feb 2, 2022
@ingvagabund
Copy link
Member Author

  1. If the Bugzilla associated with the PR has the "FastFix" keyword, the subjective assessment on the issue has already been done and a customer is impacted. These PRs should be prioritized for merge.
    • verified
    • does not apply
  2. The bug has significant impact either through severity, reduction in supportability, or number of users affected.
    • verified
    • does not apply
  3. For branches that are in the Maintenance lifecycle phase:
    • The bug is a critical fix, no reasonable workaround exists, and a recommendation for upgrade has been ruled out, or
    • The bug is a security related bug
    • Branch not in maintenance mode yet (current release + previous release for 90 days after current GA; everything older is in maintenance)
  4. The severity field of the bug must be set to accurately reflect criticality.
    • verified
  5. The PR was created with the cherry-pick bot OR the PR's description is well formed with user-focused release notes that state the bug number, impact, cause, and resolution. Where appropriate, it should also contain information about how a user can identify whether a particular cluster is affected.
    • verified

@sttts
Copy link
Contributor

sttts commented Feb 2, 2022

/approve

@sttts sttts added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Feb 2, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 2, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ingvagabund, p0lyn0mial, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 2, 2022
@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@gangwgr
Copy link

gangwgr commented Feb 2, 2022

/label cherry-pick-approved

@openshift-ci openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Feb 2, 2022
@openshift-merge-robot openshift-merge-robot merged commit a895470 into openshift:release-4.9 Feb 2, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 2, 2022

@ingvagabund: All pull requests linked via external trackers have merged:

Bugzilla bug 2044622 has been moved to the MODIFIED state.

In response to this:

bug 2044622: staticpod pruner: check if the cert directory exists to avoid panic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ingvagabund ingvagabund deleted the pruner-check-if-dir-exists-4.9 branch February 2, 2022 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants