Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1853253: remove expired TLS secret for Thanos Ruler #878

Merged
merged 2 commits into from Jul 28, 2020
Merged

Bug 1853253: remove expired TLS secret for Thanos Ruler #878

merged 2 commits into from Jul 28, 2020

Conversation

simonpasquier
Copy link
Contributor

  • I added CHANGELOG entry for this change.
  • No user facing changes, so no entry in CHANGELOG was needed.

@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 22, 2020
@openshift-ci-robot
Copy link
Contributor

@simonpasquier: This pull request references Bugzilla bug 1853253, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.6.0) matches configured target release for branch (4.6.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

[WIP] Bug 1853253: remove expired TLS secret for Thanos Ruler

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Jul 22, 2020
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 22, 2020
@simonpasquier
Copy link
Contributor Author

The first version I've pushed was without the fix to verify that the test was failing and it did.

     TestUserWorkloadMonitoring/assert_grpc_tls_rotation: user_workload_monitoring_test.go:850: expecting 4 gRPC secrets, got 5
    TestUserWorkloadMonitoring: user_workload_monitoring_test.go:77: scenario "assert grpc tls rotation" failed
--- FAIL: TestUserWorkloadMonitoring (244.97s)

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-monitoring-operator/878/pull-ci-openshift-cluster-monitoring-operator-master-e2e-aws-operator/1285970982116265984

@simonpasquier
Copy link
Contributor Author

/retest

1 similar comment
@simonpasquier
Copy link
Contributor Author

/retest

@simonpasquier
Copy link
Contributor Author

/hold cancel

@simonpasquier simonpasquier changed the title [WIP] Bug 1853253: remove expired TLS secret for Thanos Ruler Bug 1853253: remove expired TLS secret for Thanos Ruler Jul 23, 2020
@openshift-ci-robot openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 23, 2020
@simonpasquier
Copy link
Contributor Author

cc @openshift/openshift-team-monitoring

paulfantom
paulfantom approved these changes Jul 24, 2020
View reviewed changes
Copy link
Contributor

@paulfantom paulfantom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 24, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

1 similar comment
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@simonpasquier
Copy link
Contributor Author

/hold

Looking at https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-monitoring-operator/878/pull-ci-openshift-cluster-monitoring-operator-master-e2e-aws-operator/1285970982116265984, the test seems flaky

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 24, 2020
simonpasquier
simonpasquier commented Jul 24, 2020
View reviewed changes
test/e2e/user_workload_monitoring_test.go Outdated
@@ -819,6 +844,11 @@ func assertGRPCTLSRotation(t *testing.T) {
if err != nil {
t.Fatal(err)
}

got := countGRPCSecrets(f.Ns) + countGRPCSecrets(f.UserWorkloadMonitoringNs)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs retries because the main grpc-tls secret might have been rotated but not yet propagated to the other secrets.

@simonpasquier
Bug 1853253: remove expired TLS secret for Thanos Ruler
46a5e1b 
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Jul 24, 2020
@simonpasquier
Copy link
Contributor Author

/hold cancel

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 24, 2020
s-urbaniak
s-urbaniak reviewed Jul 27, 2020
View reviewed changes
test/e2e/user_workload_monitoring_test.go Outdated
if err != nil {
t.Fatalf("error waiting for grpc-tls secret: %v", err)
}

expected := countGRPCSecrets(f.Ns) + countGRPCSecrets(f.UserWorkloadMonitoringNs)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this calculation of expected grpc secret count is racy. The test code here waits until the central grpc-tls secret is created. However at this point of time there is no guarantee that all derived hashed grpc secrets are also already created as their creation is orchestrated in separate task executions which happen concurrently.

Instead, I propose to change this code here to have a const number of five (5) expected secrets, because we know them in forehand, as they are:

  1. openshift-monitoring/grpc-tls
  2. openshift-monitoring/prometheus-k8s-grpc-tls-[hash]
  3. openshift-user-workload-monitoring/prometheus-user-workload-grpc-tls-[hash]
  4. openshift-monitoring/thanos-querier-grpc-tls-[hash]
  5. openshift-user-workload-monitoring/thanos-ruler-grpc-tls-[hash]

@openshift/openshift-team-monitoring wdyt?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardcoding value here seems fine to me, but I would expect an in-code comment about this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@paulfantom added comment in addition to the commit message, ptal.

@s-urbaniak
Copy link
Contributor

/hold

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 27, 2020
@s-urbaniak
Copy link
Contributor

/cc @openshift/openshift-team-monitoring ptal

@openshift-ci-robot openshift-ci-robot requested a review from a team July 27, 2020 12:11
@openshift-ci-robot
Copy link
Contributor

@s-urbaniak: GitHub didn't allow me to request PR reviews from the following users: ptal.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @openshift/openshift-team-monitoring ptal

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@s-urbaniak
Copy link
Contributor

/cc @openshift/openshift-team-monitoring

@s-urbaniak
Copy link
Contributor

/test e2e-aws

@s-urbaniak
Copy link
Contributor

e2e test failure seems to be unrelated to monitoring

@s-urbaniak
test/e2e/user_workload_monitoring: have a constant amount of expected…
a19ca84 
… secrets.

The calculation of expected grpc secret count is potentially racy.
The current test code waits until the central grpc-tls secret is created.
However at this point of time there is no guarantee
that all derived hashed grpc secrets are also already created
as their creation is orchestrated in separate task executions which happen concurrently.

Instead, this changes it to have a const number of five (5) expected secrets,
because we know them in forehand.
@paulfantom
Copy link
Contributor

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 27, 2020
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: paulfantom, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [paulfantom,simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@s-urbaniak
Copy link
Contributor

/test e2e-aws

@s-urbaniak
Copy link
Contributor

/hold cancel
/retest

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 28, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 23622d5 into openshift:master Jul 28, 2020
@openshift-ci-robot
Copy link
Contributor

@simonpasquier: All pull requests linked via external trackers have merged: openshift/cluster-monitoring-operator#878. Bugzilla bug 1853253 has been moved to the MODIFIED state.

In response to this:

Bug 1853253: remove expired TLS secret for Thanos Ruler

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@s-urbaniak s-urbaniak s-urbaniak left review comments

@paulfantom paulfantom paulfantom approved these changes

Assignees

@paulfantom paulfantom

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@simonpasquier @openshift-ci-robot @openshift-bot @s-urbaniak @paulfantom @openshift-merge-robot