check if ClusterMachineApproverDown UsingDeprecatedAPIExtensionsV1Beta1 alerts are not firing #24071

paulfantom · 2019-11-01T14:35:36Z

Follow-up to #24019.

/cc @lilic @simonpasquier @s-urbaniak

s-urbaniak · 2019-11-01T14:44:32Z

/lgtm

paulfantom · 2019-11-01T16:15:40Z

/retest

simonpasquier · 2019-11-04T08:22:04Z

test/extended/prometheus/prometheus.go

@@ -210,7 +210,7 @@ var _ = g.Describe("[Feature:Prometheus][Conformance] Prometheus", func() {
 			tests := map[string]bool{
 				// Checking Watchdog alert state is done in "should have a Watchdog alert in firing state".
 				// FIXME(paulfantom): ClusterMachineApproverDown and UsingDeprecatedAPIExtensionsV1Beta1 shouldn't be firing!


this line can be removed

lilic · 2019-11-04T09:24:57Z

/retest

paulfantom · 2019-11-04T12:13:01Z

@simonpasquier removed comment.

Putting this on hold as UsingDeprecatedAPIExtensionsV1Beta1 is firing constantly. BZ for this issue is created at https://bugzilla.redhat.com/show_bug.cgi?id=1768417

/hold

simonpasquier

/lgtm

tnozicka · 2019-11-06T14:34:17Z

#24089 merged
/retest

tnozicka · 2019-11-06T21:38:33Z

openshift/cluster-kube-apiserver-operator#639 merged
/retest

tnozicka · 2019-11-07T07:54:44Z

/hold cancel
(the 2 PRs above fixed the e2e here)

openshift-bot · 2019-11-07T09:36:08Z

/retest

Please review the full test history for this PR and help us cut down flakes.

tnozicka · 2019-11-07T13:09:07Z

gcp-upgrade flaked on gcp
/retest

tnozicka · 2019-11-07T13:21:07Z

/hold
there seems to be a flake depending on when the test is run

tnozicka · 2019-11-11T08:16:44Z

/test e2e-aws-serial
/test e2e-gcp

paulfantom · 2019-11-11T10:27:03Z

@tnozicka failure looks legitimate to me. Log message says:

ALERTS{alertname=\"UsingDeprecatedAPIExtensionsV1Beta1\", alertstate=\"firing\", client=\"kubectl/v0.0.0 (linux/amd64) kubernetes/$Format\", code=\"200\", component=\"apiserver\", contentType=\"application/json\", endpoint=\"https\", group=\"extensions\", instance=\"10.0.0.4:6443\", job=\"apiserver\", namespace=\"default\", resource=\"deployments\", scope=\"namespace\", service=\"kubernetes\", severity=\"warning\", verb=\"DELETE\", version=\"v1beta1\"} => 1
...

tnozicka · 2019-11-11T15:47:40Z

sure, it should fail, yet I see 2 run where it was green https://prow.svc.ci.openshift.org/pr-history/?org=openshift&repo=origin&pr=24071

I am working on a fix, tracing down some more offenders

paulfantom · 2019-11-11T16:06:35Z

sure, I should fail, yet I see 2 run where it was green

Confirmed. Looks like we need to rewrite those tests to be running during a whole test suite run ASAP.

paulfantom · 2019-11-21T09:09:37Z

Rebased to resolve merge conflict

paulfantom · 2019-11-25T16:48:03Z

/retest

paulfantom · 2019-11-26T15:22:41Z

@tnozicka what is the status of fixing UsingDeprecatedAPIExtensionsV1Beta1 alerts? Asking because I just started 4.3.0-0.ci-2019-11-22-122829 and got 7 alerts with warning type and this isn't something we would like to ship to customers.

paulfantom · 2019-12-10T09:03:47Z

/retest

paulfantom · 2019-12-10T11:56:14Z

/hold cancel
/retest

…precatedAPIExtensionsV1Beta1 alerts are not firing

paulfantom · 2019-12-16T11:16:43Z

Rebased after changes from #24276

lilic

/lgtm

🤞

openshift-ci-robot · 2019-12-16T12:27:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: LiliC, paulfantom, s-urbaniak, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~test/extended/prometheus/OWNERS~~ [LiliC,paulfantom,s-urbaniak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2019-12-16T12:29:17Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-16T12:55:18Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-12-16T13:21:18Z

/retest

Please review the full test history for this PR and help us cut down flakes.

lilic · 2019-12-16T14:39:53Z

@tnozicka FYI in the latest nightly 4.4 cluster the UsingDeprecatedAPIExtensionsV1Beta1 is still firing, when will this issue be resolved do you know?

openshift-bot · 2019-12-16T14:40:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

lilic · 2019-12-16T14:42:38Z

Also the UsingDeprecatedAPIExtensionsV1Beta1 is firing 20 times on my nightly 4.4 cluster, this should be amended to group those and only fire once, not sure how the alert is created?

lilic · 2019-12-16T14:51:04Z

/hold

Until the alert is fixed, so we don't keep retrying.

tnozicka · 2019-12-17T09:11:02Z

looks like people managed to merge new offenders in the meantime
one fix is here #24305

lilic · 2019-12-18T09:39:28Z

@tnozicka Yeah, guess this is why we need to enable them ASAP. Have you considered adding an e2e test to origin instead of an alert for this? As this will be firing on user clusters and I assume a lot of confusion will happen as users can't do anything about this from their side.

p0lyn0mial · 2019-12-20T13:25:13Z

/retest

soltysh · 2019-12-20T14:48:36Z

With the updated k8s where we DO NOT enable extensions/v1beta1 every tests that uses it should fail so this is free to merge.
/hold cancel

openshift-ci-robot requested review from lilic, s-urbaniak and simonpasquier November 1, 2019 14:35

openshift-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 1, 2019

openshift-ci-robot assigned s-urbaniak Nov 1, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 1, 2019

simonpasquier reviewed Nov 4, 2019

View reviewed changes

paulfantom force-pushed the test_alerts branch from bf0e2e6 to 0dcb9b3 Compare November 4, 2019 12:12

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Nov 4, 2019

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 4, 2019

simonpasquier reviewed Nov 4, 2019

View reviewed changes

openshift-ci-robot assigned simonpasquier Nov 4, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 4, 2019

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 7, 2019

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 7, 2019

paulfantom force-pushed the test_alerts branch from 0dcb9b3 to ffdb38d Compare November 21, 2019 09:09

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Nov 21, 2019

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 10, 2019

test/extended/prometheus: check if ClusterMachineApproverDown UsingDe…

71a7b15

…precatedAPIExtensionsV1Beta1 alerts are not firing

paulfantom force-pushed the test_alerts branch from ffdb38d to 71a7b15 Compare December 16, 2019 11:16

lilic approved these changes Dec 16, 2019

View reviewed changes

openshift-ci-robot assigned lilic Dec 16, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 16, 2019

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 16, 2019

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 20, 2019

openshift-merge-robot merged commit 51e7bbc into openshift:master Dec 20, 2019

paulfantom deleted the test_alerts branch December 21, 2019 21:31

check if ClusterMachineApproverDown UsingDeprecatedAPIExtensionsV1Beta1 alerts are not firing #24071

check if ClusterMachineApproverDown UsingDeprecatedAPIExtensionsV1Beta1 alerts are not firing #24071

Conversation

paulfantom commented Nov 1, 2019

s-urbaniak commented Nov 1, 2019

paulfantom commented Nov 1, 2019

simonpasquier Nov 4, 2019

Choose a reason for hiding this comment

lilic commented Nov 4, 2019

paulfantom commented Nov 4, 2019

simonpasquier left a comment

Choose a reason for hiding this comment

tnozicka commented Nov 6, 2019

tnozicka commented Nov 6, 2019

tnozicka commented Nov 7, 2019

openshift-bot commented Nov 7, 2019

tnozicka commented Nov 7, 2019 • edited

tnozicka commented Nov 7, 2019 • edited

tnozicka commented Nov 11, 2019

paulfantom commented Nov 11, 2019

tnozicka commented Nov 11, 2019 • edited

paulfantom commented Nov 11, 2019

paulfantom commented Nov 21, 2019

paulfantom commented Nov 25, 2019

paulfantom commented Nov 26, 2019

paulfantom commented Dec 10, 2019

paulfantom commented Dec 10, 2019

paulfantom commented Dec 16, 2019

lilic left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Dec 16, 2019

openshift-bot commented Dec 16, 2019

openshift-bot commented Dec 16, 2019

openshift-bot commented Dec 16, 2019

lilic commented Dec 16, 2019

openshift-bot commented Dec 16, 2019

lilic commented Dec 16, 2019

lilic commented Dec 16, 2019

tnozicka commented Dec 17, 2019

lilic commented Dec 18, 2019

p0lyn0mial commented Dec 20, 2019

soltysh commented Dec 20, 2019

tnozicka commented Nov 7, 2019 •

edited

tnozicka commented Nov 7, 2019 •

edited

tnozicka commented Nov 11, 2019 •

edited