Bug 1986061: Monitor openshift-network-diagnostics namespace #1190

creydr · 2021-08-26T10:52:09Z

CNO deploys a service monitor named "network-check-source" which is never picked up by prometheus.

This PR adds the openshift.io/cluster-monitoring="true" label to the openshift-network-diagnostics namespace and fixes some RBAC so the service is picked up for monitoring.

How to verify:

Spin up a cluster with this patch

Login to the cluster dashboard, go to Observe -> Metrics and query for one of the metrics of the checkendpoints pod, e.g. pod_network_connectivity_check_count

cluster-network-operator/pkg/cmd/checkendpoints/controller/metrics.go

Lines 22 to 35 in 425a698

    
           endpointCheckCounter = metrics.NewCounterVec(&metrics.CounterOpts{ 
        
           	Name: "pod_network_connectivity_check_count", 
        
           	Help: "Report status of pod network connectivity checks over time.", 
        
           }, []string{"component", "checkName", "targetEndpoint", "tcpConnect", "dnsResolve"}) 
        
           tcpConnectLatencyGauge = metrics.NewGaugeVec(&metrics.GaugeOpts{ 
        
           	Name: "pod_network_connectivity_check_tcp_connect_latency_gauge", 
        
           	Help: "Report latency of TCP connect to target endpoint over time.", 
        
           }, []string{"component", "checkName", "targetEndpoint"}) 
        
           dnsResolveLatencyGauge = metrics.NewGaugeVec(&metrics.GaugeOpts{ 
        
           	Name: "pod_network_connectivity_check_dns_resolve_latency_gauge", 
        
           	Help: "Report latency of DNS resolve of target endpoint over time.", 
        
           }, []string{"component", "checkName", "targetEndpoint"})

openshift-ci · 2021-08-26T10:52:13Z

@creydr: This pull request references Bugzilla bug 1986061, which is invalid:

expected the bug to target the "4.9.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1986061: Monitor openshift-network-diagnostics namespace

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-08-26T10:52:30Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Signed-off-by: Christoph Stäbler <cstabler@redhat.com>

openshift-ci · 2021-08-27T08:33:48Z

@creydr: This pull request references Bugzilla bug 1986061, which is invalid:

expected the bug to target the "4.9.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1986061: Monitor openshift-network-diagnostics namespace

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

creydr · 2021-08-27T08:35:47Z

/bugzilla refresh

openshift-ci · 2021-08-27T08:35:55Z

@creydr: This pull request references Bugzilla bug 1986061, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

creydr · 2021-08-27T10:45:08Z

/bugzilla refresh
/retest required

openshift-ci · 2021-08-27T10:45:12Z

@creydr: This pull request references Bugzilla bug 1986061, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @zhaozhanqi

In response to this:

/bugzilla refresh
/retest required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-08-27T10:45:29Z

@creydr: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test e2e-agnostic-upgrade
/test e2e-aws-ovn-windows
/test e2e-aws-sdn-multi
/test e2e-gcp
/test e2e-gcp-ovn
/test e2e-metal-ipi-ovn-ipv6
/test images
/test unit
/test verify

The following commands are available to trigger optional jobs:

/test e2e-azure-ovn
/test e2e-azure-ovn-dualstack
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6-ipsec
/test e2e-openstack
/test e2e-openstack-kuryr
/test e2e-openstack-ovn
/test e2e-ovn-hybrid-step-registry
/test e2e-ovn-ipsec-step-registry
/test e2e-ovn-step-registry
/test e2e-vsphere-ovn
/test e2e-vsphere-windows

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-cluster-network-operator-master-e2e-agnostic-upgrade
pull-ci-openshift-cluster-network-operator-master-e2e-aws-ovn-windows
pull-ci-openshift-cluster-network-operator-master-e2e-aws-sdn-multi
pull-ci-openshift-cluster-network-operator-master-e2e-azure-ovn
pull-ci-openshift-cluster-network-operator-master-e2e-gcp
pull-ci-openshift-cluster-network-operator-master-e2e-gcp-ovn
pull-ci-openshift-cluster-network-operator-master-e2e-gcp-ovn-upgrade
pull-ci-openshift-cluster-network-operator-master-e2e-metal-ipi-ovn-ipv6
pull-ci-openshift-cluster-network-operator-master-e2e-metal-ipi-ovn-ipv6-ipsec
pull-ci-openshift-cluster-network-operator-master-e2e-openstack-ovn
pull-ci-openshift-cluster-network-operator-master-e2e-ovn-hybrid-step-registry
pull-ci-openshift-cluster-network-operator-master-e2e-ovn-ipsec-step-registry
pull-ci-openshift-cluster-network-operator-master-e2e-ovn-step-registry
pull-ci-openshift-cluster-network-operator-master-e2e-vsphere-ovn
pull-ci-openshift-cluster-network-operator-master-e2e-vsphere-windows
pull-ci-openshift-cluster-network-operator-master-images
pull-ci-openshift-cluster-network-operator-master-unit
pull-ci-openshift-cluster-network-operator-master-verify

In response to this:

/bugzilla refresh
/retest required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

creydr · 2021-08-27T10:45:57Z

/retest-required

creydr · 2021-08-27T12:07:34Z

/retest

creydr · 2021-08-27T14:01:31Z

/retest-required

openshift-bot · 2021-09-03T08:19:33Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-09-03T09:43:32Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-09-03T11:50:47Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

abhat · 2021-09-03T14:09:49Z

/override ci/prow/e2e-gcp-ovn
Fails because of: https://bugzilla.redhat.com/show_bug.cgi?id=2000589

openshift-ci · 2021-09-03T14:14:38Z

@abhat: Overrode contexts on behalf of abhat: ci/prow/e2e-gcp-ovn

In response to this:

/override ci/prow/e2e-gcp-ovn
Fails because of: https://bugzilla.redhat.com/show_bug.cgi?id=2000589

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-09-03T14:26:53Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-09-03T15:05:47Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

dcbw · 2021-09-03T17:21:51Z

/retest

openshift-bot · 2021-09-03T19:50:43Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-09-03T20:37:27Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-09-03T20:49:17Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

dcbw · 2021-09-03T21:04:51Z

/override ci/prow/e2e-gcp
openshift/origin#26444

openshift-bot · 2021-09-03T21:11:47Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-09-03T21:14:14Z

@dcbw: Overrode contexts on behalf of dcbw: ci/prow/e2e-gcp

In response to this:

/override ci/prow/e2e-gcp
openshift/origin#26444

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-09-03T21:35:46Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-09-03T21:47:47Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-09-03T21:59:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

dcbw · 2021-09-03T23:26:50Z

Still openshift/origin#26444
/retest-required

openshift-bot · 2021-09-04T04:15:26Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-09-04T05:07:24Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-09-04T05:20:25Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

dcbw · 2021-09-04T13:44:37Z

/retest-required

openshift-bot · 2021-09-04T13:58:26Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-09-04T14:10:26Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-09-04T14:22:43Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-09-04T15:48:13Z

@creydr: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-azure-ovn	`71b5c3f`	link	`/test e2e-azure-ovn`
ci/prow/e2e-gcp-ovn-upgrade	`71b5c3f`	link	`/test e2e-gcp-ovn-upgrade`
ci/prow/e2e-vsphere-ovn	`71b5c3f`	link	`/test e2e-vsphere-ovn`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

dcbw · 2021-09-04T16:05:10Z

/override ci/prow/e2e-agnostic-upgrade
https://bugzilla.redhat.com/show_bug.cgi?id=2001231

openshift-ci · 2021-09-04T16:05:49Z

@dcbw: Overrode contexts on behalf of dcbw: ci/prow/e2e-agnostic-upgrade

In response to this:

/override ci/prow/e2e-agnostic-upgrade
https://bugzilla.redhat.com/show_bug.cgi?id=2001231

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-09-04T16:06:33Z

@creydr: All pull requests linked via external trackers have merged:

openshift/cluster-network-operator#1190

Bugzilla bug 1986061 has been moved to the MODIFIED state.

In response to this:

Bug 1986061: Monitor openshift-network-diagnostics namespace

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot requested review from aojea and trozet August 26, 2021 10:52

creydr force-pushed the monitor-openshift-network-diagnostics-namespace branch 2 times, most recently from 585dccf to 81089ce Compare August 27, 2021 07:46

Monitor openshift-network-diagnostics namespace

cf06e2a

Signed-off-by: Christoph Stäbler <cstabler@redhat.com>

creydr force-pushed the monitor-openshift-network-diagnostics-namespace branch 2 times, most recently from 517904e to 7829aaf Compare August 27, 2021 08:26

Fix RBAC to monitor openshift-network-diagnostics namespace

71b5c3f

Signed-off-by: Christoph Stäbler <cstabler@redhat.com>

creydr force-pushed the monitor-openshift-network-diagnostics-namespace branch from 7829aaf to 71b5c3f Compare August 27, 2021 08:26

creydr marked this pull request as ready for review August 27, 2021 08:34

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 27, 2021

openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Aug 27, 2021

openshift-ci bot requested a review from zhaozhanqi August 27, 2021 08:35

openshift-ci bot removed the bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. label Aug 27, 2021

openshift-ci bot added the bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. label Aug 27, 2021

openshift-merge-robot merged commit 8437b07 into openshift:master Sep 4, 2021

	endpointCheckCounter = metrics.NewCounterVec(&metrics.CounterOpts{
	Name: "pod_network_connectivity_check_count",
	Help: "Report status of pod network connectivity checks over time.",
	}, []string{"component", "checkName", "targetEndpoint", "tcpConnect", "dnsResolve"})

	tcpConnectLatencyGauge = metrics.NewGaugeVec(&metrics.GaugeOpts{
	Name: "pod_network_connectivity_check_tcp_connect_latency_gauge",
	Help: "Report latency of TCP connect to target endpoint over time.",
	}, []string{"component", "checkName", "targetEndpoint"})

	dnsResolveLatencyGauge = metrics.NewGaugeVec(&metrics.GaugeOpts{
	Name: "pod_network_connectivity_check_dns_resolve_latency_gauge",
	Help: "Report latency of DNS resolve of target endpoint over time.",
	}, []string{"component", "checkName", "targetEndpoint"})

Bug 1986061: Monitor openshift-network-diagnostics namespace #1190

Bug 1986061: Monitor openshift-network-diagnostics namespace #1190

Conversation

creydr commented Aug 26, 2021 • edited

openshift-ci bot commented Aug 26, 2021

openshift-ci bot commented Aug 26, 2021

openshift-ci bot commented Aug 27, 2021

creydr commented Aug 27, 2021

openshift-ci bot commented Aug 27, 2021

creydr commented Aug 27, 2021

openshift-ci bot commented Aug 27, 2021

openshift-ci bot commented Aug 27, 2021

creydr commented Aug 27, 2021

creydr commented Aug 27, 2021

creydr commented Aug 27, 2021

openshift-bot commented Sep 3, 2021

openshift-bot commented Sep 3, 2021

openshift-bot commented Sep 3, 2021

abhat commented Sep 3, 2021

openshift-ci bot commented Sep 3, 2021

openshift-bot commented Sep 3, 2021

openshift-bot commented Sep 3, 2021

dcbw commented Sep 3, 2021

openshift-bot commented Sep 3, 2021

openshift-bot commented Sep 3, 2021

openshift-bot commented Sep 3, 2021

dcbw commented Sep 3, 2021

openshift-bot commented Sep 3, 2021

openshift-ci bot commented Sep 3, 2021

openshift-bot commented Sep 3, 2021

openshift-bot commented Sep 3, 2021

openshift-bot commented Sep 3, 2021

dcbw commented Sep 3, 2021

openshift-bot commented Sep 4, 2021

openshift-bot commented Sep 4, 2021

openshift-bot commented Sep 4, 2021

dcbw commented Sep 4, 2021

openshift-bot commented Sep 4, 2021

openshift-bot commented Sep 4, 2021

openshift-bot commented Sep 4, 2021

openshift-ci bot commented Sep 4, 2021 • edited

dcbw commented Sep 4, 2021

openshift-ci bot commented Sep 4, 2021

openshift-ci bot commented Sep 4, 2021

creydr commented Aug 26, 2021 •

edited

openshift-ci bot commented Sep 4, 2021 •

edited