Add crio metrics as an endpoint of nodes

[brancz]

As this needs a bit of a hack, I went ahead and did the relabelling rules to make this work.

@squat @s-urbaniak @mxinden @metalmatze

cc @runcom

[brancz]

Still testing this so putting a hold on this for now.

/hold

[brancz]

I am currently seeing a problem where consistently there are nodes that cannot be scraped. @runcom is investigating.

[runcom]

so, I've been testing this on a brand new cluster from today and I'm perfectly able to grab metrics from each node

[brancz]

Got it, in that case I will give it another try.

[brancz]

I think I have an idea what's happening. This seems to only happen to workers, I think the security group rules are probably off.

[runcom]

latest installer now has openshift/installer#1201 which should open ports for worker<->worker communication for metrics

[brancz]

Awesome. I'm going to give this a try soon.

[brancz]

Looking good! This is all the successful scrapes of crio of this e2e run:

Ready for a review now!

/hold cancel

[runcom]

lgtm (fwiw)

[mrunalp]

👍 lgtm

[s-urbaniak]

@brancz can we add the up query to the existing e2e tests too?

cluster-monitoring-operator/test/e2e/main_test.go

Lines 93 to 128 in e536f4f

	queries := []struct {
	query string
	expectN int
	}{
	{
	query: `up{job="node-exporter"} == 1`,
	expectN: 1,
	}, {
	query: `up{job="kubelet"} == 1`,
	expectN: 1,
	}, {
	query: `up{job="scheduler"} == 1`,
	expectN: 1,
	}, {
	query: `up{job="kube-controller-manager"} == 1`,
	expectN: 1,
	}, {
	query: `up{job="apiserver"} == 1`,
	expectN: 1,
	}, {
	query: `up{job="kube-state-metrics"} == 1`,
	expectN: 1,
	}, {
	query: `up{job="prometheus-k8s"} == 1`,
	expectN: 1,
	}, {
	query: `up{job="prometheus-operator"} == 1`,
	expectN: 1,
	}, {
	query: `up{job="alertmanager-main"} == 1`,
	expectN: 2,
	}, {
	query: `namespace:container_memory_usage_bytes:sum`,
	expectN: 1,
	},
	}

[brancz]

/retest

[brancz]

/retest

[s-urbaniak]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brancz, s-urbaniak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [brancz,s-urbaniak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[brancz]

/retest

[brancz]

/retest