Bug 1752814: Fix decreasing counter metrics when reloading HAProxy

[sgreene570]

The HAProxy router reload process calls CollectNow() in pkg/router/metrics/haproxy/haproxy.go. Currently, CollectNow() does not update e.lastScrape, which means it is possible for the normal scrape function Collect(...) to be called just after CollectNow(), but before HAProxy has actually reloaded, since CollectNow() essentially "resets" the metrics scrape interval.

The idea in this PR is to have CollectNow() set the e.lastScrape time, since CollectNow() is calling scrape() anyways. This will prevent counter metrics from decreasing across reloads since metrics are only saved for reload preservation during CollectNow() to minimize memory usage.

[openshift-ci-robot]

@sgreene570: This pull request references Bugzilla bug 1752814, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.6.0) matches configured target release for branch (4.6.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1752814: Fix decreasing counter metrics when reloading HAProxy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@sgreene570: This pull request references Bugzilla bug 1752814, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.6.0) matches configured target release for branch (4.6.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1752814: Fix decreasing counter metrics when reloading HAProxy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sgreene570]

/assign @frobware @Miciah
/cc @danehans @knobunc

[frobware]

I would be interested to know whether this changes the behaviour of openshift/origin#25484

[sgreene570]

I would be interested to know whether this changes the behaviour of openshift/origin#25484

Definitely could be the case.
Let me try to reproduce the test failure from openshift/origin#25484 and see if this PR remedies the situation at all.

[sgreene570]

/test e2e

[sgreene570]

/test e2e

[sgreene570]

Re-triggering e2e test to get an idea if this PR hits https://bugzilla.redhat.com/show_bug.cgi?id=1835371

/test e2e

[sgreene570]

/test e2e

[sgreene570]

gcp quota problems. Will retest later today.

[sgreene570]

/retest

[sgreene570]

/retest

[sgreene570]

/test e2e

[sgreene570]

/test e2e

[sgreene570]

/test e2e

[sgreene570]

/retest

[danehans]

@sgreene570 should pkg/router/metrics/haproxy/haproxy_test.go be updated? It looks like the test is setting e.lastScrape which should be removed due to the changes introduced by this PR, right?

[sgreene570]

@sgreene570 should pkg/router/metrics/haproxy/haproxy_test.go be updated? It looks like the test is setting e.lastScrape which should be removed due to the changes introduced by this PR, right?

Good point! Thanks for catching this.
Some of the time in haproxy_test.go, e.lastScrape is set intentionally to test that the scrape interval is working. That being said, I did find one instance of e.lastScrape = that can be removed from the test file in this PR. Thanks!

[sgreene570]

go: github.com/openshift/library-go@v0.0.0-20200324092245-db2a8546af81 requires
	bitbucket.org/ww/goautoneg@v0.0.0-20120707110453-75cd24fc2f2c: reading https://api.bitbucket.org/2.0/repositories/ww/goautoneg?fields=scm: 404 Not Found

Looks like library-go needs to be bumped for the router as well. I'll amends this bump to #187

[sgreene570]

Blocked on #187

[sgreene570]

/test verify

[danehans]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danehans, sgreene570

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sgreene570]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci-robot]

@sgreene570: All pull requests linked via external trackers have merged:

• openshift/router#179

Bugzilla bug 1752814 has been moved to the MODIFIED state.

In response to this:

Bug 1752814: Fix decreasing counter metrics when reloading HAProxy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sgreene570]

/cherry-pick release-4.5

[openshift-cherrypick-robot]

@sgreene570: new pull request created: #205

In response to this:

/cherry-pick release-4.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sgreene570]

/cherry-pick release-4.4

[openshift-cherrypick-robot]

@sgreene570: new pull request created: #206

In response to this:

/cherry-pick release-4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.