Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky integration tests caused by ports mismatching between IPv4 and IPv6 #3

Merged
merged 1 commit into from
May 19, 2022

Conversation

pracucci
Copy link
Contributor

@pracucci pracucci commented May 19, 2022

We're experiencing flaky integration tests in Mimir repository, caused by the test being unable to find a specific metric exposed by /metrics. For example, a typical error we get is:

2022-05-19T10:08:41.1477379Z     querier_test.go:736:
2022-05-19T10:08:41.1477665Z         	Error Trace:	querier_test.go:736
2022-05-19T10:08:41.1477925Z         	Error:      	Received unexpected error:
2022-05-19T10:08:41.1478250Z         	            	metric not found

After a long investigation and try and error, I've the feeling I've found the root cause. When running tests in CI, the container port is both exposed on IPv4 and IPv6 address. Most of the times they match (e.g. same port number for both protocols) but sometimes they don't. An example is the following (I've added debug logs):

$ docker port e2e-mimir-test-distributor 8080
0.0.0.0:49268
:::49267

$ docker port e2e-mimir-test-ingester 8080
0.0.0.0:49269
:::49268

As you can see, distributor port was published on host's 49268 for IPv4. However, the host port 49268 for IPv6 is mapping the ingester!

Our code looks for IPv4 mapping (see here). When we fetch metrics, we're resolving hostname. Instead, we should ensure we connect to the IPv4 port, that's what I'm doing in this PR.

…and IPv6

Signed-off-by: Marco Pracucci <marco@pracucci.com>
@pracucci pracucci requested a review from pstibrany May 19, 2022 10:40
Copy link
Member

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find!

@pracucci pracucci merged commit 1db01e4 into main May 19, 2022
@pracucci pracucci deleted the fix-flaky-tests branch May 19, 2022 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants