Fix flaky integration tests caused by ports mismatching between IPv4 and IPv6 #3
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We're experiencing flaky integration tests in Mimir repository, caused by the test being unable to find a specific metric exposed by
/metrics
. For example, a typical error we get is:After a long investigation and try and error, I've the feeling I've found the root cause. When running tests in CI, the container port is both exposed on IPv4 and IPv6 address. Most of the times they match (e.g. same port number for both protocols) but sometimes they don't. An example is the following (I've added debug logs):
As you can see, distributor port was published on host's 49268 for IPv4. However, the host port 49268 for IPv6 is mapping the ingester!
Our code looks for IPv4 mapping (see here). When we fetch metrics, we're resolving
hostname
. Instead, we should ensure we connect to the IPv4 port, that's what I'm doing in this PR.