[Prometheus Exporter] Fix Issue 6228: Flaky End to End Test #6697
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dashpole
reviewed
Dec 10, 2021
dashpole
reviewed
Dec 10, 2021
| @@ -106,7 +107,7 @@ func TestEndToEndSummarySupport(t *testing.T) { | |||
| PrometheusConfig: receiverConfig, | |||
| ReceiverSettings: config.NewReceiverSettings(config.NewComponentID("prometheus")), | |||
| } | |||
| // 3.5 Create the Prometheus receiver and pass in the preivously created Prometheus exporter. | |||
| // 3.5 Create the Prometheus receiver and pass in the previously created Prometheus exporter. | |||
There was a problem hiding this comment.
Any idea why we need a prometheus receiver here? We don't seem to do anything with it, and the extra calls it makes seem disruptive to the rest of the test.
There was a problem hiding this comment.
The receiver handles scraping the mocked exposition and converting it to pdata, then passing it to the exporter.
dashpole
reviewed
Dec 10, 2021
| for i := 0; i < 8; i++ { | ||
| <-waitForScrape | ||
| } | ||
| // 4. Scrape from the Prometheus receiver to ensure that we export summary metrics |
There was a problem hiding this comment.
We are scraping from the prometheus exporter, not receiver.
There was a problem hiding this comment.
The scrapes come from the receiver. That is what the wg.Wait() is waiting for. Once the receiver has scraped the target enough times then the wait group is resolved.
Aneurysm9
approved these changes
Dec 10, 2021
Aneurysm9
added
the
ready to merge
PaurushGarg
referenced
this pull request
in open-o11y/opentelemetry-collector-contrib
Dec 11, 2021
* non-blocking serve for drop wizard * Removing previous typos * Resolving feedback comments
PaurushGarg
referenced
this pull request
in open-o11y/opentelemetry-collector-contrib
Dec 11, 2021
* non-blocking serve for drop wizard * Removing previous typos * Resolving feedback comments
PaurushGarg
referenced
this pull request
in open-o11y/opentelemetry-collector-contrib
Dec 11, 2021
* non-blocking serve for drop wizard * Removing previous typos * Resolving feedback comments
PaurushGarg
referenced
this pull request
in open-o11y/opentelemetry-collector-contrib
Dec 14, 2021
* non-blocking serve for drop wizard * Removing previous typos * Resolving feedback comments
6 tasks
povilasv
referenced
this pull request
in coralogix/opentelemetry-collector-contrib
Dec 19, 2022
Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com> Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
Current design has buffer capacity of 1 for
waitForScrapeschannel, and may not be enough forscrape_intervalof 2ms.After ensuring that receiver has scraped target from test server 8 times; and before the GET request is made to fetch metrics from the exposed Prometheus exporter end points -- the Prometheus Receiver keeps trying to scrape metrics and when it does not get any response (as response is blocked because channel has only 1 buffer capacity against 2ms
scrape_interval) Prometheus Receiver generates failed scrapes (staleNaNand so on) - and that's why this test is flaky.So solution is to avoid generation of failed scrapes by ensuring that there is always a response to Prometheus Receiver GET requests from the test server.
dropWizard.Link to tracking Issue:
Issue #6228
cc @alolita @Aneurysm9