Flaky test in scrape_test.go

[pdbogen]

The "Scraped samples did not match served metrics" test in scrape_test.go is flaky:

$ make test
>> running tests
ok      github.com/prometheus/prometheus/cmd/prometheus 0.030s
?       github.com/prometheus/prometheus/cmd/promtool   [no test files]
ok      github.com/prometheus/prometheus/config 0.012s
ok      github.com/prometheus/prometheus/notifier   0.021s
ok      github.com/prometheus/prometheus/promql 0.764s
--- FAIL: TestTargetScraperScrapeOK (0.00s)
    scrape_test.go:474: Scraped samples did not match served metrics
    scrape_test.go:475: Expected: [metric_a => 1 @[1457378821.545] metric_b => 2 @[1457378821.545]]
    scrape_test.go:476: Got: [metric_b => 2 @[1457378821.545] metric_a => 1 @[1457378821.545]]
FAIL
FAIL    github.com/prometheus/prometheus/retrieval  3.624s
ok      github.com/prometheus/prometheus/retrieval/discovery    0.066s
?       github.com/prometheus/prometheus/retrieval/discovery/kubernetes [no test files]
?       github.com/prometheus/prometheus/retrieval/discovery/marathon   [no test files]
ok      github.com/prometheus/prometheus/rules  0.053s
?       github.com/prometheus/prometheus/storage    [no test files]
ok      github.com/prometheus/prometheus/storage/local  4.733s
ok      github.com/prometheus/prometheus/storage/local/codable  0.002s
?       github.com/prometheus/prometheus/storage/local/index    [no test files]
?       github.com/prometheus/prometheus/storage/local/storagetool  [no test files]
ok      github.com/prometheus/prometheus/storage/metric 0.002s
ok      github.com/prometheus/prometheus/storage/remote 0.005s
ok      github.com/prometheus/prometheus/storage/remote/graphite    0.002s
ok      github.com/prometheus/prometheus/storage/remote/influxdb    0.006s
ok      github.com/prometheus/prometheus/storage/remote/opentsdb    0.014s
ok      github.com/prometheus/prometheus/template   0.043s
?       github.com/prometheus/prometheus/util/cli   [no test files]
ok      github.com/prometheus/prometheus/util/flock 0.002s
?       github.com/prometheus/prometheus/util/httputil  [no test files]
?       github.com/prometheus/prometheus/util/stats [no test files]
ok      github.com/prometheus/prometheus/util/strutil   0.001s
?       github.com/prometheus/prometheus/util/testutil  [no test files]
?       github.com/prometheus/prometheus/util/treecache [no test files]
?       github.com/prometheus/prometheus/version    [no test files]
ok      github.com/prometheus/prometheus/web    0.005s
ok      github.com/prometheus/prometheus/web/api/legacy 0.048s
ok      github.com/prometheus/prometheus/web/api/v1 0.040s
?       github.com/prometheus/prometheus/web/ui [no test files]
Makefile:29: recipe for target 'test' failed
make: *** [test] Error 1

$ make test
>> running tests
ok      github.com/prometheus/prometheus/cmd/prometheus 0.009s
?       github.com/prometheus/prometheus/cmd/promtool   [no test files]
ok      github.com/prometheus/prometheus/config 0.005s
ok      github.com/prometheus/prometheus/notifier   0.034s
ok      github.com/prometheus/prometheus/promql 0.791s
ok      github.com/prometheus/prometheus/retrieval  3.694s
ok      github.com/prometheus/prometheus/retrieval/discovery    0.066s
?       github.com/prometheus/prometheus/retrieval/discovery/kubernetes [no test files]
?       github.com/prometheus/prometheus/retrieval/discovery/marathon   [no test files]
ok      github.com/prometheus/prometheus/rules  0.047s
?       github.com/prometheus/prometheus/storage    [no test files]
ok      github.com/prometheus/prometheus/storage/local  4.179s
ok      github.com/prometheus/prometheus/storage/local/codable  0.005s
?       github.com/prometheus/prometheus/storage/local/index    [no test files]
?       github.com/prometheus/prometheus/storage/local/storagetool  [no test files]
ok      github.com/prometheus/prometheus/storage/metric 0.009s
ok      github.com/prometheus/prometheus/storage/remote 0.005s
ok      github.com/prometheus/prometheus/storage/remote/graphite    0.004s
ok      github.com/prometheus/prometheus/storage/remote/influxdb    0.015s
ok      github.com/prometheus/prometheus/storage/remote/opentsdb    0.004s
ok      github.com/prometheus/prometheus/template   0.055s
?       github.com/prometheus/prometheus/util/cli   [no test files]
ok      github.com/prometheus/prometheus/util/flock 0.002s
?       github.com/prometheus/prometheus/util/httputil  [no test files]
?       github.com/prometheus/prometheus/util/stats [no test files]
ok      github.com/prometheus/prometheus/util/strutil   0.001s
?       github.com/prometheus/prometheus/util/testutil  [no test files]
?       github.com/prometheus/prometheus/util/treecache [no test files]
?       github.com/prometheus/prometheus/version    [no test files]
ok      github.com/prometheus/prometheus/web    0.005s
ok      github.com/prometheus/prometheus/web/api/legacy 0.064s
ok      github.com/prometheus/prometheus/web/api/v1 0.033s
?       github.com/prometheus/prometheus/web/ui [no test files]

Reproduce with:

$ TEMP=`mktemp`; N=0; until make test 2>&1 | tee $TEMP | grep -q FAIL; do N=$(($N+1)); done; echo "Failure after $N successes"; cat $TEMP; rm -f $TEMP
Failure after 2 successes
>> running tests
ok      github.com/prometheus/prometheus/cmd/prometheus 0.014s
?       github.com/prometheus/prometheus/cmd/promtool   [no test files]
ok      github.com/prometheus/prometheus/config 0.011s
ok      github.com/prometheus/prometheus/notifier   0.034s
ok      github.com/prometheus/prometheus/promql 0.780s
--- FAIL: TestTargetScraperScrapeOK (0.00s)
    scrape_test.go:474: Scraped samples did not match served metrics
    scrape_test.go:475: Expected: [metric_a => 1 @[1457380428.348] metric_b => 2 @[1457380428.348]]
    scrape_test.go:476: Got: [metric_b => 2 @[1457380428.348] metric_a => 1 @[1457380428.348]]
FAIL
FAIL    github.com/prometheus/prometheus/retrieval  3.696s
ok      github.com/prometheus/prometheus/retrieval/discovery    0.065s
?       github.com/prometheus/prometheus/retrieval/discovery/kubernetes [no test files]
?       github.com/prometheus/prometheus/retrieval/discovery/marathon   [no test files]
ok      github.com/prometheus/prometheus/rules  0.059s
?       github.com/prometheus/prometheus/storage    [no test files]

[beorn7]

Running into this pretty often. Seems to be just the verification being sensitive to random ordering.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.