Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestTargetScrapeTimeout is flaky #830

Closed
swsnider opened this Issue Jun 23, 2015 · 5 comments

Comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@swsnider @fabxc
@swsnider
Copy link
Contributor

swsnider commented Jun 23, 2015

When running a build from a completely clean copy of HEAD(cc18191), except for a patch that comments out the humanizeTimestamp test from #829, the TestTargetScrapeTimeout test fails often, but not always, when running 'make'.

failing run:

?       _/prometheus/cmd/prometheus [no test files]
?       _/prometheus/cmd/promtool   [no test files]
ok      _/prometheus/config 0.090s
ok      _/prometheus/notification   0.020s
ok      _/prometheus/promql 2.105s
--- FAIL: TestTargetScrapeTimeout (0.09s)
    target_test.go:312: Get http://127.0.0.1:50121/metrics: dial tcp 127.0.0.1:50121: i/o timeout
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
FAIL
FAIL    _/prometheus/retrieval  1.090s
ok      _/prometheus/retrieval/discovery    0.244s
ok      _/prometheus/rules  0.328s
?       _/prometheus/storage    [no test files]
ok      _/prometheus/storage/local  7.145s
ok      _/prometheus/storage/local/codable  0.028s
?       _/prometheus/storage/local/index    [no test files]
ok      _/prometheus/storage/metric 0.017s
ok      _/prometheus/storage/remote 0.026s
ok      _/prometheus/storage/remote/influxdb    0.013s
ok      _/prometheus/storage/remote/opentsdb    0.009s
ok      _/prometheus/template   0.202s
?       _/prometheus/tools/rule_checker [no test files]
?       _/prometheus/util/cli   [no test files]
ok      _/prometheus/util/flock 0.017s
?       _/prometheus/util/httputil  [no test files]
?       _/prometheus/util/route [no test files]
?       _/prometheus/util/stats [no test files]
?       _/prometheus/util/strutil   [no test files]
?       _/prometheus/util/testutil  [no test files]
?       _/prometheus/version    [no test files]
?       _/prometheus/web    [no test files]
ok      _/prometheus/web/api/legacy 0.239s
ok      _/prometheus/web/api/v1 0.116s
?       _/prometheus/web/blob   [no test files]
make: *** [test] Error 1
@swsnider

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

swsnider commented Jun 23, 2015

Full make output:

curl -o /prometheus/.build/cache/go1.4.2.linux-amd64.tar.gz -L https://golang.org/dl/go1.4.2.linux-amd64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 59.5M  100 59.5M    0     0  5444k      0  0:00:11  0:00:11 --:--:-- 6253k
tar -C /prometheus/.build/root -xzf /prometheus/.build/cache/go1.4.2.linux-amd64.tar.gz
touch /prometheus/.build/root/go/bin/go
cp -a /prometheus/Godeps/_workspace /prometheus/.build/root/gopath
ln -s /prometheus /prometheus/.build/root/gopath/src/github.com/prometheus/prometheus
make -C tools
make[1]: Entering directory `/prometheus/tools'
make -C rule_checker
make[2]: Entering directory `/prometheus/tools/rule_checker'
TMPDIR=/tmp GOROOT=/prometheus/.build/root/go GOPATH=/prometheus/.build/root/gopath /prometheus/.build/root/go/bin/go build -o rule_checker .
make[2]: Leaving directory `/prometheus/tools/rule_checker'
make[1]: Leaving directory `/prometheus/tools'
TMPDIR=/tmp GOROOT=/prometheus/.build/root/go GOPATH=/prometheus/.build/root/gopath /prometheus/.build/root/go/bin/go build -o prometheus -ldflags " -X github.com/prometheus/prometheus/version.Version 0.14.0 -X github.com/prometheus/prometheus/version.Revision cc18191 -X github.com/prometheus/prometheus/version.Branch master -X github.com/prometheus/prometheus/version.BuildUser @6079fc5222da -X github.com/prometheus/prometheus/version.BuildDate 20150623-18:10:52 -X github.com/prometheus/prometheus/version.GoVersion 1.4.2" github.com/prometheus/prometheus/cmd/prometheus
TMPDIR=/tmp GOROOT=/prometheus/.build/root/go GOPATH=/prometheus/.build/root/gopath /prometheus/.build/root/go/bin/go build -o promtool -ldflags " -X github.com/prometheus/prometheus/version.Version 0.14.0 -X github.com/prometheus/prometheus/version.Revision cc18191 -X github.com/prometheus/prometheus/version.Branch master -X github.com/prometheus/prometheus/version.BuildUser @6079fc5222da -X github.com/prometheus/prometheus/version.BuildDate 20150623-18:10:52 -X github.com/prometheus/prometheus/version.GoVersion 1.4.2" github.com/prometheus/prometheus/cmd/promtool
TMPDIR=/tmp GOROOT=/prometheus/.build/root/go GOPATH=/prometheus/.build/root/gopath /prometheus/.build/root/go/bin/go test -short ./...
?       _/prometheus/cmd/prometheus [no test files]
?       _/prometheus/cmd/promtool   [no test files]
ok      _/prometheus/config 0.090s
ok      _/prometheus/notification   0.020s
ok      _/prometheus/promql 2.105s
--- FAIL: TestTargetScrapeTimeout (0.09s)
    target_test.go:312: Get http://127.0.0.1:50121/metrics: dial tcp 127.0.0.1:50121: i/o timeout
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
time="2015-06-23T18:12:03Z" level=info msg="Starting target manager..." file=targetmanager.go line=75
time="2015-06-23T18:12:03Z" level=info msg="Stopping target manager..." file=targetmanager.go line=142
time="2015-06-23T18:12:03Z" level=info msg="Target manager stopped." file=targetmanager.go line=170
FAIL
FAIL    _/prometheus/retrieval  1.090s
ok      _/prometheus/retrieval/discovery    0.244s
ok      _/prometheus/rules  0.328s
?       _/prometheus/storage    [no test files]
ok      _/prometheus/storage/local  7.145s
ok      _/prometheus/storage/local/codable  0.028s
?       _/prometheus/storage/local/index    [no test files]
ok      _/prometheus/storage/metric 0.017s
ok      _/prometheus/storage/remote 0.026s
ok      _/prometheus/storage/remote/influxdb    0.013s
ok      _/prometheus/storage/remote/opentsdb    0.009s
ok      _/prometheus/template   0.202s
?       _/prometheus/tools/rule_checker [no test files]
?       _/prometheus/util/cli   [no test files]
ok      _/prometheus/util/flock 0.017s
?       _/prometheus/util/httputil  [no test files]
?       _/prometheus/util/route [no test files]
?       _/prometheus/util/stats [no test files]
?       _/prometheus/util/strutil   [no test files]
?       _/prometheus/util/testutil  [no test files]
?       _/prometheus/version    [no test files]
?       _/prometheus/web    [no test files]
ok      _/prometheus/web/api/legacy 0.239s
ok      _/prometheus/web/api/v1 0.116s
?       _/prometheus/web/blob   [no test files]
make: *** [test] Error 1
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jun 23, 2015

I've seen this a few times on travis. I assumed it was due to travis being unreasonably slow once in a while as I've never seen it on any of my machines.

#832 doubles the timeout - maybe you can checkout the branch and report back whether this solves the problem for you.
@swsnider

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

swsnider commented Jun 23, 2015

@fabxc That seems to fix it for me (of course one can never be sure with this sort of thing, but I would've expected to have seen it within the 5 times I tried, since it had been affecting 4 out of 5 builds previously)
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jun 25, 2015

Fixed in #832

@fabxc fabxc closed this Jun 25, 2015

simonpasquier pushed a commit to simonpasquier/prometheus that referenced this issue Oct 12, 2017

@grobie
Merge pull request prometheus#830 from jamiemoore/ec2_discovery_rolearn 
added new ec2 discovery role_arn option
42adaa6
@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 24, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.