Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: solaris builders are flaky #19369

Closed
bradfitz opened this issue Mar 2, 2017 · 2 comments

Comments

Projects
None yet
2 participants
@bradfitz
Copy link
Member

commented Mar 2, 2017

The solaris (smartos) builders are flaky.

The failures are often in TestServerAllowsBlockingRemoteAddr, as reproduced here:

$ gomote run user-bradfitz-solaris-amd64-smartosbuildlet-0 go/bin/go test -short net/http
--- FAIL: TestServerAllowsBlockingRemoteAddr (1.59s)
        serve_test.go:959: Request 1: Get http://127.0.0.1:38230: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
        serve_test.go:1008: response 1 addr = ""; want "RA:21.21.21.21:21"
2017/03/02 19:33:21 http2: server: error reading preface from client 127.0.0.1:50770: read tcp 127.0.0.1:57544->127.0.0.1:50770: use of closed file or network connection
FAIL
FAIL    net/http        7.934s

But running just that test (even 2000 times) is fine:

$ gomote run user-bradfitz-solaris-amd64-smartosbuildlet-0 go/bin/go test -short -run=TestServerAllowsBlockingRemoteAddr -count=2000 net/http
ok      net/http        7.161s

Note that the net/http tests only go passively parallel in -short mode, which results in a higher CPU rate. Are we getting throttled?

Notably, the tests pass when run without short:

$ gomote run user-bradfitz-solaris-amd64-smartosbuildlet-0 go/bin/go test net/http
ok      net/http        47.750s
$ gomote run user-bradfitz-solaris-amd64-smartosbuildlet-0 go/bin/go test net/http
ok      net/http        48.087s

Investigate.

/cc @4ad

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Mar 2, 2017

Oh, our 5 Joyent instances are "t4-standard-1G" which is 1G RAM, 1/2 CPU, 25G disk. Doesn't seem like enough CPU.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Mar 3, 2017

Quadrupled its CPU, wiped the red column, and things look happy now.

@bradfitz bradfitz closed this Mar 3, 2017

@golang golang locked and limited conversation to collaborators Mar 3, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.