random test failure: api not running

[whyrusleeping]

Saw this one, it happens more frequently than i'd like: https://travis-ci.org/ipfs/go-ipfs/jobs/158756858

Worth noting that i see it across different tests, so its not an issue in the individual tests

[whyrusleeping]

Looks like another of the same: https://s3.amazonaws.com/archive.travis-ci.org/jobs/159382429/log.txt

[whyrusleeping]

Another one: https://travis-ci.org/ipfs/go-ipfs/jobs/159304039

[whyrusleeping]

This one is happening pretty often on t0400... https://s3.amazonaws.com/archive.travis-ci.org/jobs/159432159/log.txt

[whyrusleeping]

@chriscool is there any way to tel sharness to print out daemon_err at the end of a test if it doesnt pass?

[chriscool]

@whyrusleeping using || test_fsh cat daemon_err should work.

This is done for example in test_launch_ipfs_daemon() in test-lib.sh:

https://github.com/ipfs/go-ipfs/blob/master/test/sharness/lib/test-lib.sh#L227

[chriscool]

In https://travis-ci.org/ipfs/go-ipfs/jobs/159304039, the first suspicious thing is:

expecting success: 
        ipfs add mountdir/bigfile >actual
17:41:36.344 ERROR commands/h: unexpected EOF client.go:247

In https://s3.amazonaws.com/archive.travis-ci.org/jobs/159432159/log.txt there is:

�[36mexpecting success: 
  HASH=$(echo "testing" | ipfs add -q)
  test_curl_resp_http_code "http://127.0.0.1:$API_PORT/ipfs/$HASH" "HTTP/1.1 404 Not Found"
�(B�[m
�[0;37m21:48:26.719 �[31mERROR �[0;34mcommands/h: �[0munexpected EOF �[0;37mclient.go:247�[0m
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to 127.0.0.1 port 53294: Connection refused
curl error with url: 'http://127.0.0.1:53294/ipfs/QmNYERzV2LfD2kkfahtfv44ocHzEFK1sLBaE7zdcYT2GAZ'
curl output was:

So we can see another instance of "ERROR commands/h: unexpected EOF client.go:247".

This corresponds to the following code:

https://github.com/ipfs/go-ipfs/blob/master/commands/http/client.go#L247

Maybe the error should be better handled in the go code?

[whyrusleeping]

Another: https://circleci.com/gh/ipfs/go-ipfs/5076?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

@chriscool Good catches there... It seems the daemon must be crashing during that add.

The unexpected EOF is printed because we werent expecting the http api server to drop our connection suddenly

[chriscool]

@whyrusleeping in the log you just pointed to, there is:

expecting success: 
        ipfs add -t mountdir/hello.txt >actual

02:54:47.854 ERROR commands/h: unexpected EOF client.go:247
ok 24 - ipfs add -t succeeds

and then there are failures in the following tests.

My opinion is that the ipfs add -t mountdir/hello.txt actually failed, but returns a 0 exit code as if it had succeeded. And I think it is a bug.

If the api server drops the connection unexpectedly instead of returning a proper OK, then we should consider that there is an error and we should not exit with code 0.

So I think the code in client.go around line 247 is buggy.

[whyrusleeping]

@chriscool oh! good point

[whyrusleeping]

resolved in #3271