-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make master green again! #2002
Comments
Some specimens can be found in #1989 (excluding the travis_wait commits). There are 3 major types from this PR:
I don't see more beyond the 3 types. |
(#1984 has the most recent test I ran, and they all somehow passed) |
👍 yes please |
I'm seeing this in t0060-daemon.sh quite often.
passing one in 5 times maybe, leaving a ipfs binary running n the system in failure case which is confusing on the next run. Digging deeper, the daemon simply didn't start up in time... (Also found small print bug in pollEndpoint the meantime, PR incoming) edit: feel free to cherry pick f7fb258 |
@cryptix t0060 fails if you have a daemon already running. |
Doesn't properly kill the daemon after some test in https://circleci.com/gh/ipfs/go-ipfs/1981
|
Yes, but that failure looks different, the first cases fail. |
New failure on OSX: https://travis-ci.org/ipfs/go-ipfs/jobs/96669162#L147
And a timeout on OSX sharness: https://travis-ci.org/ipfs/go-ipfs/jobs/96669163
|
I like xmas and all, but the red is getting so annoying :S |
Wouldn't this be most effective if debugged by someone on osx? Is the err specific to travis osx only? |
Saw a green on 0.3.8: #2081 |
Yeah, it should be possible to bisect this by doing many tests on each commit instead of just one There is also this project that has a bisect that uses Bayesian Search Theory to find intermittent bugs: https://github.com/Ealdwulf/bbchop Yeah, it has not been updated for a long time but it might still be interesting. |
@chriscool the bisection alg code you linked is only possible if it can be run on travis. Or if the test fails are reproducible on osx, then sure bbchop can be used.
ok I will check 10x if this cures the test fails in general. |
Yeah, unfortunately it looks like CI services like Travis and Circle don't have a nice bisecting service (maybe we should suggest it to them), but I thought it could be interesting to talk about bbchop while at it.
Great, thanks! |
"git bisect bad" at 0.3.10: #2084.
gc trigger errs: refs -r errs: random-files errs: stalled: |
current master (df65bae) #2085:
gc trigger errs: 3 stalled:
linux sharness: I bet the most likely culprit is the gc PR #1495. |
Actually, pre-gc has build errs as well green
stalled:
refs -r: add bigfile and cat: There are 21 commits between https://github.com/ipfs/go-ipfs/commits/pregc and 0.3.9 |
I have narrowed down the osx build stall to this PR #1914 ...the culprit. |
@rht great detective work! but wait, one problem: the closenotify thing caused hangs in <go1.5.1. we moved to go1.5.2, and i updated the |
reran with 1.5.2, still build errs/stalls: #2088 |
The gc errs are likely due to race condition between periodic gc and pinning, when added files are gc-ed before they are pinned, then the pinning stalls since the blocks are not available locally nor remotely. |
stalled: https://travis-ci.org/ipfs/go-ipfs/jobs/99128512 |
I am having the timeout issue with sharness tests too #2119. #2120 adds fuse support for Travis CI but also switches the Ubuntu version to Trusty. However, I ran the tests multiple times (go1.5.2) and the timeout is my only issue, albeit, intermittent -- redoing a test can eventually yield green: https://travis-ci.org/thelinuxkid/go-ipfs/builds/98464151. https://github.com/thelinuxkid/travis-rerun might help with debugging. I was able to build the Trusty image Travis uses. I will try and see if I can replicate the timeout there when I am in town again next week. |
Failed on: t0200-unixfs-ls |
I think this effort was successful. Master is quite green lately |
There are a lot of very horrible CI failures lately, ranging from time-based, to plain worrying (like
ipfs add -r
hash mismatches).the failures
The text was updated successfully, but these errors were encountered: