[NO MERGE] Travis: Run nightly test suite #7148

MarcoFalke · 2015-12-01T18:50:55Z

Ideally we want to detect fails in the extended suite early (i.e. in the PR which caused the fail). Though, this would double the runtime of the rpc tests in every PR from 5 to 10 minutes (with pruning.py disabled).

Also, we'd want to collect --coverage data until all rpcs are covered and then fail on uncovered (new) rpcs.

For now, every 24 hours a fresh nightly build is triggered within this PR.

Current status:

Disable usual rpc tests and make check, otherwise travis will block the build exceeding the 2 hour max CPU time.
Disable pruning.py
Run extended coverage rpc tests once with CPPFLAGS=-DDEBUG_LOCKORDER
Verify subtrees
~~Enable qt4 build~~

MarcoFalke · 2015-12-01T18:51:35Z

Travis vanilla will not support nightly builds, but there are 3rd party apps available if this is a direction to take:
travis-ci/travis-ci#582 (comment)
travis-ci/travis-ci#582 (comment)

jonasschnelli · 2015-12-02T07:23:32Z

Am i right: every 24h this PR get's updated with the current master (over an external script) and makes this PR act as a nightly build?

MarcoFalke · 2015-12-02T07:41:55Z

Yes, pretty much like that right now.

laanwj · 2015-12-03T10:02:55Z

Seems like a good idea.

MarcoFalke · 2015-12-09T17:47:06Z

Nice, looks like this caught it's first fish. (I should set up an alert or something)

MarcoFalke · 2015-12-09T17:47:15Z

https://travis-ci.org/bitcoin/bitcoin/builds/95840859 failed with

Running 2nd level testscript maxuploadtarget.py ...
start_node: bitcoind started, calling bitcoin-cli -rpcwait getblockcount
start_node: calling bitcoin-cli -rpcwait getblockcount returned
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11523
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11523
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11523
Assertion failed: 68 != 69
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/test_framework.py", line 135, in main
    self.run_test()
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/maxuploadtarget.py", line 203, in run_test
    assert_equal(test_nodes[0].block_receive_map[big_old_block], i+1)
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/util.py", line 394, in assert_equal
    raise AssertionError("%s != %s"%(str(thing1),str(thing2)))
Stopping nodes
Cleaning up
Failed
Traceback (most recent call last):
  File "qa/pull-tester/rpc-tests.py", line 255, in <module>
    runtests()
  File "qa/pull-tester/rpc-tests.py", line 174, in runtests
    rpcTestDir + testScriptsExt[i] + flags, shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/maxuploadtarget.py --srcdir /home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/src --coveragedir /tmp/coveragexHdFgP ' returned non-zero exit status 1

MarcoFalke · 2015-12-14T21:18:11Z

@sdaftuar maxuploadtarget failed again. Mind to look at this? (I couldn't reproduce locally)

Running 2nd level testscript maxuploadtarget.py ...
start_node: bitcoind started, calling bitcoin-cli -rpcwait getblockcount
start_node: calling bitcoin-cli -rpcwait getblockcount returned
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11593
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11593
MiniNode: Connecting to Bitcoin Node IP # 127.0.0.1:11593
Assertion failed: 2 != 3
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/test_framework.py", line 135, in main
    self.run_test()
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/maxuploadtarget.py", line 203, in run_test
    assert_equal(test_nodes[0].block_receive_map[big_old_block], i+1)
  File "/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/test_framework/util.py", line 394, in assert_equal
    raise AssertionError("%s != %s"%(str(thing1),str(thing2)))
Stopping nodes
Cleaning up
Failed
Traceback (most recent call last):
  File "qa/pull-tester/rpc-tests.py", line 256, in <module>
    runtests()
  File "qa/pull-tester/rpc-tests.py", line 175, in runtests
    rpcTestDir + testScriptsExt[i] + flags, shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/qa/rpc-tests/maxuploadtarget.py --srcdir /home/travis/build/bitcoin/bitcoin/bitcoin-i686-pc-linux-gnu/src --coveragedir /tmp/coveragesoVFH4 ' returned non-zero exit status 1

sdaftuar · 2015-12-14T22:04:07Z

@MarcoFalke Strange, I don't understand how that error can happen, but one thing I noticed is that there should be a with mininode_lock: guarding the assert_equal(test_nodes[0].block_receive_map[big_old_block]) line. Still, I can't come up with a logical scenario where there's actually a race condition there (but acquiring the lock is the right way to ensure there is no race, since both threads are updating that variable).

If we can't come up with other ideas, perhaps we can make that fix and see if the problem recurs...

laanwj · 2016-01-07T08:20:15Z

@MarcoFalke can you please include #7142 into this?

sipa · 2016-05-25T10:32:59Z

@arowser Have you even looked at this patch yourself? It is not intended to be merged, as it is automatically retriggered every 24h to detect bugs only caught by the extended tests. Please stop pinging every single pull request. Repository maintainers merge when there is sufficient agreement and review done, but they are not the only ones who do reviews. If you want to help merging things, please review yourself.

sipa · 2016-05-25T13:55:30Z

@arowser Furthermore, I consider the many comments you've left as spam, and I'll delete them when I encounter them. Please don't do that again, or I'll report it as abuse.

arowser · 2016-05-27T03:06:01Z

@sipa Sorry to everyone, I just try to create a jenkins build robot in for every pull request to support more build type that not build in travis like PPC/MIPS, but the jenkins robot auto add comments by error configure. I have remove all these comments now, sorry again for these spam comments.

MarcoFalke · 2016-06-13T10:54:50Z

Noting failure in feefilter:

  File "/home/travis/build/bitcoin/bitcoin/build/../qa/rpc-tests/p2p-feefilter.py", line 101, in run_test

    assert(allInvsMatch(txids, test_node))

sdaftuar · 2016-06-17T04:55:28Z

@MarcoFalke I figured out the feefilter issue. Looks like sometimes the second-to-last test generates transactions which don't relay within 10 seconds, and then get relayed after the feefilter is updated in the last test.

da2ce7 · 2017-02-20T08:09:51Z

@MarcoFalke, what is the status of this pull request? Do you wish to rebase?

laanwj · 2017-02-20T09:54:42Z

This pull is not intended to be merged, it just re-pushes (and re-bases) regularly to run Travis w/ full test suite, which is too slow for automatic use.

TheBlueMatt · 2017-02-20T16:19:36Z

Maybe the PR title should be updated to indicate this?

…

On February 20, 2017 4:54:46 AM EST, "Wladimir J. van der Laan" ***@***.***> wrote: This pull is not intended to be merged, it just re-pushes (and re-bases) regularly to run Travis w/ full test suite, which is too slow for automatic use. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #7148 (comment)

jnewbery · 2017-03-23T15:13:04Z

Now that Travis supports cron jobs, can this be replaced with something like #10052? I'm happy to help merging them if you think that's useful.


        [travis] verify subtrees, bitcoin/master commit

* Add draft to run a test against an old binary (0.12.1)


        [qa] Add script to check for datadir compatibility between versions


        [qa] travis: Only run extended rpc tests


        WIP: Don't merge. Trigger nightly build without pruning, rpcbind


        qa travis: --enable-werror


        qa: Stop only nodes that were started when one fails to start


        Run valgrind


        nightly: Check salvagewallet


        nightly: Only smoke test with valgrind


        nightly: Compat with 0.14.0 only for now

TheBlueMatt · 2017-04-13T13:56:37Z

I believe this can be closed now?

laanwj · 2017-04-13T14:08:41Z

I think so - @MarcoFalke please reopen if necessary.

jnewbery · 2017-04-13T14:20:27Z

This seems to include tests that aren't covered by the travis cron job, namely valgrind, testing salvagewallet and a new compatibility functional test. @MarcoFalke - do you think those should be merged into master to be run by the cron job? If so, I'm happy to help with merging/reviewing.

MarcoFalke · 2017-04-13T19:20:22Z

@jnewbery Most of the code here is just a reminder for myself to look further into that topic. Not really ready to be merged or put into our travis yaml, as the diff needs to be adjusted every couple of weeks.

jonasschnelli added the Tests label Dec 2, 2015

laanwj mentioned this pull request Dec 3, 2015

Travis: Test build against system libs (& Qt4) #7142

Merged

MarcoFalke mentioned this pull request Feb 13, 2016

Add bip68-sequence.py to extended rpc tests #7531

Merged

MarcoFalke changed the title ~~[WIP] Run extended nightly builds in travis~~ [WIP] Travis: Run extended nightly builds Mar 3, 2016

MarcoFalke changed the title ~~[WIP] Travis: Run extended nightly builds~~ [WIP] Travis: Run extended test suite Apr 2, 2016

MarcoFalke mentioned this pull request Apr 15, 2016

Potential deadlock assertion in net code on 0.12RC3 #7470

Closed

MarcoFalke mentioned this pull request Jul 8, 2016

[qa] feefilter: Clear mempool after each check #8316

Closed

sdaftuar mentioned this pull request Aug 1, 2016

Fix p2p-feefilter.py for changed tx relay behavior #8444

Merged

MarcoFalke changed the title ~~[WIP] Travis: Run extended test suite~~ [NO MERGE] Travis: Run extended test suite Feb 23, 2017

MarcoFalke changed the title ~~[NO MERGE] Travis: Run extended test suite~~ [NO MERGE] Travis: Run nightly test suite Feb 23, 2017

MarcoFalke added 2 commits Apr 16, 2016

[travis] verify subtrees, bitcoin/master commit

96284f4

* Add draft to run a test against an old binary (0.12.1)

[qa] Add script to check for datadir compatibility between versions

4118db9

MarcoFalke added 8 commits Nov 14, 2015

[qa] travis: Only run extended rpc tests

ab77daa

WIP: Don't merge. Trigger nightly build without pruning, rpcbind

4c78762

qa travis: --enable-werror

c7627e3

qa: Stop only nodes that were started when one fails to start

f688ddc

Run valgrind

e6c86fe

nightly: Check salvagewallet

fd43e8f

nightly: Only smoke test with valgrind

f2cbc68

nightly: Compat with 0.14.0 only for now

Loading status checks…

9dbb581

laanwj closed this Apr 13, 2017

luke-jr mentioned this pull request May 3, 2017

fix contrib/debian builds; prefer qt5 #10316

Closed

bitcoin / bitcoin

[NO MERGE] Travis: Run nightly test suite #7148

[NO MERGE] Travis: Run nightly test suite #7148

MarcoFalke commented Dec 1, 2015 •

edited

MarcoFalke commented Dec 1, 2015

jonasschnelli commented Dec 2, 2015

MarcoFalke commented Dec 2, 2015

laanwj commented Dec 3, 2015

MarcoFalke commented Dec 9, 2015

MarcoFalke commented Dec 9, 2015

MarcoFalke commented Dec 14, 2015

sdaftuar commented Dec 14, 2015

laanwj commented Jan 7, 2016

sipa commented May 25, 2016

sipa commented May 25, 2016

arowser commented May 27, 2016

MarcoFalke commented Jun 13, 2016

sdaftuar commented Jun 17, 2016

da2ce7 commented Feb 20, 2017

laanwj commented Feb 20, 2017

TheBlueMatt commented Feb 20, 2017

jnewbery commented Mar 23, 2017

TheBlueMatt commented Apr 13, 2017

laanwj commented Apr 13, 2017

jnewbery commented Apr 13, 2017

MarcoFalke commented Apr 13, 2017

bitcoin / bitcoin

Join GitHub today

[NO MERGE] Travis: Run nightly test suite #7148

[NO MERGE] Travis: Run nightly test suite #7148

Conversation

MarcoFalke commented Dec 1, 2015 • edited

MarcoFalke commented Dec 1, 2015

jonasschnelli commented Dec 2, 2015

MarcoFalke commented Dec 2, 2015

laanwj commented Dec 3, 2015

MarcoFalke commented Dec 9, 2015

MarcoFalke commented Dec 9, 2015

MarcoFalke commented Dec 14, 2015

sdaftuar commented Dec 14, 2015

laanwj commented Jan 7, 2016

sipa commented May 25, 2016

sipa commented May 25, 2016

arowser commented May 27, 2016

MarcoFalke commented Jun 13, 2016

sdaftuar commented Jun 17, 2016

da2ce7 commented Feb 20, 2017

laanwj commented Feb 20, 2017

TheBlueMatt commented Feb 20, 2017

jnewbery commented Mar 23, 2017

TheBlueMatt commented Apr 13, 2017

laanwj commented Apr 13, 2017

jnewbery commented Apr 13, 2017

MarcoFalke commented Apr 13, 2017

MarcoFalke commented Dec 1, 2015 •

edited