Migrate Jenkins tests to 0.10.x Bitcoin Core / Omni Core

[msgilligan]

This involves at least the following tasks:

1. Make sure dependencies for building Bitcoin Core and Omni Core 0.10.x versions are pre-installed or refreshed automatically (see @dexX7 's comment below)
2. Update bitcoin-core Jenkins job to stable 0.10.x branch
3. Temporarily disable omnicore-stable Jenkins job (previously used 0.0.8 branch, should now be 0.0.9 but uses same blockchain datadir as omnicore-stable and we can't share a blockchain datadir between 0.0.9 and 0.10.0)
4. Update omnicore-integration Jenkins job to OmniLayer/omnicore repository and the correct integration 0.10.0 branch)

[msgilligan]

If the above list is correct and complete, it should take less than an hour to make the changes and maybe an hour or so to run the tests and verify that they work correctly.

Update: Item (1) above makes things more complicated and invalidates the "couple of hours" estimate.

[dexX7]

Update bitcoin-core Jenkins job to stable 0.10.x branch

I was wondering for some time: what's the point of this? As far as I can see, this job isn't used at all.

previously used 0.0.8 branch, should now be 0.0.9

You should clear the state data, ideally by manually removing all MP* dirs and mastercore* files, or starting once with the option -startclean.

we can't share a blockchain

This is unfortunally correct. Which version is going to be used for mainnet consensus tests? Both?

Update

How are the build dependencies handled? There are two ways:

• install dependencies as described in the build notes
• build the dependencies from scratch, via the "new" Bitcoin Core building and caching system

The later is probably the preferred route, as those dependencies are updated from time to time, but I'm not 100 % sure, how it works. But that said, I've managed to use it via Circle CI and the configuration file could serve as blueprint: circle.yml

I assume we probably want to build with QT (=> no NO_QT=1 should be passed via the dependency options and --without-gui should not be used for the configuration).

As long as depends/built isn't removed (or cached otherwise), the build time should be very close to the alternative.

[msgilligan]

I was wondering for some time: what's the point of [the bitcoin-core job]? As far as I can see, this job isn't used at all.

This job can be manually run to pull Bitcoin Core from upstream bitcoin/bitcoin and build it. It will then manually run the btc-integ-regtest job. The btc-integ-regtest job runs bitcoin RPC integration tests against the bitcoind built by the bitcoin-core job.

This allows us to:

1. Verify that our BitcoinClient RPC client works against standard Bitcoin Core.
2. Develop RPC unit tests that test Bitcoin Core.
3. Verify that our test infrastructure works with standard Bitcoin Core builds

You should clear the state data, ideally by manually removing all MP* dirs and mastercore* files, or starting once with the option -startclean.

The test scripts currently do this with a rm -rf command. I think -startclean is new in 0.10.x, right?

Which version is going to be used for mainnet consensus tests? Both?

It will have to be 0.10.x. (There is only room for one blockchain on the current Jenkins server) There should be no differences in main net consensus even with DEX Phase 2 code, because it won't activate till a specific block, right?

We will lose the ability to use Jenkins to test consensus between various deployments of Master Core 0.0.9. I don't think that's too big of an issue now that 0.0.9 is stable and so are the Omniwallet and OmniChest webs that use it. If we need to run consensus tests between various 0.0.9 implementations we can use the command-line tools.

How are the build dependencies handled?

I hadn't thought about that. That complicates things a little. I'll update the task list in the initial comment to reflect that.

[dexX7]

The test scripts currently do this with a rm -rf command. I think -startclean is new in 0.10.x, right?

-startclean can be used to reset the Omni Core state, however, it only removes MP* dirs. For regtest, we probably still want to remove the whole regtest dir, to reset everything, i.e. log files and Bitcoin Core state (blocks, ...).

It's a different story for mainnet tests though: a full scan/parsing of transactions takes about an hour, so this should only be done very rarely, and we also shouldn't clear the blockchain data.

I mentioned it mostly in the context of the migration from 0.0.8 to 0.0.9.

We will lose the ability to use Jenkins to test consensus between various deployments of Master Core 0.0.9.

Is it still possible to test (local) Omni Core 0.0.10 against (remote) 0.0.9, e.g. via OmniChest or OmniWallet?

That complicates things a little.

Actually not that much.

The external packages can be installed once, and the depends dir created as well:

sudo apt-get update
sudo apt-get install --no-install-recommends --no-upgrade "bc"
mkdir -p $WORKSPACE/depends

And in a nutshell, the current script:

./autogen.sh
./configure
make
./src/test/test_bitcoin --log_format=XML --log_sink=test_bitcoin.xml --log_level=test_suite --report_level=no

... could very likely be replaced by:

make -C depends HOST=x86_64-unknown-linux-gnu
./autogen.sh
./configure --prefix=$WORKSPACE/depends/x86_64-unknown-linux-gnu --cache-file=config.cache
make
./src/test/test_bitcoin --log_format=XML --log_sink=test_bitcoin.xml --log_level=test_suite --report_level=no

Further, the artifacts to archive:

src/mastercored,src/mastercore-cli

... should be replaced by:

src/bitcoind,src/bitcoin-cli

... and maybe src/qt/bitcoin-qt, src/test/test_bitcoin, src/qt/test/test_bitcoin-qt, if you like to archive those as well.

I'm not sure, what the best way is to handle the different file names. They could be renamed by Jenkins to match those of 0.0.9, or new scripts with updated file names could created. It won't be bitcoin* forever, but also not mastercore in the future.

[msgilligan]

Is it still possible to test (local) Omni Core 0.0.10 against (remote) 0.0.9, e.g. via OmniChest or OmniWallet?

Yes -- as long as there are no consensus-altering changes in 0.0.10.

[msgilligan]

@dexX7, If I install the external packages will that break the build on the 0.0.9 branch?

[dexX7]

@msgilligan: If I install the external packages will that break the build on the 0.0.9 branch?

make -C depends HOST=x86_64-unknown-linux-gnu

... builds the dependencies, but doesn't install them, and they should be available in $WORKSPACE/depends.

./configure --prefix=$WORKSPACE/depends/x86_64-unknown-linux-gnu --cache-file=config.cache

... then instructs Bitcoin/Omni Core to use them.

Was this what you were referring to by "external packages"?

[msgilligan]

OK, https://ci.omni.foundation has been rebuilt from scratch.

The new server has the following improvements:

1. Latest Jenkins
2. Java 8 (currently used to run Jenkins and for Java builds/tests)
3. NGINX (instead of Apache)
4. SSL (self-signed)
5. All Bitcoin and Omni Core integration tests are using 0.10.x and 0.0.10 versions and the 0.10.x blockchain.
6. msc-* jobs are now named omni-*.
7. There is no omnicore-stable job because omnicore-stable and omnicore-integration must share the same local blockchain directory.

[dexX7]

Using a self-signed certificate is a nice idea (even though Chrome complains about outdated crypto).

Two quick notes:

• mastercore.log was renamed to omnicore.log, and file name or location can be changed via --omnilogfile=path/to/the.log
• the last mainnet consensus tests failed

[msgilligan]

I wonder if there's an easy way to create a getter self-signed certificate that eliminates the Chrome warning.

I guess the BASH scripts that start the tests should be updated for the names/locations of the log files. @dexX7 can you make a pull request with the appropriate changes?

The mainnet consensus tests were failing because blockchain.info was down. It was running for almost 30 hours because I let the test download the blockchain starting from block 0. I'm currently running a blockchain sync from the command-line (since I didn't want to wait for blockchain.info to come back up.) It's at block 350000. So hopefully tomorrow morning, it will be synced and I can start a mainnet consensus test.