[POC] Introducing property based testing to Core #8469

Christewart · Aug 6, 2016

This pull request is a proof of concept for introducting property based testing into Bitcoin Core

In QuickCheck the programmer writes assertions about logical properties that a function should fulfill. Then QuickCheck attempts to generate a test case that falsifies these assertions. Once such a test case is found, QuickCheck tries to reduce it to a minimal failing subset by removing or simplifying input data that are not needed to make the test fail.

This has been very useful for a Bitcoin library I've been working on and thought it would be worthwhile to develop a POC for Bitcoin Core. The property based library I am using for C++ is called rapidcheck. Here are the docs.

This pull request currently contains two properties, one testing CKey generation and the other testing serialization symmetry for CKey and CBitcoinSecret. These are rather trivial properties, but useful for illustrating the power of property based testing if there was a bug inside of Core.

I want to solicit some feedback from developers if this is something that would actually be merged into Core. Eventually we could have a large library of generators that would allow us to quickly prototype, test, and reason about the behavior of new code added to Core. Here is an example of a library of generators (in Scala) that could give you a little more of an idea of what I am talking about.

Thoughts?

You should be able to use something like $(package)-$($(package)_version) here.

No need for the empty lines.

Use = instead of : here, as well as the lines below.

nit: space introduced here.

fanquake · Aug 8, 2016

This needs a rebase/conflicts fixed.

nit: Just 2016

Does this need $(package)_build_subdir=build ?

Not sure what you mean by this, do you mean we don't need lines 11 - 21?

MarcoFalke · Aug 8, 2016

I think this approach of testing is interesting and may render useful for bitcoin. The user guide https://github.com/emil-e/rapidcheck/blob/master/doc/user_guide.md mentions that the API is not finalized, so it may be better to integrate it via a subtree (maybe in /test)?

jonasschnelli · Aug 8, 2016

I think this is useful. I'm not very familiar with property based testing, but this seems to be something that would make sense for consensus and security critical projects.

Some thoughts:

the rapidcheck dependency should probably only be required if one passes --enable-tests.
I think you can remove the changes in src/test/script_tests.cpp and its header.

If others also agree to take this into master, we should probably have a first logical slice of property based tests (maybe serialization).

Needs rebase.

MarcoFalke · Aug 18, 2016

@theuni Can you help here to decide if using depends is fine or a subtree would be cleaner? (I prefer a subtree per my above comment, but I am not familiar with the build system. Input is appreciated)

Christewart · Aug 18, 2016

I would probably need help with the build code to get it production ready. I plan to keep on adding properties to this pull request though when time permits. I'm going to try and get around implementing fanquake's comments and figure out how to do the --enable-test as per jonasschnelli's comment this weekend

Again, I think a subtree is the better choice per my previous comment. Also, I don't think we can have someone verify this binary each time it is bumped, even if it was done deterministically.

Not sure what you mean about the deterministic bump. This is just a source tarball. Verifying would be no different than any of the other depends.

The general rule of thumb, though, is whether builders will be able to use it without any extra work. Since rapidcheck isn't present in any distros (as far as I can see), the answer to that is "no".

So, unless we're willing to require a depends build to run the new tests, a subtree would be easiest. However.

Tests are a bit of a special case. It may be reasonable to keep this in depends for now and let the c-i run it until there's sufficient coverage.

There's also the fact that it uses CMake, which we really don't want to introduce into our build (nothing against CMake, we just don't want to require every build tool under the sun).

Ok, fine with me. Thanks for letting us know.

Can you try to adjust the folder name within the targz to have the version appened?

This may fix the travis issue.

I should note that I did NOT get any of this deterministic build working on my machine, I was trying to copy what I have seen on other other files to no avail -- I just ended up adding rapidcheck to my includes directory on my local machine. I'm fairly new to C++ & it's build systems so I was trying to just hack the pieces together. So I wouldn't be surprised if there are deeper issues than a naming issue although I will try and look at that later today. What exact name are you talking about? Are you talking about line 7? Line 5?

No worries, I don't understand cpp or the build system either. I can take a look later today... I think the issue is within the package you uploaded.

@Christewart Please try to fetch MarcoFalke/bitcoin@befb827 (Mf1608-rapidcheckRework) and push it here. This should fix your compile issues for now.

Please note that travis will likely reject this due to a bug in gcc4.8.x: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55914

Why is this hash different than the one I provided?

The file size is different, so ~~must~~ should the hash.

The targz you provided also included the 1.8 MB .git subfolder, which is not required.

The hash I provided should be the hash of the tarball created by GitHub: https://github.com/emil-e/rapidcheck/archive/f5d3afa4f387ecf147faf98d96710a6edfa420f1.tar.gz

Ok that makes sense. Didn't realize you removed files. Do you have any idea why this is still failing?

MarcoFalke · Aug 29, 2016

@Christewart Do you need help to address the feedback from @jonasschnelli?

Christewart · Aug 29, 2016

@MarcoFalke Yeah I'm not really sure how to get that done

sipa · Jan 8, 2017

sipa reviewed Jan 8, 2017

View changes

The intended idiom is just

CTransaction tx2(deserialize, ss);

gmaxwell · Jan 8, 2017

I like this kind of testing-- testing invariants on random test cases--, and we use it extensively in libsecp256k1. It's one of the answers to my irritation with the way many of the unit tests in Bitcoin core have worked in the past: hard-coding the exact behavior. Meaning that if you change anything, the tests all fail and you're left wondering if you broke something important or if the test was just mandating the behavior you were fixing. In either case, updating the test is often more work than the change.

Taking a dependency for it seems unfortunate; but I'm not qualified to judge if it does snazzy things that make generating cases easy. I have had bad experience with test frameworks in the past which had very high overhead making test cases that should have taken seconds take minutes, resulting in less testing. I see in this one that it uses a excessively slow PRNG based on skein, which makes me a bit dubious-- but if thats the only problem it would be easy to fix if we forked it (by replacing it with xorshift128+ or likewise).

Christewart · Jan 8, 2017

It should also be noted that they way the test are currently written they tie in the boost testing framework, if we add rapidcheck as a dependency and decide to remove boost, which #8670 suggests, we will have to rewrite these test cases to be independent of boost.

In terms of speed, you would know more than I would about choosing PRNG. Rapidcheck currently defaults to running 100 test cases against every property you have specified. You can configure this property to be higher or lower depending on the situation. I think it would make sense to configure this to lower value for local development, and a higher value for travis builds to try and exhaustively test our invariants. You can read more about configuration of rapidcheck here.

@jonasschnelli 's comment above was interesting I thought, I'm not super familiar with C++ build systems but only including the rapidcheck dependency if the flag --enable-test flag was given seems to makes sense.

I think the long term value add here is having a library of generators (for various protocol data structures) that we can use to test new protocol changes. I'm going to be adding some properties about creating tx types in the coming weeks. To get an idea of what I'm looking to add, you can look at some of the generators I have created in bitcoin-s.

TheBlueMatt · Jul 11, 2017

Hmm, looks like travis is failing to build rapidcheck here? Are you interested in rebasing and potentially fixing it?

Christewart · Jul 12, 2017

It appears there is a bug with gcc 4.8 that causes a problems with lambdas and parameter packs:

https://stackoverflow.com/questions/45062090/gcc-4-8-4-error-parameter-packs-not-expanded-with?noredirect=1#comment77097698_45062090

https://stackoverflow.com/questions/22568642/parameter-packs-not-expanded-with-another-variadic-template-bug-with-gc

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=41933

fanquake commented on an outdated diff Aug 8, 2016

View full outdated diff
depends/packages/rapidcheck.mk

@@ -0,0 +1,21 @@

+package=rapidcheck

+

fanquake Aug 8, 2016
Contributor

No need for the empty lines.

fanquake commented on an outdated diff Aug 8, 2016

View full outdated diff
depends/packages/rapidcheck.mk

@@ -0,0 +1,21 @@

+package=rapidcheck

+

+$(package)_version:1.0

fanquake Aug 8, 2016 • edited
Contributor

Use = instead of : here, as well as the lines below.

fanquake commented on an outdated diff Aug 8, 2016

View full outdated diff
src/test/key_properties.cpp

@@ -0,0 +1,48 @@

+// Copyright (c) 2012-2015 The Bitcoin Core developers

fanquake Aug 8, 2016
Contributor

nit: Just 2016

jonasschnelli added Feature Tests and removed Feature labels Aug 8, 2016

Christewart referenced this pull request Sep 7, 2016
Open
Replacing Boost Test Framework #8670

Christewart referenced this pull request Mar 12, 2017
Merged
Fix mem access violation merkleblock #9980

@@ -508,7 +508,7 @@ if test x$TARGET_OS = xdarwin; then
        AX_CHECK_LINK_FLAG([[-Wl,-dead_strip]], [LDFLAGS="$LDFLAGS -Wl,-dead_strip"])
      fi
-    -AC_CHECK_HEADERS([endian.h sys/endian.h byteswap.h stdio.h stdlib.h unistd.h strings.h sys/types.h sys/stat.h sys/select.h sys/prctl.h])
+    +AC_CHECK_HEADERS([endian.h sys/endian.h byteswap.h stdio.h stdlib.h unistd.h strings.h sys/types.h sys/stat.h sys/select.h sys/prctl.h ])

@@ -0,0 +1,21 @@
+    +package=rapidcheck
+    +
+    +$(package)_version=1.0
+    +
+    +$(package)_download_path=https://github.com/Christewart/rapidcheck/releases/download/1.0
+    +
+    +$(package)_file_name=$(package)-$($(package)_version).tar.gz
+    +
+    +$(package)_sha256_hash=c228dc21ec24618bfb6afa31d622d1f4ea71168f04ee499e1ffcfc63cd5833f4

      $(package)_file_name=$(package)-$($(package)_version).tar.gz
-    -
-    -$(package)_sha256_hash=c228dc21ec24618bfb6afa31d622d1f4ea71168f04ee499e1ffcfc63cd5833f4
-    -
-    -define $(package)_preprocess_cmds
-    -  mkdir build
-    -endef
+    +$(package)_sha256_hash=78cdb8d0185b602e32e66f4e5d1a6ceec1f801dd9641b8a9456c386b1eaaf0e5

+    +  CDataStream ss(SER_NETWORK, PROTOCOL_VERSION);
+    +  ss << tx;
+    +  deserialize_type t;
+    +  CTransaction tx2(t,ss);

Christewart	`Adding rapidcheck dependency, adding CKey properties` Adding rapidcheck dependency, adding CKey properties Successfully compiling bitcoin with rapidcheck dependency Adding new property file for CKey Serialization symmetry for CKey -> CBitcoinSecret -> CKey Adding generators for CPrivKey, CPubKey, and uint256 [depends] Rework rapidcheck.mk	`67a9b8a`
Christewart	`Adding Bloom filter properties and bloom filter generator`	`e3f2f47`
Christewart	`Adding serialization symmetry for CBloomFilter`	`2737955`
Christewart	`Adding block header serialization property` Fixing generator bug where I wasn't setting all fields on a BlockHeader	`179337a`
Christewart	`Creating transaction_gen.h, adding generator for COutPoint and serial…` …ization symmetry property for COutPoint	`0aa0d0a`
Christewart	`Adding script generator and script_propertes, first property is CScri…` …pt serialization symmetry	`92a9724`
Christewart	`Adding CTransaction generator, adding serialization symmetry property…` … for CTransaction Removing 'oneOrMoreInputs' and 'oneOrMoreOutputs' generators in favor of using rapidcheck's gen::nonEmpty function	`fdc0cf1`
Christewart	`Adding CTransactionRef and CBlock generators, adding property for CBl…` …ock serialization symmetry	`c767e6b`
Christewart	`Adding Generators outside of the rc name space, adding generator for …` …LoadedBloomFilter	`867d8de`
Christewart	`Adding merkleblock_gen.h, merkleblock serialization symmetry test`	`607725b`
Christewart	`Adding merkle_block properties` Committing to try and debug mem leak Removing comments, rebasing to master Only adding rapidcheck dependency if tests are enabled Refactoring inclues in generator files, reordering files in Makefile.test.include Undoing change in merkleblock.cpp	`31a1ca5`

bitcoin/bitcoin

[POC] Introducing property based testing to Core #8469

Christewart commented Aug 6, 2016

fanquake commented on an outdated diff Aug 8, 2016

fanquake commented on an outdated diff Aug 8, 2016

fanquake commented on an outdated diff Aug 8, 2016

fanquake commented on an outdated diff Aug 8, 2016

fanquake commented Aug 8, 2016

fanquake commented on an outdated diff Aug 8, 2016

fanquake and 1 other commented on an outdated diff Aug 8, 2016

MarcoFalke commented Aug 8, 2016

jonasschnelli commented Aug 8, 2016

jonasschnelli added Feature Tests and removed Feature labels Aug 8, 2016

MarcoFalke commented Aug 18, 2016

Christewart commented Aug 18, 2016 • edited

MarcoFalke and 1 other commented on an outdated diff Aug 19, 2016

MarcoFalke and 1 other commented on an outdated diff Aug 24, 2016

Christewart and 1 other commented on an outdated diff Aug 26, 2016

MarcoFalke commented Aug 29, 2016

Christewart commented Aug 29, 2016

Christewart referenced this pull request Sep 7, 2016

Replacing Boost Test Framework #8670

sipa reviewed Jan 8, 2017

View changes

gmaxwell commented Jan 8, 2017

Christewart commented Jan 8, 2017

Christewart referenced this pull request Mar 12, 2017

Fix mem access violation merkleblock #9980

Christewart added some commits Jun 19, 2016

TheBlueMatt commented Jul 11, 2017

Christewart commented Jul 12, 2017 • edited

sipa added a commit that referenced this pull request Jul 17, 2017

bitcoin/bitcoin

Join GitHub today

[POC] Introducing property based testing to Core #8469

Conversation

Christewart commented Aug 6, 2016

Show 1 comment Hide comment fanquake commented on an outdated diff Aug 8, 2016

Show 1 comment Hide comment fanquake commented on an outdated diff Aug 8, 2016

Show 1 comment Hide comment fanquake commented on an outdated diff Aug 8, 2016

fanquake Aug 8, 2016 • edited

Show 1 comment Hide comment fanquake commented on an outdated diff Aug 8, 2016

fanquake commented Aug 8, 2016

Show 1 comment Hide comment fanquake commented on an outdated diff Aug 8, 2016

Show 2 comments Hide comments fanquake and 1 other commented on an outdated diff Aug 8, 2016

MarcoFalke commented Aug 8, 2016

jonasschnelli commented Aug 8, 2016

jonasschnelli added Feature Tests and removed Feature labels Aug 8, 2016

MarcoFalke commented Aug 18, 2016

Christewart commented Aug 18, 2016 • edited

Show 3 comments Hide comments MarcoFalke and 1 other commented on an outdated diff Aug 19, 2016

Show 4 comments Hide comments MarcoFalke and 1 other commented on an outdated diff Aug 24, 2016

Show 3 comments Hide comments Christewart and 1 other commented on an outdated diff Aug 26, 2016

MarcoFalke Aug 26, 2016 • edited

MarcoFalke commented Aug 29, 2016

Christewart commented Aug 29, 2016

Christewart referenced this pull request Sep 7, 2016

sipa reviewed Jan 8, 2017 View changes

gmaxwell commented Jan 8, 2017

Christewart commented Jan 8, 2017

Christewart referenced this pull request Mar 12, 2017

Christewart added some commits Jun 19, 2016

TheBlueMatt commented Jul 11, 2017

Christewart commented Jul 12, 2017 • edited

sipa added a commit that referenced this pull request Jul 17, 2017

fanquake commented on an outdated diff Aug 8, 2016

fanquake commented on an outdated diff Aug 8, 2016

fanquake commented on an outdated diff Aug 8, 2016

fanquake commented on an outdated diff Aug 8, 2016

fanquake commented on an outdated diff Aug 8, 2016

fanquake and 1 other commented on an outdated diff Aug 8, 2016

MarcoFalke and 1 other commented on an outdated diff Aug 19, 2016

MarcoFalke and 1 other commented on an outdated diff Aug 24, 2016

Christewart and 1 other commented on an outdated diff Aug 26, 2016

sipa reviewed Jan 8, 2017

View changes