Merge #1918: [Wallet] Sapling notes, nullifiers and witnesses managem…

…ent.

c01357a SaplingScriptPubKeyMan: guard sapData in IncrementWitness (furszy)
0285a86 add travis job only for the unit tests on x86_64 Linux (furszy)
7963a20 hack to prevent travis no outputs timeouts (Fuzzbawls)
62f58f5 makefile: moved sapling_wallet_tests.cpp to wallet tests section. (furszy)
b8cea53 sapling_rpc_wallet_tests: remove unneeded locks. (furszy)
9e0df68 wallet: Introduce IsAmountCached to know whether a tx amount has been cached or not. (furszy)
5d99ce2 [Wallet] Sapling wallet unit tests back port. (furszy)
8931916 [Wallet] Sapling GetSaplingNoteWitnesses back ported. (furszy)
e537feb [Wallet] IsSaplingSpent back port. (furszy)
3cdf4aa [Wallet] Sapling GetConflicts implemented. (furszy)
c65ced1 UpdateNullifierNoteMapWithTx connected to wallet. (furszy)
262d5e9 CWallet::ChainTip slot back ported. (furszy)
1ddfd4f [Sapling] Update nullifier note map (block & tx) methods back ported. (furszy)
82a71ba Update CWalletTx::GetAmounts() to return COutputEntry for Sapling valueBalance. (furszy)
1056f15 [Wallet] Connect tx GetShieldedValueIn (furszy)
d9e5970 [Sapling] Create IsSaplingNullifierFromMe . (furszy)
79273b1 [Wallet] Back port FindMySaplingNotes(). (furszy)
7b42789 [Wallet] Sapling UpdatedNoteData() implemented (furszy)
2d075d4 [Wallet] MarkAffectedTransactionsDirty() for Sapling. (furszy)
0a655ef Wallet: Keep track of the used nullifiers method. (furszy)
e561075 [Wallet] Write/Read Sapling witness cache size from db. (furszy)
8287a4a [Wallet] Back port IncrementNoteWitnesses & DecrementNoteWitnesses (furszy)
f872e90 Add Sapling notes map to CWalletTx. (furszy)
d82f547 [Sapling] SaplingNoteData class created. (furszy)

Pull request description:

  Another PR decoupled from #1798. Built on top of #1912. PR starting on 69c58a2.

  Focused on introducing sapling notes, nullifiers and witnesses management capabilities to the wallet and the sspkm (essentially wallet support for Sapling primitives). Tested by a wide range of unit tests that can be found in `sapling_wallet_tests.cpp`.

  Further functional tests verifying the correct behavior of the implementation are included down the commits line in 1798: `sapling_wallet.py`, `sapling_wallet_nullifiers.py` (Haven't been able to include them here because those are depending on the shielded spend functionality that is introduced after this work and will have its own separate PR).

  #### List of commits from 1798:

  * Add Sapling notes map to CWalletTx —> 8ae2ea2
  * [Sapling] SaplingNoteData class created. —> 6a77b63
  * Back port IncrementNoteWitnesses & DecrementNoteWitnesses —> 069480a
  * [Wallet] Write/Read Sapling witness cache size from db. —> 9d98dd1
  * Wallet: Keep track of the used nullifiers method. —> 68b4ccd
  * [Wallet] MarkAffectedTransactionsDirty() for Sapling. —> 7045acf
  * [Wallet] Sapling UpdatedNoteData() implemented —> 2c4bab7
  * [Wallet] Back port FindMySaplingNotes(). —> 82714ff
  * [Sapling] Create IsSaplingNullifierFromMe . —> 7d8a575
  * [Wallet] Connect tx GetShieldedValueIn —> ed18897
  * Update CWalletTx::GetAmounts() to return COutputEntry for Sapling valueBalance. —> fdfc73c
  * [Sapling] Update nullifier note map (block & tx) methods back ported. —> 907ae3f
  * CWallet::ChainTip slot back ported —> 108162a
  * UpdateNullifierNoteMapWithTx connected to wallet. —> 46ff034
  * Cleaning compiler warnings —> 14d2d2c
  * [Wallet] Sapling GetConflicts implemented. —> 46f4f92
  * [Wallet] IsSaplingSpent back port. —> 6156b15
  * [Wallet] Sapling GetSaplingNoteWitnesses back ported. —> 12b7b7b
  * [Wallet] Sapling wallet unit tests back port —> 50e1cfc
  * Migration from Purple_Fenix upgrade to v5_dummy —> 6d49e6b (PARTIALLY)
  * Sapling wallet unit tests, set wallet minimum version to FEATURE_SAPLING. —> 5ed289c
  * wallet: Introduce IsAmountCached to know whether a tx amount has been cached or not.  —> db27b54

ACKs for top commit:
  random-zebra:
    utACK c01357a as well.
  Fuzzbawls:
    utACK c01357a

Tree-SHA512: 692ed69425fd6085bf34bbec3ab054fc5a3754d44ac65670489beb0f90f7c112253ec70345f27dc5272f3db30548df748732cc300952fd7f5a3a25ca6d76e8e0

.travis.yml

@@ -139,12 +139,25 @@ jobs:
         HOST=x86_64-unknown-linux-gnu
         PACKAGES="python3-zmq qtbase5-dev qttools5-dev-tools libqt5svg5-dev libqt5charts5-dev libqrencode-dev protobuf-compiler libdbus-1-dev libharfbuzz-dev libprotobuf-dev"
         DEP_OPTS="NO_QT=1 NO_UPNP=1 DEBUG=1 ALLOW_HOST_PACKAGES=1"
+        RUN_UNIT_TESTS=false
         RUN_FUNCTIONAL_TESTS=true
         #TEST_RUNNER_EXTRA="--coverage --extended"  # Run extended tests so that coverage does not fail, but exclude the very slow dbcrash
         GOAL="install"
         BITCOIN_CONFIG="--enable-zmq --with-gui=qt5 --enable-glibc-back-compat --enable-reduce-exports --disable-online-rust --with-params-dir=$PARAMS_DIR"
         BUILD_TIMEOUT=540
 
+    - stage: test
+      name: 'x86_64 Linux  [GOAL: install]  [bionic]  [only unit tests]'
+      env: >-
+        HOST=x86_64-unknown-linux-gnu
+        PACKAGES="python3-zmq qtbase5-dev qttools5-dev-tools libqt5svg5-dev libqt5charts5-dev libqrencode-dev protobuf-compiler libdbus-1-dev libharfbuzz-dev libprotobuf-dev"
+        DEP_OPTS="NO_QT=1 NO_UPNP=1 DEBUG=1 ALLOW_HOST_PACKAGES=1"
+        RUN_UNIT_TESTS=true
+        RUN_FUNCTIONAL_TESTS=false
+        GOAL="install"
+        BITCOIN_CONFIG="--enable-zmq --with-gui=qt5 --enable-glibc-back-compat --enable-reduce-exports --disable-online-rust --with-params-dir=$PARAMS_DIR"
+        BUILT_TIMEOUT=1200
+
     - stage: test
       name: 'x86_64 Linux  [GOAL: install]  [bionic]  [no GUI, no unit tests - only functional tests on legacy pre-HD wallets]'
       env: >-

src/Makefile.test.include

@@ -116,7 +116,8 @@ if ENABLE_WALLET
 BITCOIN_TESTS += \
   test/librust/sapling_rpc_wallet_tests.cpp \
   wallet/test/wallet_tests.cpp \
-  wallet/test/crypto_tests.cpp
+  wallet/test/crypto_tests.cpp \
+  test/librust/sapling_wallet_tests.cpp
 endif
 
 test_test_pivx_SOURCES = $(BITCOIN_TEST_SUITE) $(BITCOIN_TESTS) $(JSON_TEST_FILES) $(RAW_TEST_FILES)
@@ -167,8 +168,10 @@ if EMBEDDED_UNIVALUE
 endif
 
 %.cpp.test: %.cpp
-	@echo Running tests: `cat $< | grep -E "(BOOST_FIXTURE_TEST_SUITE\\(|BOOST_AUTO_TEST_SUITE\\()" | cut -d '(' -f 2 | cut -d ',' -f 1 | cut -d ')' -f 1` from $<
-	$(AM_V_at)$(TEST_BINARY) -l test_suite -t "`cat $< | grep -E "(BOOST_FIXTURE_TEST_SUITE\\(|BOOST_AUTO_TEST_SUITE\\()" | cut -d '(' -f 2 | cut -d ',' -f 1 | cut -d ')' -f 1`" > $<.log 2>&1 || (cat $<.log && false)
+	@echo "" && echo Running tests: `cat $< | grep -E "(BOOST_FIXTURE_TEST_SUITE\\(|BOOST_AUTO_TEST_SUITE\\()" | cut -d '(' -f 2 | cut -d ',' -f 1 | cut -d ')' -f 1` from $<
+	$(AM_V_at)$(TEST_BINARY) -l test_suite -t "`cat $< | grep -E "(BOOST_FIXTURE_TEST_SUITE\\(|BOOST_AUTO_TEST_SUITE\\()" | cut -d '(' -f 2 | cut -d ',' -f 1 | cut -d ')' -f 1`" > $<.log 2>&1 || (cat $<.log && false) & echo "$$!" > "$<.pid"
+	@while kill -0 `cat $<.pid` > /dev/null 2>&1; do echo -n "." > /dev/tty | sleep 0.5; done
+	@echo -e "\r\033[1A\033[0K" | rm $<.pid
 
 %.json.h: %.json
 	@$(MKDIR_P) $(@D)