Cypress travis (#997)

* executing test-cypress on travis * kick * optional activation * Chore: travis move to focal and running cypress-tests, cache improvements * rollback core-change * tidy up Co-authored-by: Kim Neunert <k9ert@gmx.de>
cryptoadvance · Mar 4, 2021 · 5592654 · 5592654
1 parent cb01072
commit 5592654
Show file tree

Hide file tree

Showing 4 changed files with 88 additions and 51 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -2,49 +2,72 @@ language: python
 python:
   - "3.8"
 os: linux
-dist: xenial
+dist: focal
 cache:
-    pip: true
-    ccache: true
-    directories:
-        - tests/bitcoin
+  pip: true
+  ccache: true
+  directories:
+    - tests/bitcoin
+  # cypress
+    - ~/.npm
+    - ./node_modules
+    - ~/.cache
+  override:
+    - npm ci
+    - npm run cy:verify
 
 addons:
-    apt:
-        sources:
-            - sourceline: 'ppa:bitcoin/bitcoin'
-        packages:
-            - libdb4.8-dev
-            - libdb4.8++-dev
-            - build-essential
-            - curl
-            - git
-            - libsdl2-dev
-            - libsdl2-image-dev
-            - gcc-arm-none-eabi
-            - libnewlib-arm-none-eabi
-            - libudev-dev
-            - libtool
-            - autotools-dev
-            - automake
-            - pkg-config
-            - bsdmainutils
-            - libssl-dev
-            - libevent-dev
-            - libboost-system-dev
-            - libboost-filesystem-dev
-            - libboost-chrono-dev
-            - libboost-test-dev
-            - libboost-thread-dev
-            - libusb-1.0-0-dev
-            - protobuf-compiler
-            - cython3
-            - ccache
+  apt:
+    packages:
+      - build-essential
+      - curl
+      - git
+      - libsdl2-dev
+      - libsdl2-image-dev
+      - gcc-arm-none-eabi
+      - libnewlib-arm-none-eabi
+      - libudev-dev
+      - libtool
+      - autotools-dev
+      - automake
+      - pkg-config
+      - bsdmainutils
+      - libssl-dev
+      - libevent-dev
+      - libboost-system-dev
+      - libboost-filesystem-dev
+      - libboost-chrono-dev
+      - libboost-test-dev
+      - libboost-thread-dev
+      - libusb-1.0-0-dev
+      - protobuf-compiler
+      - cython3
+      - ccache
+# From https://dev-notes.eu/2020/10/Build-Bitcoin-Core-in-Ubuntu/
+      - automake
+      - cmake
+      - curl
+      - g++-multilib 
+      - binutils-gold
+      - patch
+# For cypress
+      - nodejs
+
 
 before_install:
+  - npm ci
   - docker pull registry.gitlab.com/cryptoadvance/specter-desktop/python-bitcoind:latest
   - set -o errexit; source ./tests/install_bitcoind.sh
+install:
+  - npm ci
 script: 
   - pip install -e .
   - pip install -r test_requirements.txt
-  - pytest --cov=cryptoadvance 
+  - pytest --cov=cryptoadvance  
+  - ./utils/test-cypress.sh --debug run 
+before_cache:
+  - rm -f $HOME/.cache/pip/log/debug.log
+  - rm -f ./tests/bitcoin/config.log
+  - rm -f ./tests/bitcoin/contrib/db4/db-4.8.30.NC/build_unix/config.log
+  - rm -f ./tests/bitcoin/src/secp256k1/config.log
+  - rm -f ./tests/bitcoin/src/univalue/config.log
diff --git a/docs/continuous-integration.md b/docs/continuous-integration.md
@@ -35,7 +35,7 @@ start_bitcoind-function:
 
 # Travis-CI
 
-Travis-CI setup is very straightforward. As we're using the build-cache, the bitcoind sources and build is cached. Therefore such a build would only take 2 minutes. If the master-branch has new commits, bitcoind gets automatically rebuilt and the tests are running against the new version (tests/install_bitcoind.sh).py
+Travis-CI setup is very straightforward. As we're using the build-cache, the bitcoind sources, npm-setup (for cypress) and build is cached. Therefore such a build would only take 10 minutes. If the master-branch has new commits, bitcoind gets automatically rebuilt and the tests are running against the new version (tests/install_bitcoind.sh).py
 
 # Releasing
 
@@ -45,16 +45,14 @@ We're mostly releasing automatically. Currently the following artifacts are rele
 * specterd (daemon) is a binary for kicking off the specter-desktop service on the command-line. We have binaries for windows, Linux and macOS
 * We have an Electron-App which we're also releasing for Windows, Linux and MacOS. Unfortunately the macOS build is not yet automated
 * We release a pip-package
-* We release docker-images, these are also not yet automated
+* Usually some time after the release, the lncm is releasing [docker-images](https://hub.docker.com/r/lncm/specter-desktop). Very much appreciated, even though we can't guarantee for them, obviously.
 
 # How we release
 As we have a strict build-only-on-private-hardware build-policy, we're using GitLab private runners in order to build our releases. In order to test and develop the releasing automation, people can setup GitLab-projects which are syncing from their GitHub-forks. With such a setup it's possible to create test-releases and therefore test the whole procedure end-to-end.
 
 The automation of that kicks in if someone creates a tag which is named like "vX.Y.Z". This is specified in the gitlab-ci.yml. The release-job will only be triggered in cases of tags. One step will also check that the tag follows the convention above.
 The package upload will need a token. How to obtain the token is described in the packaging-tutorial. It's injected via GitLab-variables. ToDo: put the token on a trusted build-node.
 
-The alternative would have been to use travis-ci for releasing. In that case we would encrypt the token with a private-key from Travis and commit to the repository. This looks more safe to me then the above scenario but less safe then the todo, where we're storing the token on the build-node.
-
 ## pyinstaller system-dependent binaries
 The [pyinstaller directory](../pyinstaller) contains scripts to create the platform-specific binaries (plus electron) to use specter-desktop as a desktop-software. Some of them are created and uploaded to [GitHub-releases](https://github.com/cryptoadvance/specter-desktop/releases) via more or less special build-agents.
 The [windows-build-agent](https://docs.gitlab.com/runner/install/windows.html) needs manual installation 

diff --git a/tests/install_bitcoind.sh b/tests/install_bitcoind.sh
@@ -1,3 +1,5 @@
+#!/bin/sh
+
 echo "    --> install_bitcoind.sh Start $(date)"
 START=$(date +%s.%N)
 # Clone bitcoind if it doesn't exist, or update it if it does
@@ -7,17 +9,14 @@ bitcoind_setup_needed=false
 if [ ! -d "./bitcoin/.git" ]; then
     echo "    --> cloning bitcoin"
     git clone https://github.com/bitcoin/bitcoin.git
-    cd bitcoin
     bitcoind_setup_needed=true
-else
-    cd bitcoin
-    echo "    --> fetching bitcoin"
-    git fetch
 fi
 
+cd bitcoin
+
 # Determine if we need to pull. From https://stackoverflow.com/a/3278427
-UPSTREAM=${1:-'@{u}'}
-LOCAL=$(git rev-parse @)
+UPSTREAM=origin/master
+LOCAL=$(git describe --tags)
 if cat ../../pytest.ini | grep "addopts = --bitcoind-version" ; then
     PINNED=$(cat ../../pytest.ini | grep "addopts = --bitcoind-version" | cut -d' ' -f4)
 fi
@@ -27,27 +26,39 @@ if [ -z $PINNED ]; then
     if [ $LOCAL = $REMOTE ]; then
         echo "Up-to-date"
     elif [ $LOCAL = $BASE ]; then
-        git reset --hard origin/master
         git pull
+        git reset --hard origin/master
         bitcoind_setup_needed=true
     fi
 else
     if [ $LOCAL = $PINNED ]; then
         echo "    --> Pinned: $PINNED! Checkout not needed!"
     else
         echo "    --> Pinned: $PINNED! Checkout needed!"
+        git fetch
         git checkout $PINNED || exit 1
         bitcoind_setup_needed=true
     fi
 fi
 
 
-# Build bitcoind. This is super slow, but it is cached so it runs fairly quickly.
+
+# Build bitcoind. 
 if [ "$bitcoind_setup_needed" = "true" ] ; then
+    # Build dependencies. This is super slow, but it is cached so it runs fairly quickly.
+    cd contrib
+    # This is hopefully fullfilles (via .travis.yml most relevantly)
+    # sudo apt install make automake cmake curl g++-multilib libtool binutils-gold bsdmainutils pkg-config python3 patch
+    echo "    --> Building db4"
+    ./install_db4.sh $(pwd)
+    echo "    --> Finishing db4"
+    ls -l
+    cd ..
     echo "    --> Setup needed. Starting autogen"
     ./autogen.sh
     echo "    --> Starting configure"
-    ./configure --with-miniupnpc=no --without-gui --disable-zmq --disable-tests --disable-bench --with-libs=no --with-utils=no
+    export BDB_PREFIX="$(pwd)/contrib/db4"
+    ./configure BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include" --with-miniupnpc=no --without-gui --disable-zmq --disable-tests --disable-bench --with-libs=no --with-utils=no
 fi
 make -j$(nproc) src/bitcoind
 cd ../.. #travis is sourcing this script

diff --git a/utils/test-cypress.sh b/utils/test-cypress.sh
@@ -11,7 +11,11 @@ export SPECTER_DATA_FOLDER=~/.specter-cypress
 # We'll might change that on the "dev-function"
 export SPECTER_CONFIG=CypressTestConfig
 
-. ./.env/bin/activate
+if [ -z "$VIRTUAL_ENV" ]; then
+  # activate virtualenv. This is e.g. not needed in CI
+  source ./.env/bin/activate
+fi
+
 function check_consistency {
   if ! npm version 2> /dev/null 1>&2 ; then
     echo "npm is not on the PATH. Please install node and bring on the PATH"
@@ -22,6 +26,7 @@ function check_consistency {
       sleep 5
       ps | grep python && (echo "please investigate or kill " && exit 1)
   fi
+  $(npm bin)/cypress verify
 }
 check_consistency