V21.3 and synchronization improvements (#3140)

* Disable peers banning

* Increase max LMDB size (#3120)

from 128GB to 256GB

* This is a series of patches to reduce bandwidth consumption from a large amount of frontiers.

* Broadcasting blocks only when an election is started rather than immediately upon receipt.

* more changes regarding github actions security hardening (#3088)

https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions

Co-authored-by: Russel <russel@nano.org>

* Remove recently dropped check to restart elections

Up until "recently", a new write transaction was opened to update work on the ledger for every single block. Now with deferred work updates on the block processor, it is no more expensive than processing a new block, so it makes sense to remove this constraint of having been recently dropped. This improves quality of service.

The previously implicit check for confirmed dependents (since the election was dropped) is now explicit. The work on the ledger is updated regardless of that check passing.

The election is not immediately inserted as *active* anymore, same behavior as the normal election insertion path.

Note that if an election is active, the work is **not** updated on the ledger. That behavior also seems desirable. This could be achieved by updating the store after the block is identified as old, directly within `ledger::process`. For post-processing, a flag can be passed to `blockprocessor::process_old`, at which point `active_transactions::restart` can be scrapped since it becomes a simple election insertion + stats update (with a dependents confirmed check). Since this change would touch ledger code I am leaving for others to do it. There's also the question if the confirmed status should be checked within the ledger processing code.

Note: only tested via core_test.

# Conflicts:
#	nano/core_test/active_transactions.cpp
#	nano/core_test/node.cpp
#	nano/lib/locks.cpp
#	nano/lib/locks.hpp
#	nano/node/active_transactions.cpp
#	nano/node/active_transactions.hpp

* Show frontiers age in RPC

* This decreases the confirm_req_batches_max from 20 to 2, equivalent to 280/sec -> 28/sec. This option is also made configurable.

Confirm_req batches are used in the bootstrap process and excessive requests to reps can degrade performance and often get dropped.

# Conflicts:
#	nano/node/active_transactions.cpp
#	nano/node/confirmation_solicitor.cpp
#	nano/node/nodeconfig.cpp
#	nano/node/nodeconfig.hpp

* This removes bootstrap restarting code as it is ineffective and can cause unnecessary restarting of the bootstrap process.

* Bumping version number.

* Including missing header.

* Optionally send confirmed frontiers with an optional flag.

* Throttled removal of rep crawler targets (#3154)

* Throttled removal of rep crawler targets

* Remove recursion

Co-authored-by: Russel Waters <russel@nano.org>
Co-authored-by: Sergey Kroshnin <sergiysw@gmail.com>

.github/workflows/analyzers.yml

@@ -6,10 +6,26 @@ jobs:
   clang_format:
     runs-on: ubuntu-18.04
     steps:
-      - uses: actions/checkout@50fbc62
+      - uses: actions/checkout@50fbc622fc4ef5163becd7fab6573eac35f8462e
       - name: Get clang-format 8
         env: 
           DEBIAN_FRONTEND: noninteractive
         run: sudo update-alternatives --install /usr/bin/clang-format clang-format /usr/bin/clang-format-8 1000
       - name: Clang Format
-        run: ci/check-commit-format.sh 
+        run: ci/check-commit-format.sh
+
+  cmake_format:
+    runs-on: ubuntu-18.04
+    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
+    steps:
+      - uses: actions/checkout@50fbc622fc4ef5163becd7fab6573eac35f8462e
+      - uses: actions/setup-python@41b7212b1668f5de9d65e9c82aa777e6bbedb3a8
+        with: 
+          python-version: '3.x'
+          architecture: 'x64'
+      - uses: BSFishy/pip-action@8f2d471d809dc20b6ada98c91910b6ae6243f318
+        with: 
+          packages: |
+            cmake-format
+      - name: Check cmake-format
+        run: bash ci/check-cmake-format.sh

.github/workflows/beta_artifacts.yml

@@ -18,9 +18,13 @@ jobs:
       - uses: actions/checkout@722adc6
       - uses: chrislennon/action-aws-cli@f0f8671
       - name: tag
-        run: echo "::set-env name=TAG::`git describe --tags $GITHUB_SHA`"
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+        run: |
+          echo "TAG=${{ github.event.inputs.ref }}" >> $GITHUB_ENV
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+          ref: ${{ github.event.inputs.ref }}
+          repository: ${{ github.event.inputs.repo }}
       - name: Fetch Deps
         run: ci/actions/osx/install_deps.sh
       - name: Build Artifact
@@ -38,9 +42,13 @@ jobs:
       - uses: actions/checkout@722adc6
       - uses: chrislennon/action-aws-cli@f0f8671
       - name: tag
-        run: echo "::set-env name=TAG::`git describe --tags $GITHUB_SHA`"
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+        run: |
+          echo "TAG=${{ github.event.inputs.ref }}" >> $GITHUB_ENV
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+          ref: ${{ github.event.inputs.ref }}
+          repository: ${{ github.event.inputs.repo }}
       - name: Fetch Deps
         run: ci/actions/linux/install_deps.sh
       - name: Build Artifact
@@ -58,27 +66,42 @@ jobs:
       - uses: actions/checkout@722adc6
       - uses: chrislennon/action-aws-cli@f0f8671
       - name: tag
-        run: echo "::set-env name=TAG::`git describe --tags $GITHUB_SHA`"
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+        run: |
+          echo "TAG=${{ github.event.inputs.ref }}" >> $GITHUB_ENV
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+          ref: ${{ github.event.inputs.ref }}
+          repository: ${{ github.event.inputs.repo }}
       - name: Fetch Deps
         run: ci/actions/linux/install_deps.sh
       - name: Deploy Docker (nanocurrency/nano-beta)
         run: TRAVIS_TAG=${TAG} ci/actions/linux/deploy-docker.sh
         env:
           DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
-
+
+      - name: Login to ghcr.io
+        uses: docker/login-action@adb73476b6e06caddec5db0bc1deacbec8cdd947
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GHCR_PAT }}
+      - name: Deploy Docker (ghcr.io)
+        run: ci/actions/linux/ghcr_push.sh
+
   windows_job:
     runs-on: windows-latest
     steps:
       - uses: actions/checkout@722adc6
       - uses: chrislennon/action-aws-cli@f0f8671
       - name: tag
         run: |
-          $TRAVIS_TAG=git describe --tags $GITHUB_SHA
-          echo "::set-env name=TAG::$TRAVIS_TAG"
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+          Write-Output "TAG=${{ github.event.inputs.ref }}" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+          ref: ${{ github.event.inputs.ref }}
+          repository: ${{ github.event.inputs.repo }}
       - name: Fetch Deps
         run: ci/actions/windows/install_deps.ps1
       - name: Build Artifact

.github/workflows/develop.yml

@@ -8,11 +8,19 @@ jobs:
   linux_job:
     runs-on: ubuntu-18.04
     steps:
-      - uses: actions/checkout@722adc6
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
       - name: Fetch Deps
         run: ci/actions/linux/install_deps.sh
+      - name: Login to ghcr.io
+        uses: docker/login-action@adb73476b6e06caddec5db0bc1deacbec8cdd947
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GHCR_PAT }}
+      - name: Deploy Docker (ghcr.io)
+        run: ci/actions/linux/ghcr_push.sh
       - name: Deploy Docker (nanocurrency/nano-env)
         run: ci/actions/linux/deploy-docker.sh
         env:

.github/workflows/live_artifacts.yml

@@ -17,9 +17,13 @@ jobs:
       - uses: actions/checkout@722adc6
       - uses: chrislennon/action-aws-cli@f0f8671
       - name: tag
-        run: echo "::set-env name=TAG::`git describe --tags $GITHUB_SHA`"
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+        run: |
+          echo "TAG=${{ github.event.inputs.ref }}" >> $GITHUB_ENV
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+          ref: ${{ github.event.inputs.ref }}
+          repository: ${{ github.event.inputs.repo }}
       - name: Fetch Deps
         run: ci/actions/osx/install_deps.sh
       - name: Build Artifact
@@ -37,9 +41,13 @@ jobs:
       - uses: actions/checkout@722adc6
       - uses: chrislennon/action-aws-cli@f0f8671
       - name: tag
-        run: echo "::set-env name=TAG::`git describe --tags $GITHUB_SHA`"
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+        run: |
+          echo "TAG=${{ github.event.inputs.ref }}" >> $GITHUB_ENV
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+          ref: ${{ github.event.inputs.ref }}
+          repository: ${{ github.event.inputs.repo }}
       - name: Fetch Deps
         run: ci/actions/linux/install_deps.sh
       - name: Build Artifact
@@ -57,27 +65,42 @@ jobs:
       - uses: actions/checkout@722adc6
       - uses: chrislennon/action-aws-cli@f0f8671
       - name: tag
-        run: echo "::set-env name=TAG::`git describe --tags $GITHUB_SHA`"
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+        run: |
+          echo "TAG=${{ github.event.inputs.ref }}" >> $GITHUB_ENV
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+          ref: ${{ github.event.inputs.ref }}
+          repository: ${{ github.event.inputs.repo }}
       - name: Fetch Deps
         run: ci/actions/linux/install_deps.sh
       - name: Deploy Docker (nanocurrency/nano)
         run: TRAVIS_TAG=${TAG} ci/actions/linux/deploy-docker.sh
         env:
           DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
-
+
+      - name: Login to ghcr.io
+        uses: docker/login-action@adb73476b6e06caddec5db0bc1deacbec8cdd947
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GHCR_PAT }}
+      - name: Deploy Docker (ghcr.io
+        run: ci/actions/linux/ghcr_push.sh
+
   windows_job:
     runs-on: windows-latest
     steps:
       - uses: actions/checkout@722adc6
       - uses: chrislennon/action-aws-cli@f0f8671
       - name: tag
         run: |
-          $TRAVIS_TAG=git describe --tags $GITHUB_SHA
-          echo "::set-env name=TAG::$TRAVIS_TAG"
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+          Write-Output "TAG=${{ github.event.inputs.ref }}" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+          ref: ${{ github.event.inputs.ref }}
+          repository: ${{ github.event.inputs.repo }}
       - name: Fetch Deps
         run: ci/actions/windows/install_deps.ps1
       - name: Build Artifact

.github/workflows/release_test.yml

@@ -14,9 +14,10 @@ jobs:
     env:
       BOOST_ROOT: /tmp/boost
     steps:
-      - uses: actions/checkout@722adc6
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+
       - name: Fetch Deps
         run: TEST=1 ci/actions/osx/install_deps.sh
       - name: Run Tests
@@ -26,9 +27,10 @@ jobs:
     runs-on: ubuntu-18.04
     timeout-minutes: 60
     steps:
-      - uses: actions/checkout@722adc6
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+
       - name: Fetch Deps
         run: ci/actions/linux/install_deps.sh
       - name: Run Tests
@@ -38,9 +40,10 @@ jobs:
     runs-on: ubuntu-18.04
     timeout-minutes: 60
     steps:
-      - uses: actions/checkout@722adc6
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+
       - name: Fetch Deps
         run: ci/actions/linux/install_deps.sh
       - name: Run Tests
@@ -50,9 +53,9 @@ jobs:
     runs-on: windows-latest
     timeout-minutes: 60
     steps:
-      - uses: actions/checkout@722adc6
-      - name: Checkout Submodules
-        run: git submodule update --init --recursive
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
       - name: Fetch Deps
         run: ci/actions/windows/install_deps.ps1
       - name: Run Tests

.github/workflows/rocksdb_release_tests.yml

@@ -0,0 +1,48 @@
+name: RocksDB Release Tests
+
+on: 
+  push: 
+    tags: '*'
+
+env:
+  RELEASE: 1
+  artifact: 0
+  TEST_USE_ROCKSDB: 1
+
+jobs:
+  osx_test:
+    runs-on: macos-latest
+    env:
+      BOOST_ROOT: /tmp/boost
+    steps:
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+      - name: Fetch Deps
+        run: TEST=1 ci/actions/osx/install_deps.sh
+      - name: Run Tests
+        run: ci/build-travis.sh "/tmp/qt/lib/cmake/Qt5";
+
+  gcc_test:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+      - name: Fetch Deps
+        run: ci/actions/linux/install_deps.sh
+      - name: Run Tests
+        run: docker run -e TEST_USE_ROCKSDB -v ${PWD}:/workspace nanocurrency/nano-env:gcc /bin/bash -c "cd /workspace && RELEASE=1 ./ci/build-travis.sh /usr/lib/x86_64-linux-gnu/cmake/Qt5 ${PWD}"
+
+  clang_test:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+      - name: Fetch Deps
+        run: ci/actions/linux/install_deps.sh
+      - name: Run Tests
+        run: docker run -e TEST_USE_ROCKSDB -v ${PWD}:/workspace nanocurrency/nano-env:clang-6 /bin/bash -c "cd /workspace && RELEASE=1 ./ci/build-travis.sh /usr/lib/x86_64-linux-gnu/cmake/Qt5 ${PWD}"

.github/workflows/rocksdb_tests.yml

@@ -0,0 +1,68 @@
+name: RocksDB Tests
+
+on: [push, pull_request]
+
+env:
+  RELEASE: 0
+  artifact: 0
+  TEST_USE_ROCKSDB: 1
+
+jobs:
+  osx_test:
+    runs-on: macos-latest
+    env:
+      BOOST_ROOT: /tmp/boost
+      DEADLINE_SCALE_FACTOR: 2
+    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
+    steps:
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+      - name: Fetch Deps
+        run: TEST=1 ci/actions/osx/install_deps.sh
+      - name: Run Tests
+        run: ci/build-travis.sh "/tmp/qt/lib/cmake/Qt5";
+
+  gcc_test:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 60
+    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
+    steps:
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+      - name: Fetch Deps
+        run: ci/actions/linux/install_deps.sh
+      - name: Run Tests
+        run: docker run -e TEST_USE_ROCKSDB -v ${PWD}:/workspace nanocurrency/nano-env:gcc /bin/bash -c "cd /workspace && ./ci/build-travis.sh /usr/lib/x86_64-linux-gnu/cmake/Qt5 ${PWD}"
+
+  clang_test:
+    runs-on: ubuntu-18.04
+    timeout-minutes: 60
+    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
+    steps:
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+      - name: Fetch Deps
+        run: ci/actions/linux/install_deps.sh
+      - name: Run Tests
+        run: docker run -e TEST_USE_ROCKSDB -v ${PWD}:/workspace nanocurrency/nano-env:clang-6 /bin/bash -c "cd /workspace && ./ci/build-travis.sh /usr/lib/x86_64-linux-gnu/cmake/Qt5 ${PWD}"
+
+  windows_test:
+    runs-on: windows-latest
+    timeout-minutes: 60
+    env:
+      RELEASE: 1
+      DEADLINE_SCALE_FACTOR: 2
+    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
+    steps:
+      - uses: actions/checkout@5a4ac9002d0be2fb38bd78e4b4dbde5606d7042f
+        with:
+          submodules: "recursive"
+      - name: Windows Defender
+        run: ci/actions/windows/disable_windows_defender.ps1
+      - name: Fetch Deps
+        run: ci/actions/windows/install_deps.ps1
+      - name: Run Tests
+        run: ci/actions/windows/build.ps1