test: reset cucumber timeouts to practical limits (#3531)

Description
---
- Re-applied reasonable time outs to cucumber tests, using dynamic time-outs in test steps where possible.
- Fixed a broken test.
- Fixed some flaky tests.
- Removed some dead code.
- Removed duplicate test.

Note: Replaces #3511.

Motivation and Context
---
- Tests were flaky on slow and fast computers due to stringent time outs..
- Some test logic was flawed.
- Some flaky tests could be improved.

How Has This Been Tested?
---
Cucumber on slow and fast computers

applications/tari_base_node/src/grpc/base_node_grpc_server.rs

@@ -559,10 +559,13 @@ impl tari_rpc::base_node_server::BaseNode for BaseNodeGrpcServer {
             })?;
 
         if !base_node_response.is_empty() {
-            debug!(target: LOG_TARGET, "Sending Transaction state response to client");
             let response = tari_rpc::TransactionStateResponse {
                 result: tari_rpc::TransactionLocation::Mined.into(),
             };
+            debug!(
+                target: LOG_TARGET,
+                "Sending Transaction state response to client {:?}", response
+            );
             return Ok(Response::new(response));
         }
 
@@ -592,7 +595,10 @@ impl tari_rpc::base_node_server::BaseNode for BaseNodeGrpcServer {
             },
         };
 
-        debug!(target: LOG_TARGET, "Sending Transaction state response to client");
+        debug!(
+            target: LOG_TARGET,
+            "Sending Transaction state response to client {:?}", response
+        );
         Ok(Response::new(response))
     }
 

integration_tests/features/Mempool.feature

@@ -12,6 +12,7 @@ Feature: Mempool
     And I have 8 base nodes connected to all seed nodes
     When I mine a block on SENDER with coinbase CB1
     When I mine 2 blocks on SENDER
+    Then all nodes are at height 3
     When I create a transaction TX1 spending CB1 to UTX1
     When I submit transaction TX1 to SENDER
     Then SENDER has TX1 in MEMPOOL state
@@ -25,60 +26,46 @@ Feature: Mempool
     And I have 2 base nodes connected to all seed nodes
     When I mine a block on SENDER with coinbase CB1
     When I mine 2 blocks on SENDER
+    Then all nodes are at height 3
     When I create a transaction TX1 spending CB1 to UTX1
     When I submit transaction TX1 to SENDER
     Then SENDER has TX1 in MEMPOOL state
     Then TX1 is in the MEMPOOL of all nodes
     Given I have a base node NODE1 connected to all seed nodes
     Then NODE1 has TX1 in MEMPOOL state
     When I mine 1 blocks on SENDER
+    Then all nodes are at height 4
     Then SENDER has TX1 in MINED state
     Then TX1 is in the MINED of all nodes
 
-  @broken
+  @critical
   Scenario: Clear out mempool
     Given I have 1 seed nodes
     And I have a base node SENDER connected to all seed nodes
     When I mine a block on SENDER with coinbase CB1
     When I mine a block on SENDER with coinbase CB2
     When I mine a block on SENDER with coinbase CB3
     When I mine 4 blocks on SENDER
-    When I create a custom fee transaction TX1 spending CB1 to UTX1 with fee 16
-    When I create a custom fee transaction TX2 spending CB2 to UTX2 with fee 20
-    When I create a custom fee transaction TX3 spending CB3 to UTX3 with fee 18
+    When I create a custom fee transaction TX1 spending CB1 to UTX1 with fee 1600
+    When I create a custom fee transaction TX2 spending CB2 to UTX2 with fee 2000
+    When I create a custom fee transaction TX3 spending CB3 to UTX3 with fee 1800
     When I submit transaction TX1 to SENDER
     When I submit transaction TX2 to SENDER
     When I submit transaction TX3 to SENDER
     Then SENDER has TX1 in MEMPOOL state
     Then SENDER has TX2 in MEMPOOL state
     Then SENDER has TX3 in MEMPOOL state
-    When I mine 1 custom weight blocks on SENDER with weight 17
+    # Note: The block weight should only allow one transaction to be included in the block template
+    When I mine 1 custom weight blocks on SENDER with weight 80
     Then SENDER has TX1 in MEMPOOL state
     Then SENDER has TX2 in MINED state
     Then SENDER has TX3 in MEMPOOL state
-    When I mine 1 custom weight blocks on SENDER with weight 17
+    # Note: The block weight should only allow one transaction to be included in the block template
+    When I mine 1 custom weight blocks on SENDER with weight 80
     Then SENDER has TX1 in MEMPOOL state
     Then SENDER has TX2 in MINED state
     Then SENDER has TX3 in MINED state
 
-
-  @critical
-  Scenario: Double spend
-    Given I have 1 seed nodes
-    And I have a base node SENDER connected to all seed nodes
-    When I mine a block on SENDER with coinbase CB1
-    When I mine 4 blocks on SENDER
-    When I create a custom fee transaction TX1 spending CB1 to UTX1 with fee 16
-    When I create a custom fee transaction TX2 spending CB1 to UTX2 with fee 20
-    When I submit transaction TX1 to SENDER
-    When I submit transaction TX2 to SENDER
-    Then SENDER has TX1 in MEMPOOL state
-    Then SENDER has TX2 in MEMPOOL state
-    When I mine 1 blocks on SENDER
-    # a transaction that was removed from the pool will be reported as unknown as long as it is stored in the reorg pool for 5 mins
-    Then SENDER has TX1 in UNKNOWN state
-    Then SENDER has TX2 in MINED state
-
  @long-running
   Scenario: Double spend eventually ends up as not stored
     Given I have 1 seed nodes
@@ -92,28 +79,38 @@ Feature: Mempool
     Then SENDER has TX1 in MEMPOOL state
     Then SENDER has TX2 in MEMPOOL state
     When I mine 1 blocks on SENDER
-    # a transaction that was removed from the pool will be reported as unknown as long as it is stored in the reorg pool for 5 mins
+    # A transaction that was removed from the pool will be reported as unknown as long as it is stored in the reorg pool
+    # for 5 minutes
+    Then SENDER has TX1 in UNKNOWN state
     Then SENDER has TX1 in NOT_STORED state
     Then SENDER has TX2 in MINED state
 
   @critical @flaky
   Scenario: Mempool clearing out invalid transactions after a reorg
     Given I have a seed node SEED_A
     And I have a base node NODE_A connected to seed SEED_A
+    And I have wallet WALLET_A connected to base node NODE_A
+    And I have mining node MINING_A connected to base node NODE_A and wallet WALLET_A
     When I mine a block on NODE_A with coinbase CB_A
-    When I mine 3 blocks on NODE_A
+    And mining node MINING_A mines 3 blocks with min difficulty 1 and max difficulty 2
+    Then node SEED_A is at height 4
     Given I have a seed node SEED_B
     And I have a base node NODE_B connected to seed SEED_B
+    And I have wallet WALLET_B connected to base node NODE_B
+    And I have mining node MINING_B connected to base node NODE_B and wallet WALLET_B
     When I mine a block on NODE_B with coinbase CB_B
-    When I mine 10 blocks on NODE_B
+    And mining node MINING_B mines 10 blocks with min difficulty 20 and max difficulty 9999999999
+    Then node SEED_B is at height 11
     When I create a custom fee transaction TXA spending CB_A to UTX1 with fee 16
     When I create a custom fee transaction TXB spending CB_B to UTX1 with fee 16
     When I submit transaction TXA to NODE_A
     When I submit transaction TXB to NODE_B
     Then NODE_A has TXA in MEMPOOL state
     Then NODE_B has TXB in MEMPOOL state
-    When I mine 1 blocks on NODE_A
-    When I mine 1 blocks on NODE_B
+    And mining node MINING_A mines 1 blocks with min difficulty 1 and max difficulty 2
+    And mining node MINING_B mines 1 blocks with min difficulty 20 and max difficulty 9999999999
+    Then node SEED_A is at height 5
+    Then node SEED_B is at height 12
     And I connect node NODE_A to node NODE_B
     Then all nodes are at height 12
     Then NODE_A has TXA in NOT_STORED state

integration_tests/features/Sync.feature

@@ -70,22 +70,27 @@ Feature: Block Sync
   @long-running @flaky
   Scenario: Node should not sync from pruned node
     Given I have a base node NODE1 connected to all seed nodes
-    And I have a pruned node PNODE1 connected to node NODE1 with pruning horizon set to 5
-    When I mine 40 blocks on NODE1
+    And I have wallet WALLET1 connected to base node NODE1
+    And I have mining node MINING1 connected to base node NODE1 and wallet WALLET1
+    And I have a pruned node PNODE1 connected to node NODE1 with pruning horizon set to 6
+    When mining node MINING1 mines 40 blocks with min difficulty 20 and max difficulty 9999999999
     Then all nodes are at height 40
     When I stop node NODE1
-    Given I have a base node NODE2 connected to node PNODE1
     Given I have a pruned node PNODE2 connected to node PNODE1 with pruning horizon set to 5
-    When I mine 5 blocks on NODE2
-    Then node NODE2 is at height 5
+    Given I have a base node NODE2
+    And I have wallet WALLET2 connected to base node NODE2
+    And I have mining node MINING2 connected to base node NODE2 and wallet WALLET2
+    When mining node MINING2 mines 5 blocks with min difficulty 1 and max difficulty 2
+    And I connect node NODE2 to node PNODE1
+    And I connect node NODE2 to node PNODE2
     Then node PNODE2 is at height 40
+    Then node NODE2 is at height 5
     When I start base node NODE1
     # We need for node to boot up and supply node 2 with blocks
     And I connect node NODE2 to node NODE1
     # NODE2 may initially try to sync from PNODE1 and PNODE2, then eventually try to sync from NODE1; mining blocks
     # on NODE1 will make this test less flaky and force NODE2 to sync from NODE1 much quicker
     When I mine 10 blocks on NODE1
-    Then all transactions must have succeeded
     Then all nodes are at height 50
 
   Scenario Outline: Syncing node while also mining before tip sync
@@ -100,7 +105,7 @@ Feature: Block Sync
     Then node SEED is at height <X1>
     When I start base node SYNCER
     # Try to mine much faster than block sync, but still producing a lower accumulated difficulty
-    And mining node MINER2 mines <Y1> blocks with min difficulty 1 and max difficulty 10
+    And mining node MINER2 mines <Y1> blocks with min difficulty 1 and max difficulty 2
     Then node SYNCER is at the same height as node SEED
     @critical
     Examples:

integration_tests/features/TransactionInfo.feature

@@ -4,7 +4,6 @@ Feature: Transaction Info
 @long-running
 Scenario: Get Transaction Info
     Given I have a seed node NODE
-        # TODO: This test takes an hour if only one base node is used
     And I have a SHA3 miner MINER connected to all seed nodes
     And I have wallet WALLET_A connected to all seed nodes
     And I have wallet WALLET_B connected to all seed nodes

integration_tests/features/WalletCli.feature

@@ -76,7 +76,7 @@ Feature: Wallet CLI
         And I have wallet RECEIVER connected to base node BASE
         And I have mining node MINE connected to base node BASE and wallet SENDER
         And mining node MINE mines 15 blocks
-        Then wallets SENDER should have 12 spendable coinbase outputs
+        Then wallets SENDER should have EXACTLY 12 spendable coinbase outputs
         # TODO: Remove this wait when the wallet CLI commands involving transactions will only commence with a valid
         # TODO: base node connection.
         And I wait 30 seconds
@@ -123,6 +123,7 @@ Feature: Wallet CLI
         Then I wait for wallet WALLET to have at least 1000000 uT
         And I export the utxos of wallet WALLET via command line
 
+    @flaky
     Scenario: As a user I want to discover-peer via command line
         Given I have a seed node SEED
         And I have a base node BASE1 connected to seed SEED

integration_tests/features/WalletFFI.feature

@@ -89,7 +89,12 @@ Feature: Wallet FFI
         Then I wait for ffi wallet FFI_WALLET to have at least 1000000 uT
         And I send 1000000 uT from ffi wallet FFI_WALLET to wallet RECEIVER at fee 20
         Then ffi wallet FFI_WALLET detects AT_LEAST 2 ffi transactions to be Broadcast
-        And mining node MINER mines 10 blocks
+        # The broadcast check does not include delivery; create some holding points to ensure it was received
+        And mining node MINER mines 2 blocks
+        Then all nodes are at height 22
+        And mining node MINER mines 2 blocks
+        Then all nodes are at height 24
+        And mining node MINER mines 6 blocks
         Then I wait for wallet RECEIVER to have at least 1000000 uT
         And I have 1 received and 1 send transaction in ffi wallet FFI_WALLET
         And I start TXO validation on ffi wallet FFI_WALLET

integration_tests/features/WalletMonitoring.feature

@@ -2,7 +2,7 @@
 Feature: Wallet Monitoring
 
 
-@flaky
+  @flaky
   Scenario: Wallets monitoring coinbase after a reorg
         #
         # Chain 1:
@@ -61,17 +61,18 @@ Feature: Wallet Monitoring
     And I have wallet WALLET_A1 connected to seed node SEED_A
     And I have wallet WALLET_A2 connected to seed node SEED_A
     And I have mining node MINING_A connected to base node SEED_A and wallet WALLET_A1
-    And mining node MINING_A mines 10 blocks
+    When mining node MINING_A mines 10 blocks with min difficulty 20 and max difficulty 9999999999
     Then node SEED_A is at height 10
     Then node NODE_A1 is at height 10
-    Then wallet WALLET_A1 detects at least 7 coinbase transactions as Mined_Confirmed
+    Then wallet WALLET_A1 detects exactly 7 coinbase transactions as Mined_Confirmed
         # Use 7 of the 10 coinbase UTXOs in transactions (others require 3 confirmations)
     And I multi-send 7 transactions of 1000000 uT from wallet WALLET_A1 to wallet WALLET_A2 at fee 100
-    When I mine 100 blocks on SEED_A
-    Then node SEED_A is at height 110
-    Then node NODE_A1 is at height 110
+    When mining node MINING_A mines 10 blocks with min difficulty 20 and max difficulty 9999999999
+    Then node SEED_A is at height 20
+    Then node NODE_A1 is at height 20
     Then wallet WALLET_A2 detects all transactions as Mined_Confirmed
     Then all NORMAL transactions for wallet WALLET_A1 are valid
+    Then wallet WALLET_A1 detects exactly 17 coinbase transactions as Mined_Confirmed
         #
         # Chain 2:
         #   Collects 10 coinbases into one wallet, send 7 transactions
@@ -82,25 +83,28 @@ Feature: Wallet Monitoring
     And I have wallet WALLET_B1 connected to seed node SEED_B
     And I have wallet WALLET_B2 connected to seed node SEED_B
     And I have mining node MINING_B connected to base node SEED_B and wallet WALLET_B1
-    And mining node MINING_B mines 10 blocks
+    When mining node MINING_B mines 10 blocks with min difficulty 1 and max difficulty 2
     Then node SEED_B is at height 10
     Then node NODE_B1 is at height 10
-    Then wallet WALLET_B1 detects at least 7 coinbase transactions as Mined_Confirmed
+    Then wallet WALLET_B1 detects exactly 7 coinbase transactions as Mined_Confirmed
         # Use 7 of the 10 coinbase UTXOs in transactions (others require 3 confirmations)
     And I multi-send 7 transactions of 1000000 uT from wallet WALLET_B1 to wallet WALLET_B2 at fee 100
-    When I mine 100 blocks on SEED_B
-    Then node SEED_B is at height 110
-    Then node NODE_B1 is at height 110
+    When mining node MINING_B mines 10 blocks with min difficulty 1 and max difficulty 2
+    Then node SEED_B is at height 20
+    Then node NODE_B1 is at height 20
     Then wallet WALLET_B2 detects all transactions as Mined_Confirmed
     Then all NORMAL transactions for wallet WALLET_B1 are valid
+    Then wallet WALLET_B1 detects exactly 17 coinbase transactions as Mined_Confirmed
         #
         # Connect Chain 1 and 2
         #
     And I have a SHA3 miner NODE_C connected to all seed nodes
-    Then all nodes are at height 110
+    Then all nodes are at height 20
         # When tip advances past required confirmations, invalid coinbases still being monitored will be cancelled.
     And mining node NODE_C mines 6 blocks
-    Then all nodes are at height 116
+    Then all nodes are at height 26
+    Then wallet WALLET_A1 detects exactly 20 coinbase transactions as Mined_Confirmed
+    Then wallet WALLET_B1 detects exactly 17 coinbase transactions as Mined_Confirmed
     And I list all NORMAL transactions for wallet WALLET_A1
     And I list all NORMAL transactions for wallet WALLET_B1
     # TODO: Uncomment this step when wallets can handle reorg
@@ -124,22 +128,22 @@ Feature: Wallet Monitoring
     And I have mining node MINER2 connected to base node NODE2 and wallet WALLET2
 
     When I co-mine <numBlocks> blocks via merge mining proxy PROXY1 and mining node MINER2
-    Then all nodes are on the same chain at height <numBlocks>
+    Then all nodes are on the same chain tip
 
     And mining node MINER_SEED_A mines 5 blocks
-    Then all nodes are on the same chain at height <endBlocks>
+    Then all nodes are on the same chain tip
 
     When I wait 1 seconds
-    Then wallets WALLET1,WALLET2 should have <numBlocks> spendable coinbase outputs
+    Then wallets WALLET1,WALLET2 should have AT_LEAST <numBlocks> spendable coinbase outputs
 
     @flaky
     Examples:
-      | numBlocks | endBlocks |
-      | 10        | 15        |
+      | numBlocks |
+      | 10        |
 
-    @long-running
+    @long-running @flaky
     Examples:
-        | numBlocks | endBlocks |
-        | 100       | 105       |
-        | 1000      | 1005      |
-        | 4500      | 4505      |
+        | numBlocks |
+        | 100       |
+        | 1000      |
+        | 4500      |

integration_tests/features/WalletRoutingMechanism.feature

@@ -8,9 +8,8 @@ Scenario Outline: Wallets transacting via specified routing mechanism only
     And I have non-default wallet WALLET_A connected to all seed nodes using <Mechanism>
     And I have mining node MINER connected to base node NODE and wallet WALLET_A
     And I have <NumWallets> non-default wallets connected to all seed nodes using <Mechanism>
-    And I have a merge mining proxy PROXY connected to NODE and WALLET_A with default config
         # We need to ensure the coinbase lock heights are gone and we have enough individual UTXOs; mine enough blocks
-    When I merge mine 20 blocks via PROXY
+    And mining node MINER mines 20 blocks
     Then all nodes are at height 20
         # TODO: This wait is needed to stop base nodes from shutting down
     When I wait 1 seconds