Gateway V2 - Client stuck on "markets are not ready"

[rapcmia]

Describe the bug
When testing the gateway-v2 on staging branch we've noticed that the client would get stuck on markets are not ready for both uniswap mainnet and pangolin avalanche.

When checking uniswap mainnet, we are getting the error below of replacement fee too low [ See: https://links.ethers.org/v5-errors-REPLACEMENT_UNDERPRICED ]

Error: replacement fee too low [ See: https://links.ethers.org/v5-errors-REPLACEMENT_UNDERPRICED ] (error={"reason":"processing response error","code":"SERVER_ERROR","body":"{\"jsonrpc\":\"2.0\",\"id\":61,\"error\":{\"code\":-32000,\"message\":\"replacement transaction underpriced\"}}","error":{"code":-32000},"requestBody":"{\"method\":\"eth_sendRawTransaction\",\"params\":[\"0x02f8b1016f850861c46800850861c46800830186a0946b175474e89094c44da98b954eedeac495271d0f80b844095ea7b300000000000000000000000068b3465833fb72a70ecdf485e0e4c7bd8665fc45ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffc001a0a8eb38671de87319538aa83fa08560b0d937d66afa01012ff0acf626b7a6b9c79f12799ed98eaba12e0d714e546ed2bb2a292bf6a3b0ac2174bebfa759fcc5f3\"],\"id\":61,\"jsonrpc\":\"2.0\"}","requestMethod":"POST","url":"https://mainnet.infura.io/v3/83381371870e413aaac6b919bcfddd8f"}, method="sendTransaction", transaction="0x02f8b1016f850861c46800850861c46800830186a0946b175474e89094c44da98b954eedeac495271d0f80b844095ea7b300000000000000000000000068b3465833fb72a70ecdf485e0e4c7bd8665fc45ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffc001a0a8eb38671de87319538aa83fa08560b0d937d66afa01012ff0acf626b7a6b9c79f12799ed98eaba12e0d714e546ed2bb2a292bf6a3b0ac2174bebfa759fcc5f3", code=REPLACEMENT_UNDERPRICED, version=providers/5.6.8)
    at Logger.makeError (/usr/src/app/node_modules/@ethersproject/logger/src.ts/index.ts:261:28)
    at Logger.throwError (/usr/src/app/node_modules/@ethersproject/logger/src.ts/index.ts:273:20)
    at checkError (/usr/src/app/node_modules/@ethersproject/providers/src.ts/json-rpc-provider.ts:112:16)
    at StaticJsonRpcProvider.<anonymous> (/usr/src/app/node_modules/@ethersproject/providers/src.ts/json-rpc-provider.ts:603:20)
    at step (/usr/src/app/node_modules/@ethersproject/providers/lib/json-rpc-provider.js:48:23)
    at Object.throw (/usr/src/app/node_modules/@ethersproject/providers/lib/json-rpc-provider.js:29:53)
    at rejected (/usr/src/app/node_modules/@ethersproject/providers/lib/json-rpc-provider.js:21:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

When checking uniswap kovan testnet, it seems it able to behave as expected when we want to just show no opportunity for arbitrage however when setting the min_profitability to -1 to execute trade, we are getting an error of Value cannot be null or undefined. Tried to check whether its something to do with the order_amount but still getting the same error.

2022-06-16 10:30:34,678 - 162836 - hummingbot.strategy.amm_arb.amm_arb - INFO - Placing BUY order for 0.010000000000000 WETH at uniswap_ethereum_kovan at 24.711670000000000 price [clock=2022-06-16 10:30:34+00:00]
2022-06-16 10:30:40,764 - 162836 - hummingbot.core.gateway.gateway_http_client - INFO - An unknown error has occurred on gateway. Please send your logs to dev@hummingbot.io
2022-06-16 10:30:40,765 - 162836 - hummingbot.core.gateway.gateway_http_client - ERROR - Error on POST https://localhost:22237/amm/trade Error: {'message': 'Unknown error.', 'httpErrorCode': 503, 'errorCode': 1099, 'stack': 'Error: Value cannot be null or undefined\n    at ClassicLevel._checkValue (/usr/src/app/node_modules/abstract-level/abstract-level.js:788:14)\n    at ClassicLevel.put (/usr/src/app/node_modules/abstract-level/abstract-level.js:423:45)\n    at LocalStorage.<anonymous> (/usr/src/app/dist/src/services/local-storage.js:70:71)\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/src/app/dist/src/services/local-storage.js:5:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)'}
2022-06-16 10:30:40,765 - 162836 - GatewayEVMAMM - ERROR - Error submitting BUY swap order to uniswap on kovan for 0.010000000000000 WETH-USDC 24.711670000000000.
Traceback (most recent call last):
  File "/home/ubuntu/github/staging/hummingbot/connector/gateway_EVM_AMM.py", line 505, in _create_order
    order_result: Dict[str, Any] = await GatewayHttpClient.get_instance().amm_trade(
  File "/home/ubuntu/github/staging/hummingbot/core/gateway/gateway_http_client.py", line 404, in amm_trade
    return await self.api_request("post", "amm/trade", request_payload)
  File "/home/ubuntu/github/staging/hummingbot/core/gateway/gateway_http_client.py", line 167, in api_request
    raise e
  File "/home/ubuntu/github/staging/hummingbot/core/gateway/gateway_http_client.py", line 163, in api_request
    raise ValueError(f"Error on {method.upper()} {url} Error: {parsed_response}")
ValueError: Error on POST https://localhost:22237/amm/trade Error: {'message': 'Unknown error.', 'httpErrorCode': 503, 'errorCode': 1099, 'stack': 'Error: Value cannot be null or undefined\n    at ClassicLevel._checkValue (/usr/src/app/node_modules/abstract-level/abstract-level.js:788:14)\n    at ClassicLevel.put (/usr/src/app/node_modules/abstract-level/abstract-level.js:423:45)\n    at LocalStorage.<anonymous> (/usr/src/app/dist/src/services/local-storage.js:70:71)\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/src/app/dist/src/services/local-storage.js:5:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)'}
2022-06-16 10:30:40,769 - 162836 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1655375440.0, "order_id": "buy-WETH-USDC-1655375434678732", "order_type": "OrderType.LIMIT", "event_name": "MarketOrderFailureEvent", "event_source": "uniswap_ethereum_kovan"}

For pangolin avalanche, getting a call revert exception resulting to the client getting stuck on markets are not ready

2022-06-17 12:00:00 | error | 	call revert exception [ See: https://links.ethers.org/v5-errors-CALL_EXCEPTION ] (method="balanceOf(address)", data="0x", errorArgs=null, errorName=null, errorSignature=null, reason=null, code=CALL_EXCEPTION, version=abi/5.6.3) | Error: call revert exception [ See: https://links.ethers.org/v5-errors-CALL_EXCEPTION ] (method="balanceOf(address)", data="0x", errorArgs=null, errorName=null, errorSignature=null, reason=null, code=CALL_EXCEPTION, version=abi/5.6.3)
    at Logger.makeError (/usr/src/app/node_modules/@ethersproject/logger/src.ts/index.ts:261:28)
    at Logger.throwError (/usr/src/app/node_modules/@ethersproject/logger/src.ts/index.ts:273:20)
    at Interface.decodeFunctionResult (/usr/src/app/node_modules/@ethersproject/contracts/node_modules/@ethersproject/abi/src.ts/interface.ts:427:23)
    at Contract.<anonymous> (/usr/src/app/node_modules/@ethersproject/contracts/src.ts/index.ts:400:44)
    at step (/usr/src/app/node_modules/@ethersproject/contracts/lib/index.js:48:23)
    at Object.next (/usr/src/app/node_modules/@ethersproject/contracts/lib/index.js:29:53)
    at fulfilled (/usr/src/app/node_modules/@ethersproject/contracts/lib/index.js:20:58)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

Tried to crosscheck with traderjoe(avalanche), sushiswap(mainnet/kovan) and development branch and got the same behavior.

Steps To Reproduce

1. Install and clone the staging branch (or development branch)
2. Manually create a gateway docker image from ./gateway folder
3. Setup gateway for pangolin (avalanche) and uniswap (mainnet or kovan testnet)
4. Setup simple AMM Arbitrage strategy then start the client
5. Notice the log panel then check the docker logs

Release version
staging 1.5 and dev-1.6

Attachments
gateway-v2-staging-branch.zip

[rapcmia]

Tried to investigate commits from development branch up to d4593a7f4164a165258ad5d2ba297d85c5b62a58 and still getting the same issue.

[james-hummingbot]

The problem is the nonce manager. It is providing the wrong number so the approve transaction will never succeed. This PR should resolve it #5381

[rapcmia]

fixed