When `wallet.json` has lots of chains (>= 100), it'll take long time (> 60s) to query validator

[kikakkz]

Description
When we have lots of chains in wallet.json，it'll be very slow to execute the Transfer operation. I add a simple semaphore to limit concurrent requests to a single validator and sometimes it still responds slow, and it always responds fast when I run the same test with localnet.

Steps

• 1 git clone https://github.com/linera-io/linera-protocol.git
• 2 cd linera-protocol; git checkout testnet_babbage
• 3 cargo build --release --features storage-service
• 4 Copy the script section as test.sh, and modify the linera path to your local PC path
• 5 chmod a+x test.sh
• 6 ./test.sh
  • It will create wallet and storage in currently path then request chain from Babbage to test automatically
• 7 Modify chains count in the script to get different transfer time comsuption

Expected result
For me it should be reasonable to manage 200 chains in an 48 cores / 256GiB machine, and won't get significant delay when transfer.

Actual result

• If we only have 3 chains in the wallet, it take 1s to transfer.
• If we have 50 chains in the wallet, it still take 1.8s to transfer.
• If we have 100 chains in the wallet, it will take more than 2m, and finally fail.
• If we have 200 chains in the wallet, it will take more than 2m, and finally fail.

Script

#!/bin/bash

kill -9 $(ps | grep linera | awk '{print $1}')

PORT=31980

export PATH=$PWD/../linera-protocol/target/release:$PATH
export LINERA_WALLET=$PWD/wallet.json
STORAGE_DIR=$PWD/client.db
export LINERA_STORAGE=rocksdb:$STORAGE_DIR

rm $LINERA_WALLET $STORAGE_DIR -rf

linera wallet init --faucet https://faucet.testnet-babbage.linera.net --with-new-chain

FROM_CHAIN=$(linera wallet show | grep 'Public Key' | awk '{print $2}')

linera wallet request-chain --faucet https://faucet.testnet-babbage.linera.net --storage $LINERA_STORAGE

TO_CHAIN=$(linera wallet show | grep 'Public Key' | grep -v $FROM_CHAIN | awk '{print $2}')

for i in $(seq 1 200); do
  linera wallet request-chain --faucet https://faucet.testnet-babbage.linera.net --storage $LINERA_STORAGE
done

linera service --port $PORT &

sleep 60

echo "  Transfer from $FROM_CHAIN to $TO_CHAIN ..."

PAYLOAD=$(echo '{
  "query": "mutation {
    transfer(
      chainId: \"'$FROM_CHAIN'\",
      owner: \"0x00\",
      recipient: {
        Account: {
          chain_id: \"'$TO_CHAIN'\",
          owner: \"0x00\"
        }
      },
      amount: \"2.5\"
    )
  }"
}' | tr -d '\000-\037')

time curl -X POST -H 'Content-Type: application/json' -d "$PAYLOAD" http://localhost:$PORT

kill -9 $(ps | grep linera | awk '{print $1}')

[kikakkz]

Today I tested again for 99 chains for 10 times and it's all good (responded a bit longer with 2~3.5 seconds), 😄. And with 100 chains then all validators returned timeout.

[Twey]

A suspect to consider:

linera-views/src/backends/scylla_db.rs
60:const MAX_MULTI_KEYS: usize = 100 - 1;

[kikakkz]

A suspect to consider:

linera-views/src/backends/scylla_db.rs
60:const MAX_MULTI_KEYS: usize = 100 - 1;

Is it for validator side? As I know scylla isn't used in client.

I can do a quick test firstly.

[kikakkz]

Update: I tested with single validator run in docker which I think it used scylla as storage, and it worked well with 100 chains.

[kikakkz]

Update: Limit the cpu cores to 4 with taskset, then run the same test, it worked well with 9 chains.

[kikakkz]

Update: Run with taskset -c 0-3 linera --send-timeout-ms 30000 --recv-timeout-ms 30000 --tokio-threads 40 --tokio-blocking-threads 8 service --port $PORT, 96 chains worked well.

[usagi32]

maybe max_claims_per_chain or max_stream_queries both of these are set to 100 by default
max_claims_per_chain restricts the claims by the faucet
max_stream_queries is being used at some places maybe its getting blocked after 100 chains

[usagi32]

@kikakkz did you try something with my last suggestions?