Investigate how Iroha operates under load under Ubuntu/Debian

[dima74]

It was reported that:

• Iroha container under Ubuntu restarts after ~10 minutes
• Iroha container under Debian restarts after ~7 hours

Need to investigate it

---

Notes about kura.lock:

• kubernetes perioadically checks if iroha is alive using /health route
• if three checks in a row fails, it restarts the container
• restart is performed non-gracefully, thus iroha doesn't free kura.lock
• after restart iroha can't take kura.lock thus can't start

[dima74]

kura.lock problem should be fixed by https://github.com/soramitsu/iroha-deploy/pull/166

Update: it fixes only problem with genesis peer restart, but still have problem that if iroha is shutdown non-gracefully (e.g. because of OOM), then after restart iroha fails to start

[dima74]

Reproduced the problem, here are some results

• Preconditions:
  • Testing only FindAllAccounts query
  • Using genesis with 2500 accounts
  • Using iroha version 50e8104
  • Using iroha2-perf load generator version 44b2499839ee6a6851dd9f39bb5bfa8d02b4fa6c
  • Testing is performed locally, results in the docker/k8s should be similar
• Observed problem that iroha consumes a lot of memory even under small load:
  • 10 queries per second ~2.6GB
  • 20 queries per second ~6.6GB
  • 30 queries per second ~11.3GB
• Background: how queries are handled
  • Consider iroha received query FindAllAccounts. It executes it and finds all 2500 accounts. Then it splits response to batches (default batch size is 10), returns first batch to the client, and saves remaining batches to LiveQueryStore. When client requests second batch, it is removed from LiveQueryStore. Normally client requests all batches, so after it query data will be removed from LiveQueryStore. However if client doesn't request batches, LiveQueryStore has special pruning mechanism - every 30 seconds it removes old queries (inactive more then 30 seconds).
• Background: how load generator works
  • When making query request in the load generator, currently only first batch is requested.
• So iroha uses a lot of memory in such scenario because it stores all queries results for last 30 seconds in LiveQueryStore
• So:
  • I think load generator should be modified to request the whole data, not only the first batch (either retrieve all batches in multiple requests, or increase batch size)
  • Iroha can be easily DDOSed by requesting only first batch of heavy query. This might be a security problem

[mversic]

Iroha can be easily DDOSed by requesting only first batch of heavy query. This might be a security problem

this is not acceptable. We should at least have some limit to the number of live queries to prevent OOM

[mversic]

Queue already has some form of DDOS implemented. There is global limit and per user limit on the number of transactions that can be in the queue. We should at least do something like that

[DCNick3]

Pruning can also lead to DDoS, because non-malicious actors wouldn't be able to complete their queries as the store is being constantly thrashed. Maybe use the fact that queries are signed and add a per-AccountId limit?

UPD: I see that you are proposing a per-used limit too. Sorry, didn't notice that; a global + a per-used limit would be good