Refactor internal transactions in `Node Broker`

[CyberROFL]

• Эксперимент с регистрацией нод
• Надо перейти на обычные транзакции без ограничений на inflight
• Разделить состояние в памяти на Uncommited (что ещё коммитим, и на котором основываем Execute транзакций) и Committed (то, что уже закоммичено, и из которого мы можем отвечать на запросы без запуска транзакций)
• Поменять транзакции, чтобы Execute использовал Uncommited, а Complete использовал Committed
• Транзакций 8
• Тесты с блокировкой стораджа

Changelog entry

Speed up the startup of dynamic nodes in the cluster.

Changelog category

• Performance improvement

[pixcc]

Benchmarking

Setup

• 6 baremetal cpu128_soc2_mem512G_net25G_4nvme hosts
• storage groups are mirror-3-dc (3 nodes), but every node physically in the same dc
• 6 storage nodes evenly run on the hosts
• 100 database nodes evenly run on the hosts
• actor system config is taken from the largest internal production cluster

  sys:
    executors:
      system:
        threads: 9
        spin_threshold: 1
      user:
        spin_threshold: 1
        threads: 16
      batch: 
        threads: 7
        spin_threshold: 1
      io:
        threads: 1
      ic:
        threads: 3
        max_threads: 5
        max_avg_ping_deviation: 500
        spin_threshold: 10
        time_per_mailbox_micro_secs: 100
        priority: 40
    scheduler:
      resolution: 64
      spin_threshold: 0
      progress_threshold: 10000
  

Workload description

The workload imitates the registration of new nodes with a parallelism of 1000.

Each of the 100 dynamic nodes in 10 threads synchronously registers 10 new database nodes using gRPC requests without timeout. Each thread randomly selects one of the 6 hosts as the endpoint before every gRPC request.

The workload was repeated 30 times.

Important observations

• The first iteration of the workload results in a latency spike, regardless of changes. Since this happens only for the first iteration and does not repeat thereafter, it can be counted as an warmup outlier.
• The client latency of node registration is highly dependent on which node receives the gRPC request. VIEWER, NAMESERVICE, and IC CPU consumption was increased on 2 of the 6 nodes. The consumption scorelly increased with the total number of nodes in the cluster (30 000 in the end of benchmark). The hypothesis is that VIEWER is trying to find out the state of all nodes and is sending requests to them. When a registration request hit these 2 out of 6 nodes, client latency degraded. In the benchmarking performed, the NodeBroker tablet was not run on these 2 unusual nodes.

Before the changes:

NodeBroker-side total latency

With outlier

Without outlier

NodeBroker commit latency

Client-side total latency

After the changes:

NodeBroker-side total latency

p50 is under represented because its often hit the bucket "less than 1 millisecond"

With outlier

Without outlier

NodeBroker commit latency

Client-side total latency

Summary

The NodeBroker-side latency of node registration become 13x faster in 99th percentile (41.6 ms -> 2.89 ms). It affects client-side latency even in presence of bad nodes with interfering load that act as a proxy for the request to the NodeBroker. Client-side latency become 4x faster in 50th percentile (30.44 ms -> 7.56 ms), 2.7x faster in 90th (88.58 ms -> 32.56 ms), 1.7x faster in 95th (129.58 ms -> 76.38 ms) and 1.27x faster in 99th (230.27 ms -> 181.22 ms).