Replication is done from periodic fiber

[mkaruza]

First improvement is to implement replication loop to run from periodic fiber.

[mkaruza]

Testing on AWS

Master, replica and dfly_bench were all running on different r6g.xlarge machines. Master and replica node were running with default parameters while dfly_bench was executing benchmark as:

--qps -100 --pipeline 150 -test_time 300 --ratio=1:0 -d 32 --proactor_threads=2 -c 5

There was 2 sequential runs of each variation and between them process was stopped and dump files were deleted.

~ NO REPLICATION (single node) / COMMIT SHA:  cdd1dac394e6f7f80203118e4877ae08bb32ab37 ~

#1

Total time: 5m0.010406326s. Overall number of requests: 45003000, QPS: 150010, P99 lat: 2477.62us
Latency summary, all times are in usec:
Count: 45003000 Average: 1422.3411  StdDev: 331594.18
Min: 126.0000  Median: 1454.9489  Max: 12195.0000

# 2

Total time: 5m0.0104313s. Overall number of requests: 45003000, QPS: 150010, P99 lat: 2484.81us
Latency summary, all times are in usec:
Count: 45003000 Average: 1421.1491  StdDev: 343550.49
Min: 129.0000  Median: 1445.5747  Max: 14969.0000

~ BASELINE REPLICATION / COMMIT SHA:  cdd1dac394e6f7f80203118e4877ae08bb32ab37~

#1

Total time: 5m0.010414144s. Overall number of requests: 45003000, QPS: 150010, P99 lat: 6795.81us
Latency summary, all times are in usec:
Count: 45003000 Average: 4367.2007  StdDev: 751170.53
Min: 127.0000  Median: 4529.2920  Max: 35692.0000

# 2

Total time: 5m0.010429362s. Overall number of requests: 45003000, QPS: 150010, P99 lat: 6782.82us
Latency summary, all times are in usec:
Count: 45003000 Average: 4249.0597  StdDev: 740713.29
Min: 127.0000  Median: 4417.1392  Max: 26482.0000

~ FIBER REPLICATION / COMMIT SHA:  95b13eb0513cb400fe6b8ddf7933f3b35b17d917 ~

# 1

Total time: 5m0.010410131s. Overall number of requests: 45003000, QPS: 150010, P99 lat: 2962.03us
Latency summary, all times are in usec:
Count: 45003000 Average: 2029.7570  StdDev: 284920.38
Min: 123.0000  Median: 2043.6726  Max: 8950.0000

# 2

Total time: 5m0.010418991s. Overall number of requests: 45003000, QPS: 150010, P99 lat: 3270.76us
Latency summary, all times are in usec:
Count: 45003000 Average: 2189.9105  StdDev: 312585.28
Min: 115.0000  Median: 2202.5841  Max: 14640.0000

MainLoop PROFILING

replication_profiling.zip

	MainLoop
NO REPLICATION (1)	5.91%
NO REPLICATION (2)	8.34%
BASELINE REPLICATION (1)	18.69%
BASELINE REPLICATION (2)	21.91%
FIBER REPLICATION (1)	11.48%
FIBER REPLICATION (2)	9.28%

[mkaruza]

After testing with another implementation where replication is done form JournalStreamer::Write where data is replicated after reaching threshold and periodic fiber is used only for dispatching stalled non replicate data results are:

~ THRESHOLD REPLICATION / COMMIT SHA:  86f57e8e8c22180d147fb591bef6b5385cfc007e ~

#1

Total time: 5m0.010412159s. Overall number of requests: 45003000, QPS: 150010, P99 lat: 3468.47us
Latency summary, all times are in usec:
Count: 45003000 Average: 2530.1405  StdDev: 310953.58
Min: 205.0000  Median: 2565.8457  Max: 6911.0000

# 2

Total time: 5m0.010408689s. Overall number of requests: 45003000, QPS: 150010, P99 lat: 3505.76us
Latency summary, all times are in usec:
Count: 45003000 Average: 2677.1159  StdDev: 325400.01
Min: 201.0000  Median: 2718.4549  Max: 6302.0000

# 3

Total time: 5m0.010409073s. Overall number of requests: 45003000, QPS: 150010, P99 lat: 3472.67us
Latency summary, all times are in usec:
Count: 45003000 Average: 2549.4679  StdDev: 313965.66
Min: 198.0000  Median: 2589.7666  Max: 5622.0000

	MainLoop
THRESHOLD REPLICATION (1)	19.66%
THRESHOLD REPLICATION (2)	9.69 %
THRESHOLD REPLICATION (3)	19.52 %

• It is unclear why second run has significant lower MainLoop value