Extremely high latencies when using measurement.interval=both to address coordinated omission

[seybi87]

Hi folks,

we are running latency focused benchmarks (YCSB Client 0.18.0-SNAPSHOT) with a target throughput and using the measurement.interval=both as described in CHANGES.md

But the reported [Intended-READ] latencies seem to be odd for some databases, resulting in over 100s latencies. See below an example for ScyllaDB (same goes for other tested databases such as Couchbase, PostgreSQL with JBDC, ...)

Command line: -db site.ycsb.db.scylla.ScyllaCQLClient -s -p scylla.hosts=100.20.60.80,54.71.135.219,52.26.243.220 -p scylla.keyspace=ycsb -p scylla.username=scylla -p scylla.password=3iaq1MWzBf2AbEv -p scylla.writeconsistencylevel=QUORUM -p scylla.readconsistencylevel=QUORUM -p workload=site.ycsb.workloads.CoreWorkload -p maxexecutiontime=1800 -threads 400 -p recordcount=1000000000 -p operationcount=999900000 -target 117000 -p measurement.interval=both -p fieldcount=10 -p fieldlength=100 -p requestdistribution=uniform -p insertorder=ordered -p readproportion=0.95 -p updateproportion=0.05 -p insertproportion=0.0 -p scanproportion=0.0 -p maxscanlength=1000 -p scanlengthdistribution=uniform -t

[READ], Operations, 196890273
[READ], AverageLatency(us), 3573.3141977714663
[READ], MinLatency(us), 441
[READ], MaxLatency(us), 216319
[READ], 95thPercentileLatency(us), 6619
[READ], 99thPercentileLatency(us), 9263
[Intended-READ], Operations, 196890273
[Intended-READ], AverageLatency(us), 6.311351787929676E7
[Intended-READ], MinLatency(us), 67584
[Intended-READ], MaxLatency(us), 112459775
[Intended-READ], 95thPercentileLatency(us), 106364927
[Intended-READ], 99thPercentileLatency(us), 108920831

Using less threads (10/50/100) results in comparably high latencies for [Intended-READ]

Running the identical workload (with a lower target throughput) against MongoDB Atlas results in reasonable latencies:

Command line: -db site.ycsb.db.MongoDbClient -s -p mongodb.url=mongodb://root:pCm9YZpSAres@mongodb-m-social-co-shard-00-00.ll86m.mongodb.net:27016,mongodb-m-social-co-shard-00-01.ll86m.mongodb.net:27016,mongodb-m-social-co-shard-00-02.ll86m.mongodb.net:27016,mongodb-m-social-co-shard-01-00.ll86m.mongodb.net:27016,mongodb-m-social-co-shard-01-01.ll86m.mongodb.net:27016,mongodb-m-social-co-shard-01-02.ll86m.mongodb.net:27016,mongodb-m-social-co-shard-02-00.ll86m.mongodb.net:27016,mongodb-m-social-co-shard-02-01.ll86m.mongodb.net:27016,mongodb-m-social-co-shard-02-02.ll86m.mongodb.net:27016/?ssl=true&authSource=admin&maxPoolSize=400&w=majority&j=true&readPreference=primary -p batchsize=1 -p workload=site.ycsb.workloads.CoreWorkload -p maxexecutiontime=1800 -threads 400 -p recordcount=1000000000 -p operationcount=999900000 -target 31000 -p measurement.interval=both -p fieldcount=10 -p fieldlength=100 -p requestdistribution=uniform -p insertorder=ordered -p readproportion=0.95 -p updateproportion=0.05 -p insertproportion=0.0 -p scanproportion=0.0 -p maxscanlength=1000 -p scanlengthdistribution=uniform -t

[READ], Operations, 53000501
[READ], AverageLatency(us), 3504.40887875758
[READ], MinLatency(us), 910
[READ], MaxLatency(us), 1605631
[READ], 95thPercentileLatency(us), 7243
[READ], 99thPercentileLatency(us), 13471
[Intended-READ], Operations, 53000501
[Intended-READ], AverageLatency(us), 8938.169294927986
[Intended-READ], MinLatency(us), 975
[Intended-READ], MaxLatency(us), 2416639
[Intended-READ], 95thPercentileLatency(us), 8815
[Intended-READ], 99thPercentileLatency(us), 66687

Any help how to address this issue is much appreciated.