Runtime performance analysis

[tillrohrmann]

Some points to consider:

Tasks

Beta Give feedback

1. concurrency limits - I think that’s 1k at the moment, I think that it: pretty conservative (should be 1 million imho)
2. queue sizes - overall it’s ok for the queues to be in the MBs (32, 64, 128..) preallocated ring buffers.
3. Logging could relax a bit
4. Jemalloc sounds like a good idea, but I don’t have a lot of knowledge about it. It didn’t help with a micro benchmark, but probably it is something the manifests with a very long tests.
5. Rocksdb currently uses the WAL, which means that every write goes trough fsync. Hopefully that once we’ll switch to the Raft based impl, we will put way less stuff into the WAL (just the command vs all the side effects) Could be interesting just to measure the effect of not WALing.
6. Generally rocksdb - so far the workload fits in memory, let’s see where do we spend the time. Perhaps tweaking some septic CFs settings, might help us. For example the status CF could use a Hash based memtable to better optimize for point lookups.
7. Investigate potential optimization for key extraction #99
   performance +1
8. Skip parsing results of completable journal entry #94
   enhancement partition-processor performance +3
9. Consider using zerovec for performance sensitive ser/de code #230
   performance +1
10. Use Jemalloc/mimalloc as memory allocator #303
    performance +1
11. Add benchmarks for measuring throughput #560
    +0
    
12. Heavy contention on spawn_blocking lock #562
    performance +1
    
13. Remove unnecessary format in Consensus module #567
    +0
    
14. SegmentedQueue might enter allocate/deallocate cycle when being close to empty #523
    performance +1
    
15. InvocationTask::append_path could be optimized removing the slice allocation #612
    invoker performance +2
    
16. Memoize ServiceInvocationId Hash in Invoker #611
    invoker performance +2
17. Presize serialization buffers in Codec depending on type #735
    +0
    
18. Make Storage::Transcation a GAT so that it can be implemented by borrowing Storage #737
    +0
    
19. ServiceInvocationResponseSink::Ingress SocketAddr is costly for serde #1020
    +0
    
20. Reorder deduplication logic for opportunity to batch on sender side #1035
    +0
    
21. Support read-your-own-writes in the PartitionProcessor to support batch command applications #1037
    +0
    
22. Avoid transactional RocksDB access when only reading #1039
    +0
    
23. Share key_buffer and value_buffer across transactions #1042
    +0
    
24. Shuffle w/o excessive scanning of RocksDB #1044
    +0
    
25. Port to Hyper 1.0 #96
    4 of 6
    admin invoker +2
    

[slinkydeveloper]

Some useful links to setup flamegraphs with benchmarks:

• https://github.com/koute/bytehound
• https://github.com/tikv/pprof-rs
• https://www.jibbow.com/posts/criterion-flamegraphs/

[tillrohrmann]

A tool we could use for more advanced performance tests is: https://hyperfoil.io/