Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use fine grained locks to enhance performance on large multi-cores #160

Merged
merged 12 commits into from
Jan 13, 2022

Conversation

gdanezis
Copy link
Collaborator

@gdanezis gdanezis commented Jan 12, 2022

This addresses issue #158 .

How is this PR addressing the issue of the Mutex contention in the authority store:

  • As a reminder, the issue was that a mutex was used to ensure that the read-then-modify operations on the database were atomic. In particular locks associated with objects need to be checked as a pre-condition for an order or certificate effects to be written to the database. This was done through a single Mutex lock, that saw a lot of contention, leading to performance of around 3K tps on our bench when using 72 cores (very bad, and anout 1/3 of when using 4-8 cores).
  • The solution is to use a fixed size array of N Mutexes. Each object is mapped to a mutex from the array (via its digest -- we need to randomize this to avoid DoS) therefore different objects only 1/N chance of conflicting with the mutex for another object. Sorting and de-duplicating the list of object mutexes on which a write has to block avoids deadlocks.

New performance on the c5n.metal AWS machine (72 logical cores, 3.888 USD per Hour):

$ ./target/release/bench --cpus 72 --db-dir /data/dbs --num-accounts 144000 --max-in-flight 72000
Total time: 3751240us, items: 144000, tx/sec: 38387.30659728516

fastpay/src/bench.rs Show resolved Hide resolved
fastpay_core/src/authority/authority_store.rs Outdated Show resolved Hide resolved
fastpay_core/src/authority/authority_store.rs Show resolved Hide resolved
Copy link
Contributor

@huitseeker huitseeker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super duper nice work, thanks!

@gdanezis
Copy link
Collaborator Author

Many thanks @huitseeker rebased, and waiting for tests to pass. Then landing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[fastx linux perf] Authority Store Mutex leads to poor performance on Linux multi-core
2 participants