refactor: a more friendly dynamic_dispatch bench #332

xiaguan · 2024-04-12T12:07:34Z

What's changed and what's your intention?

Just a little bit enhancement for the bench.

Checklist

I have written the necessary rustdoc comments
I have added the necessary unit tests and integration tests
I have passed make all (or make fast instead if the old tests are not modified) in my local environment.

Related issues or PRs (optional)

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: add foyer bench, reorg workspace Signed-off-by: MrCroxx <mrcroxx@outlook.com> * sort cargo file Signed-off-by: MrCroxx <mrcroxx@outlook.com> * make fmt and clippy happy Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix bug Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* ci: add asan test Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix Cargo.toml and CI Signed-off-by: MrCroxx <mrcroxx@outlook.com> * regen ci Signed-off-by: MrCroxx <mrcroxx@outlook.com> * rename CI step, test asan fail Signed-off-by: MrCroxx <mrcroxx@outlook.com> * regen CI Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* chore: add license checker Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix license checker config Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…x#15) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: introduce instrusive indexers and collections Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: introduce FTL-like storage engine Signed-off-by: MrCroxx <mrcroxx@outlook.com> * update ci Signed-off-by: MrCroxx <mrcroxx@outlook.com> * sort cargo file Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix memory leak Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: enable direct i/o on linux target - enable direct i/o on linux target - refine flusher and reclaimer Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix unit test Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

) * chore: remove unused old storage engien and other components Signed-off-by: MrCroxx <mrcroxx@outlook.com> * update ci Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…xx#31) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: impl storage recovery Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: add segment fifo eviction policy Signed-off-by: MrCroxx <mrcroxx@outlook.com> * export fifo fs store Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: export mods Signed-off-by: MrCroxx <mrcroxx@outlook.com> * make cargo sort happy Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

This reverts commit a8a56cc.

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix: deadlock fix deadlocks: 1. exclusive lock was not `Send` but unsafe impled. 2. submit flush task await blocks `set_region_evictable` 3. no new reclamation task created after `set_region_evctable` if all write process are waiting Signed-off-by: MrCroxx <mrcroxx@outlook.com> * update ci Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* fix: gracefullly shutdown runners Use `Store::shutdown_runners()` to gracefully shutdown runners.` Signed-off-by: MrCroxx <mrcroxx@outlook.com> * log warn if send failed instead of panic Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

- fix region advance - fix recovery slice destroy - seal last dirty region when shutdown Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: introduce prometheus metrics Signed-off-by: MrCroxx <mrcroxx@outlook.com> * calc metrics Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix registry metrics Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* feat: add eviction alg hit ratio bench * refactor: use enum cache instead of generic cache The enum cache is more easy to use since it only has two generic parameters. The generic cache is more like an internal interface. Signed-off-by: xiaguan <751080330@qq.com> * chore : sort foyer_memory's dependencies Signed-off-by: xiaguan <751080330@qq.com> * chore: use hakari to gen better dependencies Signed-off-by: xiaguan <751080330@qq.com> --------- Signed-off-by: xiaguan <751080330@qq.com>

* chore: update license Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: fix license checker Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: update license header Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* Add S3Fifo eviction for memory * fix: refine s3fifo, fix some bugs Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: fix license Signed-off-by: MrCroxx <mrcroxx@outlook.com> * refactor: expose s3fifo, fix hakari, add s3fifo fuzzy test Signed-off-by: MrCroxx <mrcroxx@outlook.com> * bench: add s3fifo to hit ratio bench Signed-off-by: MrCroxx <mrcroxx@outlook.com> * test: add s3fifo uts Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com> Co-authored-by: MrCroxx <mrcroxx@outlook.com>

) * chore: bump foyer-memory to 0.1.3 Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: bump foyer-intrusive to 0.3.1 Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…roxx#314) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…x#315) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* refactor: remove usage of unstable features Signed-off-by: MrCroxx <mrcroxx@outlook.com> * refactor: some more Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

…roxx#317) * refactor: use crate allocator_api2 to make foyer build on stable Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: use stable pipeline for ci tests, use nightly for sanitizer Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: update ci prefix key Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: try fix ci Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: update ci nightly version Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: try resolve github issue Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* test: run CI on both linux and macos Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: fix build on macos Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix: try fix ci run on targets Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* doc: update README and add a simple example Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: update Cargo.toml Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* chore: fix ci coverage Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: fix check Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

foyer-memory/benches/bench_dynamic_dispatch.rs

MrCroxx · 2024-04-12T12:37:25Z

foyer-memory/benches/bench_dynamic_dispatch.rs

+    // Warmup
+    for _ in 0..WARMUP_LOOPS {
+        let s = rand_string(thread_rng().gen_range(0..100));
+        let _ = (t.f2)(&s);
+    }


Any reason to add a warmup loop for this case? 👀

When the iteration number equals 10,000, there is significant performance variability on my machine.

With count the random generation or not? I think the variability may come from that. 🤔

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Signed-off-by: xiaguan <751080330@qq.com>

MrCroxx added 30 commits May 24, 2023 15:31

feat: introduce read only file store (MrCroxx#9)

24cccaf

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: add simple test for read only file store and fix bugs (MrCroxx#10)

7a2467c

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: fix bug with bench (MrCroxx#12)

3e9cc5f

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: add license checker (MrCroxx#14)

5c09e60

* chore: add license checker Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix license checker config Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: wrapper cache type, reorg utils, add metrics framework (MrCrox…

e38ad6a

…x#15) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: fix read only store recovery, impl random drop (MrCroxx#16)

f44aec1

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: add metrics (MrCroxx#17)

8e80c6c

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: add log support (MrCroxx#19)

79610aa

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: add write stall config for read only file store (MrCroxx#20)

23e2612

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: introduce instrusive indexers and collections (MrCroxx#21)

506f3ef

* feat: introduce instrusive indexers and collections Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: remove unused deps (MrCroxx#28)

64b72b4

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: foyer storage bench support flexible writers and readers (MrCro…

85d8341

…xx#31) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: impl storage recovery (MrCroxx#32)

7c81272

* feat: impl storage recovery Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: introduce segment fifo eviction policy (MrCroxx#35)

ad3f125

* feat: add segment fifo eviction policy Signed-off-by: MrCroxx <mrcroxx@outlook.com> * export fifo fs store Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: export mods (MrCroxx#37)

98a0af7

* feat: export mods Signed-off-by: MrCroxx <mrcroxx@outlook.com> * make cargo sort happy Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: export extern crate (MrCroxx#38)

a8a56cc

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

Revert "fix: export extern crate (MrCroxx#38)" (MrCroxx#39)

09bf57a

This reverts commit a8a56cc.

fix: disable O_DIRECT on non-linux targets (MrCroxx#40)

691054a

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: ignore Cargo.lock and bump toolchain (MrCroxx#46)

7fc8542

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: export storage config (MrCroxx#50)

1018bca

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: AdmitAll & ReinsertNone no longer requires KV default (MrCroxx#51)

011a401

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: region advance & recovery & seal (MrCroxx#53)

e61eb5d

- fix region advance - fix recovery slice destroy - seal last dirty region when shutdown Signed-off-by: MrCroxx <mrcroxx@outlook.com>

MrCroxx and others added 24 commits March 14, 2024 10:55

refactor: impl Clone for config for simplification (MrCroxx#293)

4db04a2

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: bump foyer-memory to 0.1.1 (MrCroxx#294)

fabed44

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

refactor: export DefaultCacheEventListener (MrCroxx#296)

050a3d0

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: bump foyer-memory version (MrCroxx#297)

524b645

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: bump foyer-storage to 0.5.1 (MrCroxx#309)

5899ee4

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: fix state() for s3fifo (MrCroxx#312)

b50813b

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: bump foyer-memory to 0.1.4 (MrCroxx#313)

5a410c8

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

refactor: use generic type for cursor instead of associated type (MrC…

823a879

…roxx#314) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

refactor: remove unstable feature let_chains and lint_reasons (MrCrox…

3aa689d

…x#315) Signed-off-by: MrCroxx <mrcroxx@outlook.com>

refactor: remove usage of unstable features (MrCroxx#316)

2693669

* refactor: remove usage of unstable features Signed-off-by: MrCroxx <mrcroxx@outlook.com> * refactor: some more Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

doc: update README and add a simple example (MrCroxx#319)

475aee0

* doc: update README and add a simple example Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: update Cargo.toml Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: add msrv for foyer, update bager in readme (MrCroxx#321)

2abe3df

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

fix: do not truncate when open wal (MrCroxx#322)

cb2616e

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: bump foyer to 0.7.0 (MrCroxx#323)

48c3990

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

refactor: loose query key bound for in-memory cache (MrCroxx#324)

cc64fc9

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

feat: impl contains for in-memory cache (MrCroxx#325)

e47c090

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

perf: add dynamic dispatch bench (MrCroxx#326)

818e3f9

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

chore: fix ci coverage (MrCroxx#331)

3fda8cc

* chore: fix ci coverage Signed-off-by: MrCroxx <mrcroxx@outlook.com> * chore: fix check Signed-off-by: MrCroxx <mrcroxx@outlook.com> --------- Signed-off-by: MrCroxx <mrcroxx@outlook.com>

MrCroxx reviewed Apr 12, 2024

View reviewed changes

refactor: refine generic handle type (MrCroxx#330)

6b820fa

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

xiaguan force-pushed the refactor_dynamic_bench branch from eecb910 to c247346 Compare April 12, 2024 13:22

refactor: a more friendly dynamic_dispatch bench

41b6e07

Signed-off-by: xiaguan <751080330@qq.com>

xiaguan force-pushed the refactor_dynamic_bench branch from c247346 to 41b6e07 Compare April 12, 2024 13:32

MrCroxx force-pushed the main branch from 04d5ac3 to a2b91e3 Compare April 17, 2024 01:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: a more friendly dynamic_dispatch bench #332

refactor: a more friendly dynamic_dispatch bench #332

xiaguan commented Apr 12, 2024

MrCroxx Apr 12, 2024

xiaguan Apr 12, 2024

MrCroxx Apr 12, 2024

xiaguan Apr 12, 2024

refactor: a more friendly dynamic_dispatch bench #332

Are you sure you want to change the base?

refactor: a more friendly dynamic_dispatch bench #332

Conversation

xiaguan commented Apr 12, 2024

What's changed and what's your intention?

Checklist

Related issues or PRs (optional)

MrCroxx Apr 12, 2024

Choose a reason for hiding this comment

xiaguan Apr 12, 2024

Choose a reason for hiding this comment

MrCroxx Apr 12, 2024

Choose a reason for hiding this comment

xiaguan Apr 12, 2024

Choose a reason for hiding this comment