Improve performance with lot of files

[FreddyFunk]

No description provided.

[github-actions]

Benchmark Results 📊⏱️📐

Mode: RANDOM
Total test data size on disk: 866M
Directories: 2000
Files: 4002

Command	Mean [s]	Min [s]	Max [s]	Relative
find ./data ! -empty -type f -exec md5sum {} + | sort | uniq -w32 -dD	1.788 ± 0.001	1.787	1.791	6.00 ± 0.03
ddk_dev -p data	0.298 ± 0.001	0.297	0.300	1.00
ddk -p data	0.300 ± 0.002	0.297	0.303	1.01 ± 0.01

Mode: 1GB
Total test data size on disk: 3.1G
Directories: 0
Files: 3

Command	Mean [s]	Min [s]	Max [s]	Relative
find ./data ! -empty -type f -exec md5sum {} + | sort | uniq -w32 -dD	6.107 ± 0.009	6.102	6.122	10.52 ± 0.06
ddk_dev -p data	0.581 ± 0.002	0.579	0.583	1.00 ± 0.01
ddk -p data	0.581 ± 0.003	0.576	0.585	1.00

Mode: 1MB
Total test data size on disk: 1.1G
Directories: 1110
Files: 1111

Command	Mean [s]	Min [s]	Max [s]	Relative
find ./data ! -empty -type f -exec md5sum {} + | sort | uniq -w32 -dD	2.241 ± 0.004	2.236	2.246	7.95 ± 0.23
ddk_dev -p data	0.282 ± 0.008	0.272	0.293	1.00
ddk -p data	0.940 ± 0.002	0.938	0.943	3.34 ± 0.10

Mode: 1KB
Total test data size on disk: 48M
Directories: 1110
Files: 11110

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
find ./data ! -empty -type f -exec md5sum {} + | sort | uniq -w32 -dD	220.4 ± 1.9	217.8	222.9	1.00
ddk_dev -p data	269.3 ± 1.2	268.5	271.5	1.22 ± 0.01
ddk -p data	7832.3 ± 36.4	7792.4	7876.4	35.54 ± 0.35

[codecov]

Codecov Report

Base: 98.59% // Head: 98.58% // Decreases project coverage by -0.01% ⚠️

Coverage data is based on head (f6efd1e) compared to base (eadb4b6).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##              dev      #56      +/-   ##
==========================================
- Coverage   98.59%   98.58%   -0.01%     
==========================================
  Files           4        4              
  Lines         285      283       -2     
==========================================
- Hits          281      279       -2     
  Misses          4        4              

Flag	Coverage Δ
unittests	98.58% <100.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/file_system/filter/deduplication.cpp	100.00% <100.00%> (ø)
src/file_system/fsinfo.cpp	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.