Use jemalloc

[sharkdp]

I keep (compiled) old versions of fd around to find performance regressions.

I noticed that there was a drop in performance between my fd-7.0.0 and the fd-7.1.0 binary. However, when re-compiling these old versions with the current version of Rust (1.37), both of them were slow.

So this had to do something with the version of Rust I was using at that time. The only thing that came to my mind was the change from jemalloc to the system allocator.

Turns out this makes a pretty big difference (23%):

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
./fd-sys-alloc '[0-9]\.jpg$'	246.8 ± 3.4	244.1	257.1	1.2
./fd-jemalloc '[0-9]\.jpg$'	201.0 ± 3.0	196.1	206.9	1.0

This PR re-enables jemalloc for fd.

[sharkdp]

Released in v7.4.0.

[ignatenkobrain]

@sharkdp

1. Can this be made optional?
2. What distribution / glibc do you use?
3. From where do you get Rust, is it coming from distro or official builds?

[sharkdp]

1. Yes, I think so. We could probably introduce a cargo-feature to enable/disable it during build time? Are you asking for packaging reasons?
2. Arch Linux, glibc version:

    > ldd --version 
    ldd (GNU libc) 2.29
   

3. official builds.

[ignatenkobrain]

Yes, I think so. We could probably introduce a cargo-feature to enable/disable it during build time? Are you asking for packaging reasons?

Yep.

---

Is there some easy way to do such benchmark? I would try to check the Fedora builds.

[sharkdp]

Is there some easy way to do such benchmark?

Yes. Build the current master and compare it against a version with these lines removed:

fd/src/main.rs

Lines 38 to 41 in d48aeda

	// We use jemalloc for performance reasons, see https://github.com/sharkdp/fd/pull/481
	#[cfg(all(not(windows), not(target_env = "musl")))]
	#[global_allocator]
	static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;

Benchmark scripts are available at: https://github.com/sharkdp/fd-benchmarks You can use the regression.sh script to compare two versions against each other.

[ignatenkobrain]

Great. I'll do bench on latest Fedora and come back with results.

[ignatenkobrain]

⋊> ~/P/u/f/fd-benchmarks on master ◦ bash regression.sh                 12:18:21

No pattern

Benchmark #1: ./fd-master --hidden --no-ignore '' '/home/brain'
  Time (mean ± σ):      1.227 s ±  0.110 s    [User: 3.647 s, System: 2.237 s]
  Range (min … max):    1.096 s …  1.393 s    10 runs
 
Benchmark #2: ./fd-feature --hidden --no-ignore '' '/home/brain'
  Time (mean ± σ):      1.554 s ±  0.051 s    [User: 4.878 s, System: 3.043 s]
  Range (min … max):    1.494 s …  1.653 s    10 runs
 
Summary
  './fd-master --hidden --no-ignore '' '/home/brain'' ran
    1.27 ± 0.12 times faster than './fd-feature --hidden --no-ignore '' '/home/brain''
Error: The following error occurred while exporting: entity not found

Simple pattern

Benchmark #1: ./fd-master '.*[0-9]\.jpg$' '/home/brain'
  Time (mean ± σ):     735.2 ms ±   9.5 ms    [User: 2.992 s, System: 2.595 s]
  Range (min … max):   724.1 ms … 748.2 ms    10 runs
 
Benchmark #2: ./fd-feature '.*[0-9]\.jpg$' '/home/brain'
  Time (mean ± σ):     910.8 ms ±  74.2 ms    [User: 3.525 s, System: 2.841 s]
  Range (min … max):   834.5 ms … 1093.6 ms    10 runs
 
Summary
  './fd-master '.*[0-9]\.jpg$' '/home/brain'' ran
    1.24 ± 0.10 times faster than './fd-feature '.*[0-9]\.jpg$' '/home/brain''
Error: The following error occurred while exporting: entity not found

Simple pattern (-HI)

Benchmark #1: ./fd-master -HI '.*[0-9]\.jpg$' '/home/brain'
  Time (mean ± σ):     750.3 ms ±   7.4 ms    [User: 3.481 s, System: 2.305 s]
  Range (min … max):   742.0 ms … 764.2 ms    10 runs
 
Benchmark #2: ./fd-feature -HI '.*[0-9]\.jpg$' '/home/brain'
  Time (mean ± σ):     886.9 ms ±  15.0 ms    [User: 3.872 s, System: 2.778 s]
  Range (min … max):   867.4 ms … 908.7 ms    10 runs
 
Summary
  './fd-master -HI '.*[0-9]\.jpg$' '/home/brain'' ran
    1.18 ± 0.02 times faster than './fd-feature -HI '.*[0-9]\.jpg$' '/home/brain''
Error: The following error occurred while exporting: entity not found

File extension

Benchmark #1: ./fd-master -HI --extension jpg '' '/home/brain'
  Time (mean ± σ):     773.1 ms ±   8.0 ms    [User: 3.610 s, System: 2.346 s]
  Range (min … max):   765.4 ms … 785.9 ms    10 runs
 
Benchmark #2: ./fd-feature -HI --extension jpg '' '/home/brain'
  Time (mean ± σ):     911.6 ms ±  19.9 ms    [User: 3.986 s, System: 2.796 s]
  Range (min … max):   885.0 ms … 941.8 ms    10 runs
 
Summary
  './fd-master -HI --extension jpg '' '/home/brain'' ran
    1.18 ± 0.03 times faster than './fd-feature -HI --extension jpg '' '/home/brain''
Error: The following error occurred while exporting: entity not found

File type

Benchmark #1: ./fd-master -HI --type l '' '/home/brain'
  Time (mean ± σ):     751.5 ms ±   4.9 ms    [User: 3.497 s, System: 2.313 s]
  Range (min … max):   745.5 ms … 759.1 ms    10 runs
 
Benchmark #2: ./fd-feature -HI --type l '' '/home/brain'
  Time (mean ± σ):     886.2 ms ±  11.8 ms    [User: 3.880 s, System: 2.797 s]
  Range (min … max):   871.8 ms … 907.9 ms    10 runs
 
Summary
  './fd-master -HI --type l '' '/home/brain'' ran
    1.18 ± 0.02 times faster than './fd-feature -HI --type l '' '/home/brain''
Error: The following error occurred while exporting: entity not found
This script will now ask for your password in order to gain root/sudo
permissions. These are required to reset the harddisk caches in between
benchmark runs.

[sudo] password for brain: 
Okay, acquired superpowers :-)


Cold cache

Benchmark #1: ./fd-master -HI '.*[0-9]\.jpg$' '/home/brain'
  Time (mean ± σ):      3.519 s ±  0.174 s    [User: 1.895 s, System: 4.430 s]
  Range (min … max):    3.338 s …  3.685 s    3 runs
 
Benchmark #2: ./fd-feature -HI '.*[0-9]\.jpg$' '/home/brain'
  Time (mean ± σ):      3.719 s ±  0.023 s    [User: 2.446 s, System: 5.049 s]
  Range (min … max):    3.698 s …  3.744 s    3 runs
 
Summary
  './fd-master -HI '.*[0-9]\.jpg$' '/home/brain'' ran
    1.06 ± 0.05 times faster than './fd-feature -HI '.*[0-9]\.jpg$' '/home/brain''
Error: The following error occurred while exporting: entity not found

where fd-master is the master branch and fd-feature is without jemalloc.

[sharkdp]

Thank you very much for the feedback!

It looks like you can reproduce my results (10%-30% speedup when using jemalloc).