Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do not search hidden folders / files #2

Closed
sharkdp opened this issue May 9, 2017 · 1 comment
Closed

Do not search hidden folders / files #2

sharkdp opened this issue May 9, 2017 · 1 comment

Comments

@sharkdp
Copy link
Owner

sharkdp commented May 9, 2017

No description provided.

@sharkdp sharkdp mentioned this issue May 9, 2017
Closed
@sharkdp
Copy link
Owner Author

sharkdp commented May 12, 2017

Closed via 12bfee0

@sharkdp sharkdp closed this as completed May 12, 2017
tavianator added a commit to tavianator/fd that referenced this issue Apr 26, 2019
@tavianator
Check the pattern before anything else, since it doesn't require meta…
85526a2 
…data

This should partially address sharkdp#432 by decreasing the number of stat() calls:

    $ strace -c -f ./fd-before '.h$' -j1 /usr -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    100.00    3.700169           3    983938     38022 stat
    $ strace -c -f ./fd-after '.h$' -j1 /usr -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    100.00    0.671723           4    162052     38021 stat

Though it's not as good as possible:

    $ strace -c -f find /usr -name '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    18.75    0.449866           3    136199           newfstatat
    $ strace -c -f bfs /usr -name '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    15.01    0.216024           3     60211         1 statx

Performance is much better when metadata is required:

    $ hyperfine ./fd-{before,after}" '.h$' -j1 /usr -S +1k"
    Benchmark sharkdp#1: ./fd-before '.h$' -j1 /usr -S +1k
      Time (mean ± σ):      2.707 s ±  0.042 s    [User: 890.8 ms, System: 1939.7 ms]
      Range (min … max):    2.659 s …  2.786 s    10 runs

    Benchmark sharkdp#2: ./fd-after '.h$' -j1 /usr -S +1k
      Time (mean ± σ):      1.562 s ±  0.034 s    [User: 726.2 ms, System: 957.9 ms]
      Range (min … max):    1.536 s …  1.648 s    10 runs

    Summary
      './fd-after '.h$' -j1 /usr -S +1k' ran
        1.73 ± 0.05 times faster than './fd-before '.h$' -j1 /usr -S +1k'

While remaining the same when it's not:

    tavianator@graviton $ hyperfine ./fd-{before,after}" '.h$' -j1 /usr"
    Benchmark sharkdp#1: ./fd-before '.h$' -j1 /usr
      Time (mean ± σ):      1.341 s ±  0.016 s    [User: 664.3 ms, System: 761.2 ms]
      Range (min … max):    1.309 s …  1.361 s    10 runs

    Benchmark sharkdp#2: ./fd-after '.h$' -j1 /usr
      Time (mean ± σ):      1.338 s ±  0.012 s    [User: 684.1 ms, System: 741.1 ms]
      Range (min … max):    1.310 s …  1.350 s    10 runs

    Summary
      './fd-after '.h$' -j1 /usr' ran
        1.00 ± 0.02 times faster than './fd-before '.h$' -j1 /usr'
tavianator added a commit to tavianator/fd that referenced this issue Apr 29, 2019
@tavianator
Check the pattern before anything else, since it doesn't require meta…
ef9a5e2 
…data

This should partially address sharkdp#432 by decreasing the number of stat() calls:

    $ strace -c -f ./fd-before '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     15.71    8.831948           7   1192279     46059 stat
    $ strace -c -f ./fd-after '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
      7.92    1.972474          10    183907     46046 stat

Though it's not as few as possible:

    $ strace -c -f find /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     19.01    0.946500           5    161649           newfstatat
    $ strace -c -f bfs /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     13.73    0.406565           5     69005           statx

Performance is much better when metadata is required:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1 -S +1k"
    Benchmark sharkdp#1: ./fd-before '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      4.623 s ±  0.154 s    [User: 1.465 s, System: 3.354 s]
      Range (min … max):    4.327 s …  4.815 s    10 runs

    Benchmark sharkdp#2: ./fd-after '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      2.650 s ±  0.058 s    [User: 1.258 s, System: 1.592 s]
      Range (min … max):    2.568 s …  2.723 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1 -S +1k' ran
        1.74 ± 0.07 times faster than './fd-before '\.h$' /usr -j1 -S +1k'

While remaining the same when it's not:

    $ hyperfine ./fd-{before,after}" '.h$' /usr -j1"
    Benchmark sharkdp#1: ./fd-before '.h$' /usr -j1
      Time (mean ± σ):      2.314 s ±  0.052 s    [User: 1.185 s, System: 1.291 s]
      Range (min … max):    2.260 s …  2.441 s    10 runs

    Benchmark sharkdp#2: ./fd-after '.h$' /usr -j1
      Time (mean ± σ):      2.316 s ±  0.040 s    [User: 1.162 s, System: 1.315 s]
      Range (min … max):    2.263 s …  2.375 s    10 runs

    Summary
      './fd-before '.h$' /usr -j1' ran
        1.00 ± 0.03 times faster than './fd-after '.h$' /usr -j1'
tavianator added a commit to tavianator/fd that referenced this issue Apr 30, 2019
@tavianator
Check the pattern before anything else, since it doesn't require meta…
33788ef 
…data

This should partially address sharkdp#432 by decreasing the number of stat() calls:

    $ strace -c -f ./fd-before '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     15.71    8.831948           7   1192279     46059 stat
    $ strace -c -f ./fd-after '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
      7.92    1.972474          10    183907     46046 stat

Though it's not as few as possible:

    $ strace -c -f find /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     19.01    0.946500           5    161649           newfstatat
    $ strace -c -f bfs /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     13.73    0.406565           5     69005           statx

Performance is much better when metadata is required:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1 -S +1k"
    Benchmark sharkdp#1: ./fd-before '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      4.623 s ±  0.154 s    [User: 1.465 s, System: 3.354 s]
      Range (min … max):    4.327 s …  4.815 s    10 runs

    Benchmark sharkdp#2: ./fd-after '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      2.650 s ±  0.058 s    [User: 1.258 s, System: 1.592 s]
      Range (min … max):    2.568 s …  2.723 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1 -S +1k' ran
        1.74 ± 0.07 times faster than './fd-before '\.h$' /usr -j1 -S +1k'

While remaining the same when it's not:

    $ hyperfine ./fd-{before,after}" '.h$' /usr -j1"
    Benchmark sharkdp#1: ./fd-before '.h$' /usr -j1
      Time (mean ± σ):      2.314 s ±  0.052 s    [User: 1.185 s, System: 1.291 s]
      Range (min … max):    2.260 s …  2.441 s    10 runs

    Benchmark sharkdp#2: ./fd-after '.h$' /usr -j1
      Time (mean ± σ):      2.316 s ±  0.040 s    [User: 1.162 s, System: 1.315 s]
      Range (min … max):    2.263 s …  2.375 s    10 runs

    Summary
      './fd-before '.h$' /usr -j1' ran
        1.00 ± 0.03 times faster than './fd-after '.h$' /usr -j1'
tavianator added a commit to tavianator/fd that referenced this issue May 3, 2019
@tavianator
Check the pattern before anything else, since it doesn't require meta…
4cf6af4 
…data

This should partially address sharkdp#432 by decreasing the number of stat() calls:

    $ strace -c -f ./fd-before '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     15.71    8.831948           7   1192279     46059 stat
    $ strace -c -f ./fd-after '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
      7.92    1.972474          10    183907     46046 stat

Though it's not as few as possible:

    $ strace -c -f find /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     19.01    0.946500           5    161649           newfstatat
    $ strace -c -f bfs /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     13.73    0.406565           5     69005           statx

Performance is much better when metadata is required:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1 -S +1k"
    Benchmark sharkdp#1: ./fd-before '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      4.623 s ±  0.154 s    [User: 1.465 s, System: 3.354 s]
      Range (min … max):    4.327 s …  4.815 s    10 runs

    Benchmark sharkdp#2: ./fd-after '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      2.650 s ±  0.058 s    [User: 1.258 s, System: 1.592 s]
      Range (min … max):    2.568 s …  2.723 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1 -S +1k' ran
        1.74 ± 0.07 times faster than './fd-before '\.h$' /usr -j1 -S +1k'

While remaining the same when it's not:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1"
    Benchmark sharkdp#1: ./fd-before '\.h$' /usr -j1
      Time (mean ± σ):      2.382 s ±  0.038 s    [User: 1.221 s, System: 1.286 s]
      Range (min … max):    2.325 s …  2.433 s    10 runs

    Benchmark sharkdp#2: ./fd-after '\.h$' /usr -j1
      Time (mean ± σ):      2.362 s ±  0.034 s    [User: 1.193 s, System: 1.294 s]
      Range (min … max):    2.307 s …  2.422 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1' ran
        1.01 ± 0.02 times faster than './fd-before '\.h$' /usr -j1'
sharkdp pushed a commit that referenced this issue May 8, 2019
@tavianator @sharkdp
Check the pattern before anything else, since it doesn't require meta…
5cbd840 
…data

This should partially address #432 by decreasing the number of stat() calls:

    $ strace -c -f ./fd-before '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     15.71    8.831948           7   1192279     46059 stat
    $ strace -c -f ./fd-after '\.h$' /usr -j1 -S +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
      7.92    1.972474          10    183907     46046 stat

Though it's not as few as possible:

    $ strace -c -f find /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     19.01    0.946500           5    161649           newfstatat
    $ strace -c -f bfs /usr -iname '*.h' -size +1k >/dev/null
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     13.73    0.406565           5     69005           statx

Performance is much better when metadata is required:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1 -S +1k"
    Benchmark #1: ./fd-before '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      4.623 s ±  0.154 s    [User: 1.465 s, System: 3.354 s]
      Range (min … max):    4.327 s …  4.815 s    10 runs

    Benchmark #2: ./fd-after '\.h$' /usr -j1 -S +1k
      Time (mean ± σ):      2.650 s ±  0.058 s    [User: 1.258 s, System: 1.592 s]
      Range (min … max):    2.568 s …  2.723 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1 -S +1k' ran
        1.74 ± 0.07 times faster than './fd-before '\.h$' /usr -j1 -S +1k'

While remaining the same when it's not:

    $ hyperfine ./fd-{before,after}" '\.h$' /usr -j1"
    Benchmark #1: ./fd-before '\.h$' /usr -j1
      Time (mean ± σ):      2.382 s ±  0.038 s    [User: 1.221 s, System: 1.286 s]
      Range (min … max):    2.325 s …  2.433 s    10 runs

    Benchmark #2: ./fd-after '\.h$' /usr -j1
      Time (mean ± σ):      2.362 s ±  0.034 s    [User: 1.193 s, System: 1.294 s]
      Range (min … max):    2.307 s …  2.422 s    10 runs

    Summary
      './fd-after '\.h$' /usr -j1' ran
        1.01 ± 0.02 times faster than './fd-before '\.h$' /usr -j1'
sharkdp added a commit that referenced this issue Sep 15, 2019
@sharkdp
Use jemalloc
4d1aea2 
Benchmark #1: ./fd-sys-alloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     246.8 ms ±   3.4 ms    [User: 960.1 ms, System: 810.0 ms]
  Range (min … max):   244.1 ms … 257.1 ms    12 runs

Benchmark #2: ./fd-jemalloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     201.0 ms ±   3.0 ms    [User: 833.9 ms, System: 666.9 ms]
  Range (min … max):   196.1 ms … 206.9 ms    14 runs

Summary
  './fd-jemalloc '[0-9]\.jpg$' /home/shark' ran
    1.23 ± 0.03 times faster than './fd-sys-alloc '[0-9]\.jpg$' /home/shark'
sharkdp added a commit that referenced this issue Sep 15, 2019
@sharkdp
Use jemalloc
b9138a9 
Benchmark #1: ./fd-sys-alloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     246.8 ms ±   3.4 ms    [User: 960.1 ms, System: 810.0 ms]
  Range (min … max):   244.1 ms … 257.1 ms    12 runs

Benchmark #2: ./fd-jemalloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     201.0 ms ±   3.0 ms    [User: 833.9 ms, System: 666.9 ms]
  Range (min … max):   196.1 ms … 206.9 ms    14 runs

Summary
  './fd-jemalloc '[0-9]\.jpg$' /home/shark' ran
    1.23 ± 0.03 times faster than './fd-sys-alloc '[0-9]\.jpg$' /home/shark'
sharkdp added a commit that referenced this issue Sep 15, 2019
@sharkdp
Use jemalloc
5a15486 
Benchmark #1: ./fd-sys-alloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     246.8 ms ±   3.4 ms    [User: 960.1 ms, System: 810.0 ms]
  Range (min … max):   244.1 ms … 257.1 ms    12 runs

Benchmark #2: ./fd-jemalloc '[0-9]\.jpg$' /home/shark
  Time (mean ± σ):     201.0 ms ±   3.0 ms    [User: 833.9 ms, System: 666.9 ms]
  Range (min … max):   196.1 ms … 206.9 ms    14 runs

Summary
  './fd-jemalloc '[0-9]\.jpg$' /home/shark' ran
    1.23 ± 0.03 times faster than './fd-sys-alloc '[0-9]\.jpg$' /home/shark'
aswild referenced this issue in aswild/fd Jul 10, 2021
@aswild
enable jemalloc only on 64-bit musl
87bafc7 
I don't know why fd was disabling jemalloc on musl, when that's the libc
where jemalloc really helps (about 3.6x faster than the default malloc
on musl 1.2.2). On glibc 2.33 (arch) and 2.28 (buster), jemalloc is only
2-3% faster than the default allocator, which isn't worth the extra
compile time.

Hyperfine runs searching in a linux kernel checkout where the Arch
5.12.15 config has been compiled on an i7-7700K running at stock speeds.

glibc:
    Benchmark #1: ./fd.gnu-default asdfasdfasdfasdf /workspace/linux-mainline
      Time (mean ± σ):      83.4 ms ±   2.9 ms    [User: 506.0 ms, System: 106.3 ms]
      Range (min … max):    79.8 ms … 111.6 ms    500 runs

    Benchmark #2: ./fd.gnu-jemalloc asdfasdfasdfasdf /workspace/linux-mainline
      Time (mean ± σ):      81.1 ms ±   3.1 ms    [User: 483.4 ms, System: 103.5 ms]
      Range (min … max):    77.6 ms … 118.3 ms    500 runs

    Summary
      './fd.gnu-jemalloc asdfasdfasdfasdf /workspace/linux-mainline' ran
	1.03 ± 0.05 times faster than './fd.gnu-default asdfasdfasdfasdf /workspace/linux-mainline'

musl:
    Benchmark #1: ./fd.musl-default asdfasdfasdfasdf /workspace/linux-mainline
      Time (mean ± σ):     274.0 ms ±   3.8 ms    [User: 2.040 s, System: 0.069 s]
      Range (min … max):   266.7 ms … 305.9 ms    500 runs

    Benchmark #2: ./fd.musl-jemalloc asdfasdfasdfasdf /workspace/linux-mainline
      Time (mean ± σ):      76.1 ms ±   3.8 ms    [User: 480.4 ms, System: 71.6 ms]
      Range (min … max):    72.1 ms … 112.1 ms    500 runs

    Summary
      './fd.musl-jemalloc asdfasdfasdfasdf /workspace/linux-mainline' ran
	3.60 ± 0.18 times faster than './fd.musl-default asdfasdfasdfasdf /workspace/linux-mainline'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sharkdp