Fish as the leader of fastest shell

[pickfire]

> cat test.fish
for i in (seq 1000)
    command ls > /dev/null
end
> cat test.sh
for i in $(seq 1000) ; do
    ls > /dev/null
done
> time fish test.fish; time mksh test.sh; time bash test.sh
0.62user 1.09system 0:07.28elapsed 23%CPU (0avgtext+0avgdata 4396maxresident)k
0inputs+0outputs (0major+108240minor)pagefaults 0swaps
0.21user 0.62system 0:06.92elapsed 11%CPU (0avgtext+0avgdata 1888maxresident)k
0inputs+0outputs (0major+116674minor)pagefaults 0swaps
0.29user 0.81system 0:07.78elapsed 14%CPU (0avgtext+0avgdata 2780maxresident)k
0inputs+0outputs (0major+145628minor)pagefaults 0swaps

@ridiculousfish Benchmarks are taken on latest git fish with archlinuxarm on raspberry pi 2.

[oranja]

This benchmark is not really fair. Even on the relatively simple Pi-2, I expect the first run to be much slower then the rest, due to caching.

[pickfire]

@oranja, it's not the first run, I took many run (5) and get the best result.

[faho]

There's a few things to note here:

• When comparing fish to other shells, config files are of interest - fish will read all of them, while other shells only read special (presumably stripped-down) files for non-interactive use.

@pickfire: You've cleared your config.fish, so that's not included, but this is something to keep in mind. Though presumably if the end result is the same you don't care how much more work fish did to get there.

• fish has a --profile option that might be of use here.
• Add functions and "snippets" hierarchy #2498 will slow this down when it's used (i.e. when you have snippets) - though none of my attempts have sped it up much
• We're actually doing okay when it comes to this test (i.e. starting and calling a command) - other things are much more in need of optimization (see Example of slowness compared to Bash (while or test problem?) #2007)

[pickfire]

@faho I really hope that fish can be faster than mksh, if you are interested in --profile that I had done:

log.txt

[faho]

@pickfire: Okay, that log shows (sort --numeric-sort -k 2 is your friend here) that 7.5s were spent on the entirety of the for-loop, plus 23.8ms on sourcing /usr/share/fish/config.fish - i.e. about as long as 3 ls-calls. So it's completely dominated by calling ls. (The numbers are probably a bit higher here because the profiling itself isn't free)

So it's actually interesting that mksh manages to be faster.

On my system it's close enough to not matter either way (consistently mksh < fish < bash < zsh, but within 0.08s of each other - this is with extensive fish customization, a medium bashrc and a comparatively tiny zshrc).

[pickfire]

Ah, I didn't know there is /usr/share/fish/config.fish. It is actually for completion, totally worthless.

Yeah, I figured it out, like you said mksh < fish < bash < zsh but not sure why mksh always wins.

@faho And one more thing, mkshrc by default is actually a bit huge which has 600+ lines (comments included).

[faho]

Ah, I didn't know there is /usr/share/fish/config.fish. It is actually for completion, totally worthless.

Not quite - look at the very first non-comment line: set -g IFS \n \t. I don't think you'd want to run a shell without that. Setting up the function path is also rather important.

@faho And one more thing, mkshrc by default is actually a bit huge which has 600+ lines (comments included).

I didn't actually have that because arch only puts it in /etc/skel. Though even after copying it to ~ it doesn't slow mksh down even one centisecond. That's rather impressive. Though of course size doesn't matter (.............) - this file has everything protected from aliasing by prepending "".

And, as we saw earlier, the config file doesn't matter - it barely shows up in fish, even with my (comparatively) extensive customization.

[pickfire]

Woah, now to get the fastest shell title, I guess we really need to be faster than mksh which can doesn't even slow down a centisecond by loading a 600+ lines file.

[ridiculousfish]

Universal variables are also relevant - it's easy to forget about those but they get loaded at startup too.

[pickfire]

@ridiculousfish My universal variable is no more than 25, I don't know if universal variables can be optimized. Well, of course it is better so.

[ghost]

@pickfire If you don't mind, can you post the output of set -U?

[pickfire]

@bucaran Oh I found it a huge now. log.txt

[faho]

@pickfire: That's the short version. You'll want to add "-L" to the options you pass to set - for all we know, these CFLAGS or fish_user_abbreviations could be gigantic. They're probably not, as your times look about right, but something like set -U fish_user_paths $fish_user_paths ~/.local/bin in config.fish would baloon the variable and cause fish to become slower (if only because it'd need to allocate all that memory).

(Also, there's some sort of token in there you might want to censor)

@bucaran: fishtape is yours, right? Any reason why you're keeping all those variable copies around?

[ghost]

@faho The latest fishtape correctly removes those variables.

@pickfire Thanks. Please update fishtape, the lingering universals was caused by a bug in an older version.

[pickfire]

Updated fishtape.

@faho, ah there is some password there. By the way, I didn't know that gigantic $fish_user_paths can make fish slow too.

@bucaran, it seems like the xargs command that I give you didn't worked, it is just updating the index, didn't work. Let me check.

[pickfire]

@ridiculousfish I did another benchmark recently, now including dash and the results for fish is a lot worse. There results are the following (I did three test and get the best result):

dash < mksh < bash < fish

> fish -v
fish, version dedc7f6
> bash --version
GNU bash, version 4.3.42(1)-release (armv7l-unknown-linux-gnueabihf)
> echo $KSH_VERSION
@(#)MIRBSD KSH R52 2016/03/04
> pacman -Qs dash
local/dash 0.5.8-1
> time fish test.fish
0.68user 1.98system 0:09.15elapsed 29%CPU (0avgtext+0avgdata 4712maxresident)k
0inputs+0outputs (0major+109838minor)pagefaults 0swaps
> time bash test.sh
0.27user 1.12system 0:08.86elapsed 15%CPU (0avgtext+0avgdata 2840maxresident)k
0inputs+0outputs (0major+149668minor)pagefaults 0swaps
> time mksh test.sh
0.15user 0.80system 0:07.74elapsed 12%CPU (0avgtext+0avgdata 1900maxresident)k
0inputs+0outputs (0major+119260minor)pagefaults 0swaps
> time dash test.sh
0.10user 0.63system 0:06.98elapsed 10%CPU (0avgtext+0avgdata 1900maxresident)k
0inputs+0outputs (0major+100868minor)pagefaults 0swaps

Even after I reset those fish configuration files, the result are even worse.

[ridiculousfish]

Thanks pickfire. What exactly were the test scripts? What was the contents of the config files? What was the contents of the universal variable file?

[pickfire]

@ridiculousfish The test scripts are the same as the first post on this page.

I had test it with all the lines in the config files removed, it is even slower. So I don't think it will be the case, https://github.com/pickfire/dotfiles/blob/alarm/home/config/fish/config.fish

env.txt, it is a lot longer than before. My password is there too, everyone can help me to play the wargames and earn points for me. :)

[ridiculousfish]

Removing stuff like source $fisher_home/config.fish makes running the script in the first post slower? That's rather suspicious, don't you think?

I still can't reproduce this incidentally.

[pickfire]

Now I still don't get why the results is so different compared to last time. Even mksh surprised me.

[krader1961]

@pickfire: In a previous job working for Sequent Computer Systems one of my duties was dealing with performance issues reported by customers and doing benchmarking. Benchmarking is really, really, hard if you care about reproducibility of the results. Heck, just designing an appropriate benchmark that measures something meaningful is really hard.

In the benchmarking community there are a huge number of micro-benchmarks that people (mostly companies who want to sell you something) use to show that product X is better than Y. They're mostly crap, or at least don't tell a consumer anything useful about how a system will behave under typical conditions.

I modified the simple scripts in your original comment. I replaced the ls command with true. So outside of whatever external commands each shell runs during startup (as well as other things like reading config files) we're measuring just the speed of the parser and execution engine in each shell. Furthermore, I ensured fish wasn't using my personal configuration or universal vars:

mkdir /tmp/bogus
env XDG_CONFIG_HOME=/tmp/bogus XDG_DATA_HOME=/tmp/bogus time fish test.fish

Fish took almost three times longer, both elapsed and cpu user mode time, than bash on OS X, and more than 20x slower than ksh. Profiling showed that all the extra time was in running builtin source /usr/local/share/fish/config.fish. So I did mv /usr/local/share/fish /usr/local/share/fish.old and reran the test. That resulted in fish being indistinguishable from ksh and an order of magnitude faster than bash.

So there might be opportunities to improve the efficiency of the global config.fish file to reduce the cost of starting a new fish shell. Beyond that I don't know what you're trying to accomplish with this issue. Fish is never going to be a speed daemon due to its, by design, far larger dependency on external commands than other shells.

[pickfire]

@krader1961 Yup, I am going to benchmark it with your benchmarking script, I guess it is really important to keep the startup process fast as it not only waste our own time and instead it waste the time of the fishers.

I will benchmark it without my configurations file and variables but /usr/local/share/fish must be included, that's runned by everyone during startup. The file should now be:

for i in (seq 1000)
    command true > /dev/null
end

And will be runned by env XDG_CONFIG_HOME=/tmp/bogus XDG_DATA_HOME=/tmp/bogus time fish test.fish.

[alphapapa]

Just want to say thanks to all of your for your work on speeding up Fish. :)

[pickfire]

@krader1961 This test is even insane, fish is almost 10x slower compare to other shells.

> cat test.fish
for i in (seq 1000)
    command true > /dev/null
end
> cat test.sh
for i in $(seq 1000) ; do
    true > /dev/null
done
> env XDG_CONFIG_HOME=/tmp/bogus XDG_DATA_HOME=/tmp/bogus time fish test.fish
0.67user 2.01system 0:07.28elapsed 36%CPU (0avgtext+0avgdata 4548maxresident)k
0inputs+0outputs (0major+83066minor)pagefaults 0swaps
> time bash test.sh
0.13user 0.04system 0:00.18elapsed 91%CPU (0avgtext+0avgdata 2820maxresident)k
0inputs+0outputs (0major+312minor)pagefaults 0swaps
> time mksh test.sh
0.04user 0.03system 0:00.08elapsed 85%CPU (0avgtext+0avgdata 1528maxresident)k
0inputs+0outputs (0major+202minor)pagefaults 0swaps
> time dash test.sh
0.03user 0.01system 0:00.05elapsed 74%CPU (0avgtext+0avgdata 1744maxresident)k
0inputs+0outputs (0major+164minor)pagefaults 0swaps

The results are still the same as above, dash < mksh < bash < fish

[krader1961]

@pickfire: Why did you modify my script? Specifically, why did you change true to command true > /dev/null? That will cause fish to run the external /usr/bin/true (or perhaps /bin/true). Whereas in the other shells it's going to be a builtin. You're not comparing apples to apples. And drop the redirection to > /dev/null from both scripts. The true command doesn't produce any output and the redirection is just noise.

[pickfire]

@krader1961 Fish is still the slowest if so.

> env XDG_CONFIG_HOME=/tmp/bogus XDG_DATA_HOME=/tmp/bogus time fish test.fish
0.27user 0.02system 0:00.30elapsed 95%CPU (0avgtext+0avgdata 4664maxresident)k
0inputs+0outputs (0major+1096minor)pagefaults 0swaps

[pickfire]

Even after dropping redirection, fish is still the slowest too:

> cat test.sh
for i in $(seq 1000) ; do
    true
done
> cat test.fish
for i in (seq 1000)
    true
end
> env XDG_CONFIG_HOME=/tmp/bogus XDG_DATA_HOME=/tmp/bogus time fish test.fish
0.21user 0.04system 0:00.26elapsed 94%CPU (0avgtext+0avgdata 4548maxresident)k
0inputs+0outputs (0major+1103minor)pagefaults 0swaps
> time bash test.sh
0.08user 0.00system 0:00.09elapsed 86%CPU (0avgtext+0avgdata 2824maxresident)k
0inputs+0outputs (0major+310minor)pagefaults 0swaps
> time mksh test.sh
0.02user 0.01system 0:00.03elapsed 76%CPU (0avgtext+0avgdata 1604maxresident)k
0inputs+0outputs (0major+203minor)pagefaults 0swaps
> time dash test.sh
0.01user 0.00system 0:00.02elapsed 43%CPU (0avgtext+0avgdata 1532maxresident)k
0inputs+0outputs (0major+165minor)pagefaults 0swaps

[ridiculousfish]

Yes, fish is the slowest at running true because true is a builtin in other shells, but not fish. If we think this is interesting or important, we could easily make true a builtin in fish too. But I don't think we need to - nobody has asked for it.

@pickfire it's dangerous to report benchmark results without also reporting what exactly we're measuring, and why the results are what they are. People can latch onto them and draw wrong conclusions.

Basically we should be coming at it with the attitude "why does fish perform this way on this particular benchmark?", not "fish is the fastest/slowest and this proves it."

I'd love to have you involved in crafting a real benchmarking suite, with benchmarks for specific areas: startup, external command execution, builtin / function execution, memory usage, and real-world scripts are some that come to mind. Having analogous scripts for other shells could be part of that too. If you'd like to propose some scripts to be part of the benchmark suite, I'll be glad to help review them.

[pickfire]

@ridiculousfish Nice job, I would love it. So should I start the benchmarking suite project in fish-shell? I guess it would be nice but I am actually not very sure about how to do a good benchmarking suite.

[faho]

Yes, fish is the slowest at running true because true is a builtin in other shells, but not fish.

@ridiculousfish: It's in builtin -n, so it should already be a builtin. builtin true also succeeds.

[ridiculousfish]

Oh, so it is! Shows what I know!

[ridiculousfish]

@pickfire Here's one way to go about it:

1. Build a test harness. This is a wrapper script that isolates the environment, runs a benchmark script N times, does the results aggregation (best-of-N or whatever), and then outputs the results in some nice format.
2. Build the actual benchmark scripts. These should either be highly focused (testing just startup, just external command execution, just the test builtin, etc) or realistic, like the git command completion or other real-world scripts.

A fine way to start would be to make a benchmark directory like the test directory, have one simple benchmark in it, and be able to run make benchmark to get benchmark results.

[krader1961]

There are two reasons to run benchmarks:

1. Verify that a change doesn't make the project slower relative to its current implementation.

2. Compare the performance to another implementation.

@pickfire has been focused on the latter. I would argue that the former, avoiding performance regressions, is more valuable at this point in fish's evolution. Our performance relative to comparable shells is "good enough" that there is little reason to invest in improving it. Especially when you consider all the other open issues that are bugs or enhancement requests that would greatly improve the usability of fish.

Once we have a framework that helps us know when we're introducing changes that make performance worse we can worry about improving performance relative to our competition.

[pickfire]

So basically the building blocks are:

directory
|-- Makefile
|-- bench.sh
|-- bench.fish
`-- README.md

The Makefile should be the command for make all (benchmark), make clean (clean up) and make version (show version). I guess I will include dash, bash, fish, mksh (zsh not selected as it is as slow as a sloth).

The bench.sh and bench.fish should be the same as the testing script in above and probably more features like:

• Shell startup time (bash -c exit or similar)
• External command execution (probably some commands like git)
• Testing loops (if, for, while)

Did I miss anything?

[ridiculousfish]

Yes! but let's try to avoid recursive Makefiles, which are inevitably painful.

One way is to structure it roughly like the test directory:

benchmark
|-- driver.fish
|-- bench1.fish
|-- bench1.sh

driver.fish would be the entry point, and would be passed the instance of fish to test, and an optional list of benchmarks to run. (It doesn't have to be a fish script - it could be Python or whatever instead)

Then a top-level Makefile target:

.PHONY benchmark
benchmark:
    ./benchmark/driver.fish ./fish ALL

etc.

[floam]

Ideally the driver would make some attempt to use something better than time if possible. On OS X it's usually dtruss and procsystime I reach for.

[floam]

Huh, don't quite understand what the heck is up with this isChrooted'd/AppleInternal stuff when running pickfires scripts in fish and bash. Is that an artifact of dtrace? (Yes it is)

Full dtruss here: test.txt

Elapsed Times for command /usr/local/bin/bash ftest.sh,

         SYSCALL          TIME (ns)
         getegid                874
         geteuid                880
         getpgrp                887
         getppid                973
          getgid                984
   getdtablesize               1084
       sigreturn               1589
bsdthread_register               1789
shared_region_check_np               1920
       issetugid               2055
          setgid               2206
     sigaltstack               2256
          getuid               2363
          getpid               2368
       proc_info               3400
   thread_selfid               3731
       getrlimit               4348
           csops               4614
          munmap               4822
            pipe               5912
           wait4               6151
           lseek               8418
            dup2               9433
       sigaction              12486
          setuid              14275
        mprotect              15631
     sigprocmask              16046
  close_nocancel              16761
         fstat64              18991
          sysctl              50879
   read_nocancel              55010
           close              69284
           fcntl              72451
   open_nocancel              90424
            open             104684
            mmap             128540
          stat64             185526
           pread            1558081
           ioctl          121309283
            fork          130389676
            read          144959512
          TOTAL:          399140597

Syscall Counts for command /usr/local/bin/bash ftest.sh,

         SYSCALL              COUNT
bsdthread_register                  1
            dup2                  1
            exit                  1
            fork                  1
   getdtablesize                  1
         getegid                  1
         geteuid                  1
          getgid                  1
         getpgrp                  1
         getppid                  1
          getuid                  1
          munmap                  1
            pipe                  1
       proc_info                  1
          setgid                  1
          setuid                  1
shared_region_check_np                  1
       sigreturn                  1
          getpid                  2
       getrlimit                  2
       issetugid                  2
     sigaltstack                  2
   thread_selfid                  2
           wait4                  2
           csops                  3
           ioctl                  3
          sysctl                  3
  close_nocancel                  6
           fcntl                  6
           lseek                  6
            open                  6
   open_nocancel                  6
           pread                  6
           close                  8
        mprotect                  8
         fstat64                 10
            mmap                 10
       sigaction                 11
   read_nocancel                 12
     sigprocmask                 12
            read                 36
          stat64                 50
          TOTAL:                232

Elapsed Times for command /usr/local/bin/fish test.fish,

         SYSCALL          TIME (ns)
       getrlimit               1539
shared_region_check_np               1746
__pthread_canceled               1831
       issetugid               1945
bsdthread_register               1960
          getpid               2203
           lseek               2285
       proc_info               2993
   thread_selfid               3512
       sigreturn               3855
           csops               4132
          munmap               4197
       fstatfs64               7165
           wait4               9113
           pread               9856
  fcntl_nocancel              10351
        mprotect              14733
         fstat64              27481
       sigaction              29516
            pipe              41471
     getattrlist              49750
            mmap              62749
  close_nocancel              73591
          sysctl              76079
          access              87031
           close             103277
           fcntl             106945
            open             130192
   open_nocancel             149849
     posix_spawn             163600
          stat64             344438
 getdirentries64             693993
   read_nocancel             995082
__pthread_sigmask            1367895
            read            2217242
           ioctl          120801834
          select          121916415
            fork          129911756
          TOTAL:          379433602

Syscall Counts for command /usr/local/bin/fish test.fish,

         SYSCALL              COUNT
bsdthread_register                  1
            fork                  1
       getrlimit                  1
          munmap                  1
     posix_spawn                  1
       proc_info                  1
shared_region_check_np                  1
__pthread_canceled                  2
          getpid                  2
       issetugid                  2
           lseek                  2
       sigreturn                  2
   thread_selfid                  2
           csops                  3
       fstatfs64                  3
           pread                  3
           wait4                  4
  fcntl_nocancel                  5
          sysctl                  5
 getdirentries64                  6
          access                  7
            mmap                  7
     getattrlist                  8
        mprotect                  8
   open_nocancel                 11
            pipe                 11
            open                 13
  close_nocancel                 14
          select                 14
         fstat64                 16
           ioctl                 16
   read_nocancel                 21
           close                 31
       sigaction                 41
           fcntl                 71
          stat64                101
            read                148
__pthread_sigmask               1983
          TOTAL:               2569

I was a bit disappointed that the number of syscalls was so much higher than bash, that it performs so much slower. I had thought it was getting better! :( . It clearly lost this test...

Until I ran it again, except with the option that shows the timings also so I could at least show off the tool...

They actually performed nearly exactly the same when you compare the time on the clock. Goes to show that this stuff is HARD. Complicated systems and lots of data can sometimes be more harmful than helpful if are not very carful.

[pickfire]

@ridiculousfish Of course! No recursive Makefile.

@floam Woah, that is some stuff that I don't know about.

I would like to ask one question, what is actually the driver that both of you are talking about?

[ridiculousfish]

driver is the test harness, which I sketched as step 1 in this comment.

[krader1961]

Thanks, @pickfire, for making us think about this. However, I'm going to close it because it is too broad. Issue #2007 is an example that is sufficiently precise to allow us to know when the issue has been addressed. Making fish fast is a desirable goal but is so open ended as to be useless as an actionable issue.