Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Compile ~/.zcompdump when it has been modified, for another minor speed up #316

Closed
wants to merge 6 commits into
from

Conversation

Projects
None yet
6 participants
Contributor

oknowton commented Apr 30, 2011

Things are a getting a bit too fast to consistently benchmark now...

I ran 20 consecutive benchmarks with and without this commit. Without this change zsh takes between 0.11 and 0.15 seconds to start up. With this change it clocks between 0.09 and 0.13 seconds.

When something causes ~/.zcompdump to change the startup time for me is about 0.45 seconds. This shouldn't happen very often, I think only when something in the fpath changes.

Contributor

sorin-ionescu commented Apr 30, 2011

What about executing zcompile on all .zsh files?

Contributor

oknowton commented Apr 30, 2011 edited by mcornella

I tried that. There was no measurable improvement on my system.

sorin-ionescu reply@reply.github.com wrote:

What about executing zcompile on all .zsh files?

Contributor

sorin-ionescu commented May 1, 2011

Always quote paths.

# Compile zcompdump, if modified, to increase startup speed.
if [ "$HOME/.zcompdump" -nt "$HOME/.zcompdump.zwc" ]; then
  zcompile "$HOME/.zcompdump"
fi
Contributor

oknowton commented May 1, 2011

On Sun, 1 May 2011 09:22:26 -0700
sorin-ionescu
reply@reply.github.com
wrote:

Always quote paths.

# Compile zcompdump if modified for speed.
if [ "$HOME/.zcompdump" -nt "$HOME/.zcompdump.zwc" ]; then
  zcompile "$HOME/.zcompdump"
fi

I fixed all of the quoting that I could... zcompile fails to write
the file if it is quoted. I also fixed a bug in my logic. I thought I
tested it, but it wasn't compiling zcompdump unless a .zwc file
already existed.

I did also go back and have another shot at recompiling every zsh
script in oh-my-zsh as well as my .zshrc and whatnot. If it made
an improvement it wasn't big enough to detect with /usr/bin/time's
granularity.

I'm assuming I see a measurable difference when compiling the compdump
file because it is so much bigger than every other script (about 32KB
uncompiled, 71KB compiled for me).

I'd be interested to see what sort of difference compiling the compdump
makes on slower machines. The slowest machine I have access to around
here is a dual core Athlon X2 3800. Realistically I'd be curious how
it does on something like a low end atom cpu in a netbook.

Lets see if a crappy table displays reasonably well:

                       uncompiled    compiled                   

dual core Athlon X2 3800 0.17 0.14
quad core mobile Core i7 0.12 0.14
quad core Xeon E5405 0.06 0.05

This is probably the last speed up I am going to look for. I ran the
output of zsh -x -i -c exit through a little perl script that put
microsecond differential timestamps on every line, as a sort of poor
man's profiler.

I was able to streamline two small bits of my custom configuration that
were taking up 0.08 seconds combined (keychain and perlbrew setups).
Loading the compdump was the only significant time waster that was
easy, or even possible, to optimize.

Again, I'd love to hear some /usr/bin/time zsh -i -c exit timing
numbers on slower machines, if you have them! Someone has to have a
netbook :).

Pat

Contributor

sorin-ionescu commented May 1, 2011

The idea of using $HOME instead of tilda is quoting weirdness where tilda is not expanded depending on configuration.

Contributor

oknowton commented May 1, 2011

On Sun, 1 May 2011 15:25:54 -0700
sorin-ionescu
reply@reply.github.com
wrote:

The idea of using $HOME instead of tilda is quoting weirdness
where tilda is not expanded depending on configuration.

The majority of the code in oh-my-zsh use the tilde. I've never met
a shell configuration that didn't expand tildes as expected. If this is
actually a problem for people then I'd rather go in and change it
everywhere in the tree instead of adding a piece of code that does
things differently than almost everywhere else.

Pat

Contributor

sorin-ionescu commented May 1, 2011

Your code is broken. Tilda is never expanded inside of double quotes. That is probably what you have experienced with zcompile. My rule is to always use $HOME instead of ~ inside of double quotes and not have to worry. Also, be careful of if [ "any strings" ] since that is always true.

~ ❯ echo "~/.zcompdump"    
~/.zcompdump
~ ❯ echo "$HOME/.zcompdump"
/Users/sorin/.zcompdump
~ ❯ 

As for your if statement, I think what you meant to write is:

# Compile zcompdump, if modified, to increase startup speed.
if [ "$HOME/.zcompdump" -nt "$HOME/.zcompdump.zwc" -o ! -e "$HOME/.zcompdump.zwc" ]; then
    zcompile "$HOME/.zcompdump"
fi
Contributor

oknowton commented May 1, 2011

On Sun, 1 May 2011 15:55:19 -0700
sorin-ionescu
reply@reply.github.com
wrote:

Your code is broken. Tilda is never expanded inside of double quotes.
That is probably what you have experienced with zcompile. My rule
is to always use $HOME instead of ~ inside of double quotes and
not have to worry. Also, be careful of if [ "any strings" ] since
that is always true.

My new code is horribly broken! I didn't even see the goofy quoting
mistake, I just tested removing each/both compdump files and watched
them get properly recreated on the next zsh run. I assumed that it was
functioning correctly and didn't look at it again.

~ ❯ echo "~/.zcompdump"    
~/.zcompdump
~ ❯ echo "$HOME/.zcompdump"
/Users/sorin/.zcompdump
~ ❯ 

As for your if statement, I think what you meant to write is:

# Compile zcompdump, if modified, to increase startup speed.
if [ "$HOME/.zcompdump" -nt "$HOME/.zcompdump.zwc" -o ! -e

"$HOME/.zcompdump.zwc" ]; then zcompile "$HOME/.zcompdump"
fi

I was wrong to assume that the tilde would expand inside the double
quotes. I don't think I've ever run into a case where I've needed to
use quotes in a pathname that had a tilde.

I'm going to remove the quoting. There are no variables in the
pathnames to potentially add spaces. If this is broken for anyone then
their $HOME has a space or something in it and oh-my-zsh has probably
never worked for them.

Pat

Contributor

sorin-ionescu commented May 2, 2011

Dude, just use the code I pasted. It will work in all environments. Stop breaking it. It's just 3 lines of code. There is no need for ego. ZSH is not broken; it uses tilda correctly.

Contributor

oknowton commented May 2, 2011 edited by mcornella

I don't see how keeping consistent with the rest of the tree involves ego.

I'm a fan of consistency. If it were my project I would change it everywhere or not at all.

Contributor

sorin-ionescu commented May 2, 2011

It is safe, to use tilda, unquoted variables, and spaces in double brackets [[ ... ]], but one shouldn't for muscle memory since test and [ do not behave in the same way.

if [[ -e ~/Library/Application\ Support ]]; then
    echo "It exists!"
fi

ZSH is not consistent. Both $HOME and tilda are used. It just depends on whose branch you have and what plugins you have. Writing broken code just because it looks consistent with other code is by no means a good idea. But, to each his own. Needless to say, whatever speeds up ZSH by even a fraction of a second needs to be added. So, I commend you for opening this pull request.

Contributor

oknowton commented May 2, 2011

On Sun, 1 May 2011 17:54:14 -0700
sorin-ionescu
reply@reply.github.com
wrote:

ZSH is not consistent. Both $HOME and tilda are used. It just
depends on whose branch you have and what plugins you have. Writing
broken code just because it looks consistent with other code is by no
means a good idea. But, to each his own.

I only have my code and robbyrussel's code in my fork. As I've been
writing code I have just been running quick greps on the code that is
already committed. I'm horrible I writing shell scripts, and I'm
especially bad at building conditionals. I constantly have to look up
the operators and I often get quoting wrong.

I just figured I'm better off deferring to the maintainer of the
project. If he agrees I'd be happy to take the few minutes it would
take to swap tildes for $HOMEs in the dozen or two places it would need
to happen in the code base.

The code isn't actually "broken." It just doesn't meet your coding
standards. :) As it is right now it is completely functional.

Needless to say, whatever speeds up ZSH by even a fraction of a
second needs to be added. So, I commend you for opening this pull
request.

I wholeheartedly agree! :)

Pat

Contributor

nel commented May 2, 2011

Consistently from 0.21 to 0.20/0.19 on a MBP Core 2 Duo 2.53, quite nice for 3 lines of code :)

Contributor

oknowton commented May 5, 2011

I had another reason to pull my old Fujitsu P2120 out the closet earlier today. I figured while it was out I'd run a few quick benchmarks on it so that we could add it to the list. It's little 933mhz Transmeta Crusoe TM5800 is quite slow. Probably slower than anything most people are running oh-my-zsh on, I hope!

                          uncompiled    compiled

Transmeta TM5800 0.40 0.33
MBP core 2 duo 2.53 0.21 0.20/0.19
dual core Athlon X2 3800 0.17 0.14
quad core mobile Core i7 0.12 0.14
quad core Xeon E5405 0.06 0.05

I also tried zcompiling everything under ~/.oh-my-zsh. I tried both compiling to individual .zwc files and I also tried compiling everything into a single .zwc file. I figured a speed difference might show up on a slower machine like this but I didn't get a measurable speed up.

Pat

@oknowton oknowton closed this May 5, 2011

@oknowton oknowton reopened this May 9, 2011

Contributor

kalos commented Jun 22, 2011

Interesting request... but I prefer split the zcompdump files for multiple hosts:

# Load and run compinit
autoload -U compinit

zsh_cache=$HOME/.zsh/cache
test ! -d $zsh_cache && mkdir -p $zsh_cache

compinit -d $zsh_cache/zcomp-$HOST

if [ "$zsh_cache/zcomp-$HOST" -nt "$zsh_cache/zcomp-$HOST.zwc" -o ! -e "$zsh_cache/zcomp-$HOST.zwc" ]; then
  zcompile "$zsh_cache/zcomp-$HOST"
fi
Owner

robbyrussell commented Aug 31, 2011

@oknowton apologies for the delay here. I'm not seeing that I can cleanly merge this right now. Am happy to give this a whirl if you can get it ready for me.

Contributor

sorin-ionescu commented Sep 2, 2011

A ZSH cold start can take between 1 and 3 seconds depending on its mood. Though, on average, it takes around 1.38.

I have tried the following, but it takes 0.25 to run on my system negating whatever speedup .zwc files provide.

# Load the automatic recompiler.
autoload -Uz zrecompile

# Compile files.
for zsh_file in $HOME/.z{login,logout,profile,shenv,shrc,compdump}(N) $OMZ/*.zsh(N); do
  zrecompile -q -p -U -z "$zsh_file"
done

# Compile function directories.
for (( i=1; i <= $#fpath; i++ )); do
  function_dir="$fpath[i]"
  [[ "$function_dir" == (.|..) ]] \
    || [[ "$function_dir" == (.|..)/* ]] \
    || [[ ! -w "$function_dir:h" ]] && continue
  function_files=($function_dir/^*(\.(rej|orig)|~|\#)(N-.))
  [[ -n "$function_files" ]] \
    && function_files=(${${(M)function_files%/*/*}#/}) \
    && ( cd "$function_dir:h" && zrecompile -q -p -U -z "${function_dir:t}.zwc" "$function_files[@]" ) \
    && fpath[i]="$fpath[i].zwc"
done
unset function_dir
unset function_files

The following numbers are after a cold start.

no compiling: 0.59
zcompdump compiling: 0.55
full compiling: 0.88

Contributor

oknowton commented Sep 2, 2011

On Fri, 2 Sep 2011 10:18:18 -0700
sorin-ionescu
reply@reply.github.com
wrote:

A ZSH cold start can take between 1 and 3 seconds depending
on its mood. Though, on average, it takes around 1.38.

I haven't thought/worried very much about startup time with a cold
cache. I've imagined that this is a rare occurrence for most people.

I can only imagine that the biggest thing slowing down a cold start is
the large number of files that need to be read. If you're lucky, and
your install is fresh, all (or most) of the .zsh files will be in
adjacent blocks and everything will get pulled in in a single
contiguous read.

I'd be surprised if git edits files in place. It probably creates a
new file for every patch and moves it over the top of the old one. If
that really is the case, then every time git updates a file you're
likely adding a single seek. Each seek adds almost 10ms on a 7200 RPM
drive.

I'm traveling and only have an SSD with me (and I can't be dropping
caches on my servers) so I can't test this. You can test this by
moving your .oh-my-zsh directory and copying in back into place, so
that it will be likely to get one contiguous chunk of disk space.

I have tried the following, but it takes 0.25 to run on my system
negating whatever speedup .zwc files provide.

    && ( cd "$function_dir:h" && zrecompile -q -p -U -z

"${function_dir:t}.zwc" "$function_files[@]" ) \ &&

If I'm reading this correctly you are generating a one .zwc file for
every .zsh file. This might require twice as many stat calls (and
possibly twice as many seeks) to verify that the compiled copy is newer
than the original.

I'm sure my memory is fuzzy here, but... I do remember that you can
compile multiple .zsh files and stuff them into a single compiled .zwc
file. I tried this but it I'm pretty sure it was a wash for hot start
up. I don't remember if I tested cold start.

If that works, you'll only generate one extra seek instead of,
potentially, doubling them. If you have a slower drive you probably
only had to generate a dozen extra seeks to cause a 25 ms slow down.
Unfortunately, it is still going to try to stat every .zsh file.

I just did a quick, inaccurate count of files in .oh-my-zsh that will
be loaded. There's about 35 without counting plugins, so cutting
that number down might potentially buy us up to a tenth second or
more.

The following numbers are after a cold start.

no compiling: 0.59
zcompdump compiling: 0.55
full compiling: 0.88

Pat

Contributor

oknowton commented Sep 2, 2011

OK. I seem to have lied. I do have a spinning drive available. It isn't helping me benchmark anything, though. The Core i7 processor in my laptop completely ignores any sort of minimum or maximum clock frequency settings or governor changes. Without any changes to my oh-my-zsh config, my cold cache start up time is varying by nearly 0.1 seconds (almost 25%).

I was attempted to test a naive little optimization. I did a cat *.zsh > ../tmp.zsh; rm *.zsh; mv ../tmp.zsh . in .oh-my-zsh/custom and /lib. It worked fine in /custom, /lib was less happy. If I had a way to get some numbers out of this, I would have investigated the error on the lib/tmp.zsh file. :)

Contributor

sorin-ionescu commented Sep 2, 2011

You are not reading that correctly. I'm generating one zwc file per directory, except for the files in $HOME. I'll try to recopy .oh-my-zsh since git reflog was around 800 entries due to as many rebases.

On Sep 2, 2011, at 16:13, oknowtonreply@reply.github.com wrote:

On Fri, 2 Sep 2011 10:18:18 -0700
sorin-ionescu
reply@reply.github.com
wrote:

A ZSH cold start can take between 1 and 3 seconds depending
on its mood. Though, on average, it takes around 1.38.

I haven't thought/worried very much about startup time with a cold
cache. I've imagined that this is a rare occurrence for most people.

I can only imagine that the biggest thing slowing down a cold start is
the large number of files that need to be read. If you're lucky, and
your install is fresh, all (or most) of the .zsh files will be in
adjacent blocks and everything will get pulled in in a single
contiguous read.

I'd be surprised if git edits files in place. It probably creates a
new file for every patch and moves it over the top of the old one. If
that really is the case, then every time git updates a file you're
likely adding a single seek. Each seek adds almost 10ms on a 7200 RPM
drive.

I'm traveling and only have an SSD with me (and I can't be dropping
caches on my servers) so I can't test this. You can test this by
moving your .oh-my-zsh directory and copying in back into place, so
that it will be likely to get one contiguous chunk of disk space.

I have tried the following, but it takes 0.25 to run on my system
negating whatever speedup .zwc files provide.

   && ( cd "$function_dir:h" && zrecompile -q -p -U -z

"${function_dir:t}.zwc" "$function_files[@]" ) \ &&

If I'm reading this correctly you are generating a one .zwc file for
every .zsh file. This might require twice as many stat calls (and
possibly twice as many seeks) to verify that the compiled copy is newer
than the original.

I'm sure my memory is fuzzy here, but... I do remember that you can
compile multiple .zsh files and stuff them into a single compiled .zwc
file. I tried this but it I'm pretty sure it was a wash for hot start
up. I don't remember if I tested cold start.

If that works, you'll only generate one extra seek instead of,
potentially, doubling them. If you have a slower drive you probably
only had to generate a dozen extra seeks to cause a 25 ms slow down.
Unfortunately, it is still going to try to stat every .zsh file.

I just did a quick, inaccurate count of files in .oh-my-zsh that will
be loaded. There's about 35 without counting plugins, so cutting
that number down might potentially buy us up to a tenth second or
more.

The following numbers are after a cold start.

no compiling: 0.59
zcompdump compiling: 0.55
full compiling: 0.88

Pat

Reply to this email directly or view it on GitHub:
#316 (comment)

Owner

robbyrussell commented Nov 25, 2012

Closing this for now since it's been dormant for a year

Contributor

docwhat commented Jun 1, 2013

FYI: I added PR #1829 that would be a good first step for this, since it'll prevent problems with ZSH changing versions or using NFS mounted home directories.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment