jenv init is really slow

[ggascoigne]

I finally got around to trying to work out why my shell startup time was so slow. Turns out that over 70% of the time spent in my .bashrc file is spent executing the eval "$(jenv init -)" line.

$ time eval "$(jenv init -)"

real    0m3.164s
user    0m2.879s
sys 0m0.258s

This seems a bit excessive. I should add that I have a ton of java version installed.

$ jenv versions
  system
  1.6
  1.6.0.65
  1.7
  1.7.0.11
  1.7.0.75
  1.7.0.76
* 1.8 (set by /Users/ggp/.jenv/version)
  1.8.0.20
  1.8.0.25
  1.8.0.31
  1.8.0.40
  1.8.0.45
  1.8.0.51
  1.8.0.60
  1.8.0.65
  1.8.0.66
  1.8.0.71
  1.8.0.72
  1.8.0.73
  1.8.0.74
  1.8.0.77
  1.8.0.91
  1.8.0.92
  oracle64-1.6.0.65
  oracle64-1.7.0.11
  oracle64-1.7.0.67
  oracle64-1.7.0.75
  oracle64-1.7.0.76
  oracle64-1.8.0.20
  oracle64-1.8.0.25
  oracle64-1.8.0.31
  oracle64-1.8.0.40
  oracle64-1.8.0.45
  oracle64-1.8.0.51
  oracle64-1.8.0.60
  oracle64-1.8.0.65
  oracle64-1.8.0.66
  oracle64-1.8.0.71
  oracle64-1.8.0.72
  oracle64-1.8.0.73
  oracle64-1.8.0.74
  oracle64-1.8.0.77
  oracle64-1.8.0.91
  oracle64-1.8.0.92

Bash version is:

$ bash --version
GNU bash, version 4.3.42(1)-release (x86_64-apple-darwin14.5.0)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[hoesler]

I had the same slow shell startup experience on my Mac after the list of *env managers started to grow (currently rbenv, pyenv and jenv). It seems that rehashing is to blame for that (besides ASL lookups).

My research revealed these approaches to speed up shell initialization:

1. Add all $PATH modifying stuff in .profile, or your shell specific profile (e.g. .bash_profile), as these files are supposed to be sourced only once at the time of login. On Mac, however, all shells are login shells and therefore all login scripts are called each time you start up a shell.

2. Lazy load your *env manager:

   rbenv() {
     eval "$(command rbenv init -)"
     rbenv "$@"
   }

   As the actual init function is called only after the first call to *env. The downside of this approach is that you have to use the default system version at startup.

3. Initialize your *env without rehashing and (optionally) start a background rehashing process:

   eval "$(jenv init - --no-rehash)"
   (jenv rehash &) 2> /dev/null

4. Use a tool which does not use shims (for ruby there is chruby) in combination with direnv.

For me, option no. 3 seems like the best approach. I see no drawbacks, it imposes only minor modifications, is applicable to all rbenv clones and it has sped up my shells startup time by factor ~1,75:

Before:

$ for i in $(seq 1 10); do /usr/bin/time zsh -i -c exit; done
        1.17 real         0.71 user         0.40 sys
        1.17 real         0.71 user         0.39 sys
        1.13 real         0.70 user         0.39 sys
        1.16 real         0.71 user         0.40 sys
        1.11 real         0.71 user         0.39 sys
        1.12 real         0.72 user         0.39 sys
        1.10 real         0.70 user         0.38 sys
        1.11 real         0.70 user         0.39 sys
        1.10 real         0.70 user         0.39 sys
        1.10 real         0.70 user         0.39 sys

After:

$ for i in $(seq 1 10); do /usr/bin/time zsh -i -c exit; done
        0.63 real         0.26 user         0.25 sys
        0.64 real         0.26 user         0.26 sys
        0.64 real         0.26 user         0.25 sys
        0.65 real         0.26 user         0.25 sys
        0.66 real         0.27 user         0.26 sys
        0.61 real         0.26 user         0.25 sys
        0.65 real         0.26 user         0.26 sys
        0.65 real         0.26 user         0.25 sys
        0.63 real         0.26 user         0.25 sys
        0.65 real         0.26 user         0.25 sys

Hope that helps.

[gcuisinier]

Thanks.

I will take a look at this to improve performance.

[ggascoigne]

@hoesler Option 3 is giving me an even better performance improvement using bash, thank you so much.

Before:

$ for i in $(seq 1 10); do /usr/bin/time bash -i -c 'exit 2>/dev/null'; done
        8.09 real         4.67 user         2.12 sys
        5.09 real         4.33 user         0.67 sys
        5.01 real         4.27 user         0.65 sys
        5.04 real         4.30 user         0.64 sys
        4.99 real         4.27 user         0.64 sys
        5.04 real         4.28 user         0.65 sys
        5.02 real         4.29 user         0.64 sys
        5.02 real         4.29 user         0.65 sys
        5.22 real         4.38 user         0.68 sys
        5.10 real         4.31 user         0.69 sys

After:

$ for i in $(seq 1 10); do /usr/bin/time bash -i -c 'exit 2>/dev/null'; done
        1.98 real         1.39 user         0.50 sys
        1.90 real         1.32 user         0.51 sys
        1.95 real         1.37 user         0.51 sys
        1.91 real         1.33 user         0.50 sys
        1.91 real         1.34 user         0.50 sys
        1.93 real         1.35 user         0.51 sys
        1.91 real         1.34 user         0.50 sys
        1.92 real         1.34 user         0.51 sys
        1.90 real         1.33 user         0.50 sys
        1.88 real         1.31 user         0.50 sys

[cvakiitho]

Sorry for reply to dead topic, but in case someone else ends here and wants to have even faster shell start - put the eval into .profile or .zprofile for zshrc, so it is loaded only once after login.

before:

for i in $(seq 1 10); do /usr/bin/time zsh-i -c exit; done
        0.47 real         0.21 user         0.18 sys
        0.44 real         0.20 user         0.17 sys
        0.43 real         0.20 user         0.16 sys
        0.68 real         0.22 user         0.18 sys
        0.42 real         0.19 user         0.16 sys
        0.42 real         0.19 user         0.16 sys
        0.42 real         0.19 user         0.16 sys
        0.42 real         0.19 user         0.16 sys
        0.42 real         0.19 user         0.16 sys
        0.44 real         0.20 user         0.16 sys

after

for i in $(seq 1 10); do /usr/bin/time zsh -i -c exit; done
        0.19 real         0.09 user         0.08 sys
        0.17 real         0.08 user         0.08 sys
        0.17 real         0.08 user         0.07 sys
        0.16 real         0.07 user         0.07 sys
        0.15 real         0.07 user         0.06 sys
        0.16 real         0.07 user         0.07 sys
        0.17 real         0.07 user         0.07 sys
        0.15 real         0.07 user         0.06 sys
        0.15 real         0.07 user         0.07 sys
        0.15 real         0.07 user         0.06 sys

[kaplin]

I'm doubling that. Wait 10 seconds for each terminal window to open is not acceptable.

[marcopelegrini]

Sorry for reply to dead topic, but in case someone else ends here and wants to have even faster shell start - put the eval into .profile or .zprofile for zshrc, so it is loaded only once after login.

Mind that for macos, every new window/tab is a login shell, so essentially there's no difference... your benchmark script would have to be /usr/bin/time zsh -il -c exit for macos

https://unix.stackexchange.com/questions/119627/why-are-interactive-shells-on-osx-login-shells-by-default

[cvakiitho]

Sorry for reply to dead topic, but in case someone else ends here and wants to have even faster shell start - put the eval into .profile or .zprofile for zshrc, so it is loaded only once after login.

Mind that for macos, every new window/tab is a login shell, so essentially there's no difference... your benchmark script would have to be /usr/bin/time zsh -il -c exit for macos

https://unix.stackexchange.com/questions/119627/why-are-interactive-shells-on-osx-login-shells-by-default

Ha! thanks for clearing that up, didn't notice the difference when starting the shell(and since then I have jenv init in zsh alias to have it on demand).

[AdamH4]

Any idea how to do this in fish shell?

I used the following:

eval (jenv init - --no-rehash)
(jenv rehash &) 2> /dev/null

But this is the output

~/.config/fish/config.fish (line 27): Command substitutions not allowed
(jenv rehash &) 2> /dev/null
^
from sourcing file ~/.config/fish/config.fish
	called during startup
source: Error while reading file '/Users/adam/.config/fish/config.fish'

[rmartine-ias]

@AdamH4 it's not perfect, but this seems to work for the lazy-loading solution:

~/.config/fish/conf.d/53_jenv.fish:

if ! type -fq jenv
    exit 0
end
if ! status --is-login
    exit 0
end

set -a PATH "$HOME"/.jenv/bin

function jenv --wraps=jenv
    functions --erase jenv
    jenv init - | source
    jenv $argv
end

function java --wraps=java
    functions --erase java
    jenv init - | source
    java $argv
end

Backgrounding the rehashing, to me, was only saving ~0.2 seconds on shell startup. This saves all of them.

Downsides I'm too lazy to fix:

• java shouldn't re-source jenv if it's already been sourced
• java pattern should be repeated for everything in ~/.jenv/shims/ (automatically)
• The background rehashing could be done in addition to this pattern; maybe this would work? Leaving it for someone else to try.
• all can be wrapped in --is-interactive prolly

[rmartine-ias]

It's not perfect, (in fact, it's downright ugly) but this is as fast and more correct than my above comment. Adding completions adds ~25ms on my system, so disable that if you don't care. It adds a lazily-loaded second layer of shims, that deregister themselves and initiate jenv on being called, as well as throwing up a background thread to rehash both the jenv shims and the second-layer shims. Note: if you're used to calling exec fish, replace that with exec fish -l in your head.

Adding completions could be faster if they were not generated fresh every time.

~/.config/fish/conf.d/53_jenv.fish:

if ! type -fq jenv
    exit 0
end

# Getting completions file out of jenv init is slow
function __make_cached_jenv_completions
    set -l complete (jenv init - | grep completions | sed -E 's/source \'(.*)\'/\1/')
    ln -sf $complete $argv[1]
end

if status --is-login
    set -l completions_path "$HOME/.cache/jenv/_completions_path"
    if [ -f $completions_path ]
        source $completions_path
        fish -c "__make_cached_jenv_completions $completions_path" &
    else
        mkdir -p (dirname $completions_path)
        __make_cached_jenv_completions $completions_path
        source $completions_path
    end

    set -l shim_shims "$HOME/.local/share/jenv_shim_shims"

    set -ga fish_function_path $shim_shims

    # Rehash shims in the background; don't print anything.
    # See ~/.config/fish/functions/jenv.fish for second layer rehashing.
    # https://github.com/fish-shell/fish-shell/issues/238#issuecomment-1015806466
    begin; fish -c "jenv rehash" &; end
end

~/.config/fish/functions/jenv.fish:

function jenv --wraps jenv
    if contains rehash $argv
        command jenv rehash

        # set to value of $(jenv root)/shims
        set -l jenv_shims "$HOME/.jenv/shims"
        set -l shim_shims "$HOME/.local/share/jenv_shim_shims"
        if [ ! -d $shim_shims ] || [ (ls $jenv_shims | wc -l) != (ls $shim_shims | wc -l) ]
            mkdir -p $shim_shims

            for file in $jenv_shims/*
                set shim $(basename $file)
                echo "\
#!/usr/bin/env fish
function $shim
    # Erase second-layer shims from path
    set index (contains -i \"$shim_shims\" \$fish_function_path)
    and set -ge fish_function_path[\$index]

    # Start jenv and set path to include first-layer shims
    jenv init - --no-rehash | source
    set -gxp PATH \"$jenv_shims\"

    # Call original function
    $shim \$argv
end\
" > $shim_shims/$shim.fish

            end
        end
    # Getting commands is slow! Calling any actual jenv command is slow!
    # And it happens during generating completions
    # This block saves ~20ms on shell startup
    else if contains commands $argv
        set -l com_file "$HOME/.cache/jenv_commands/$(string join -- '_' (string escape -- $argv))"
        if [ -f "$com_file" ]
            # For some godawful reason this takes ~30ms on my machine
            # Bash, fish, tmux, not!tmux, iterm, Terminal.app, whatever -- cat is slow.
            # I do not know why, and I hate it.
            # Consecutive runs are faster.
            # This is still faster than running jenv commands.
            command cat "$com_file"
        else
            mkdir -p (dirname $com_file)
            command jenv $argv
        end
        begin; fish -c "command jenv $argv > $com_file" &; end
    else
        command jenv $argv
    end
end

[rmartine-ias]

Okay, last go of it for now, I got completions faster by pre-generating them instead of generating them fresh on every shell start. This way fish can just lazy-load them, like every other set of completions.

Only downside is they have to manually be regenerated, along with shims, by running jenv rehash yourself. "Background" rehashing was turning out to still take considerable time. I did set this up to run after jenv add is called, but there may be other cases you need to do it yourself. (An upgrade to jenv, maybe. This could also be automatic, but eeehh.)

~/.config/fish/conf.d/53_jenv.fish:

if ! type --no-functions -q jenv
    exit 0
end

if status --is-login
    set -ga fish_function_path "$HOME/.local/share/jenv_shim_shims"
end


# Getting completions out of jenv init is slow. This gets called by rehash.
# See ~/.config/fish/functions/jenv.fish for rehashing.
# Rehash should be called manually.
function __make_jenv_completions
    function __fish_jenv_needs_command
      set cmd (commandline -opc)
      if [ (count $cmd) -eq 1 -a $cmd[1] = 'jenv' ]
        return 0
      end
      return 1
    end

    function __fish_jenv_using_command
      set cmd (commandline -opc)
      if [ (count $cmd) -gt 1 ]
        if [ $argv[1] = $cmd[2] ]
          return 0
        end
      end
      return 1
    end

    set -l commands (jenv commands | string join ' ')

    functions __fish_jenv_needs_command
    functions __fish_jenv_using_command
    echo 'complete -f -c jenv -n \'__fish_jenv_needs_command\' -a "'$commands'"'
    echo 'complete -f -c jenv -n \'__fish_jenv_using_command help\' -a "'$commands'"'
    for cmd in (string split ' ' -- "$commands")
        # these two are broken
        if not contains -- $cmd disable-plugin doctor
            set -l completions (string join ' ' -- (jenv completions $cmd))
            if ! test -z $completions
                string join '' -- 'complete -f -c jenv -n "__fish_jenv_using_command '$cmd'" -a "' $completions '"'
            end
        end
    end
end

~/.config/fish/functions/jenv.fish:

function jenv --wraps jenv
    if [ (count $argv) -gt 0 ] && [ $argv[1] = rehash ]
        command jenv rehash

        fish -c "__make_jenv_completions > ~/.config/fish/completions/jenv.fish" &

        # set to value of $(jenv root)/shims
        set -l jenv_shims "$HOME/.jenv/shims"
        set -l shim_shims "$HOME/.local/share/jenv_shim_shims"
        if [ ! -d $shim_shims ] || [ (ls $jenv_shims | wc -l) != (ls $shim_shims | wc -l) ]
            mkdir -p $shim_shims

            for file in $jenv_shims/*
                set shim $(basename $file)
                echo "\
#!/usr/bin/env fish
function $shim
    # Erase second-layer shims from path
    set index (contains -i \"$shim_shims\" \$fish_function_path)
    and set -ge fish_function_path[\$index]

    # Start jenv and set path to include first-layer shims
    jenv init - --no-rehash | source
    set -gxp PATH \"$jenv_shims\"

    # Call original function
    $shim \$argv
end\
" > $shim_shims/$shim.fish

            end
        end
    else if [ (count $argv) -gt 0 ] && [ $argv[1] = add ]
        command jenv $argv
        and jenv rehash
    else
        command jenv $argv
    end
end

[nyg]

eval "$(jenv init - --no-rehash)"
(jenv rehash &) 2> /dev/null

This made me stumble upon zsh-defer, made by Zsh guru romkatv :). It can be used to defer the execution of any command:

zsh-defer eval "$(jenv init -)"
zsh-defer . "$NVM_DIR/nvm.sh"
…

Start-up times (login interactive shell):

• Sourcing jenv & nvm without zsh-defer: 800ms
• Sourcing jenv & nvm with zsh-defer: 170ms
• Sourcing none of the above: 90ms

[zkyangc]

For zsh init:

lazy_load_jenv() {
  unset -f java jenv
  eval export PATH="${HOME}/.jenv/shims:${PATH}"
  eval "$(jenv init -)"
}

jenv() {
  lazy_load_jenv
  jenv $@
}

java() {
  lazy_load_jenv
  java $@
}