New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

speed up ssh completion #4377

Closed
ThomasAH opened this Issue Sep 5, 2017 · 6 comments

Comments

Projects
None yet
3 participants
@ThomasAH

ThomasAH commented Sep 5, 2017

A wish list entry as discussed in issue #4374:
Completion of ssh hostnames in fish 2.6 and current git master becomes slower when many host names are available in known_hosts (and possibly ~/.ssh/config)

In my case:

$ wc -l /etc/ssh/ssh_known_hosts ~/.ssh/known_hosts
   314 /etc/ssh/ssh_known_hosts
   470 /home/thomas/.ssh/known_hosts
$ __fish_print_hostnames | wc -l
  1510
$ __fish_print_hostnames | grep . | sort | uniq | wc -l
  1282

Most entries have one or more IPs in addition to hostnames, some have multiple hostnames.
Some hosts are listed twice due to multiple public keys (e.g. ssh-rsa and ssh-ed25519).

~/.ssh/config has multiple entries (some with multiple hostnames), too:

$ grep '^Host ' ~/.ssh/config|wc -l
  140
$ grep '^Host ' ~/.ssh/config|cut -d' ' -f2-|tr ' ' '\n'|wc -l
  233

Timing is done with:
time fish -c __fish_print_hostnames >/dev/null

Which yields 0.32s (best of multiple runs) on Intel Xeon CPU E5-2620 @ 2.10GHz
and 0.51s on AMD Opteron 2352.

For comparison: bash performs completion here without any noticeable delay.

@faho

This comment has been minimized.

Show comment
Hide comment
@faho

faho Sep 5, 2017

Member

So.... there's a bunch of stuff we can do here. Some is general and requires a bunch of work, some is specific to the ssh completion script and the __fish_print_hostnames function.

The biggest one is this:

complete -x -c ssh -d Hostname -a "
(__fish_print_hostnames)

(
	# Prepend any username specified in the completion to the hostname
	echo (commandline -ct)|sed -ne 's/\(.*@\).*/\1/p'
)(__fish_print_hostnames)
"

That actually calls __fish_print_hostnames twice! Considering that that function is about 80% of the time spent in competing this, just fixing that saves about 40%.

For more improvement, we need profiling. Fortunately fish has that built-in. Unfortunately it's not accessible from inside a running session, so this will include fish's startup time (and yes, we have issues open about that). It will also include an error since the commandline is wrong, but that shouldn't matter.

On my system, about 90% of the time spent in __fish_print_hostnames is actually spent in the if getent hosts block, so I'm going to need some data from someone with a more involved ssh setup.

@ThomasAH: Please run fish --profile /tmp/profile -c '__fish_print_hostnames' on your system, audit that file for anything you consider private and upload it. If you don't want to go through the entire file, run sort -nk2 /tmp/profile | tail -n 20 and upload that instead.

Member

faho commented Sep 5, 2017

So.... there's a bunch of stuff we can do here. Some is general and requires a bunch of work, some is specific to the ssh completion script and the __fish_print_hostnames function.

The biggest one is this:

complete -x -c ssh -d Hostname -a "
(__fish_print_hostnames)

(
	# Prepend any username specified in the completion to the hostname
	echo (commandline -ct)|sed -ne 's/\(.*@\).*/\1/p'
)(__fish_print_hostnames)
"

That actually calls __fish_print_hostnames twice! Considering that that function is about 80% of the time spent in competing this, just fixing that saves about 40%.

For more improvement, we need profiling. Fortunately fish has that built-in. Unfortunately it's not accessible from inside a running session, so this will include fish's startup time (and yes, we have issues open about that). It will also include an error since the commandline is wrong, but that shouldn't matter.

On my system, about 90% of the time spent in __fish_print_hostnames is actually spent in the if getent hosts block, so I'm going to need some data from someone with a more involved ssh setup.

@ThomasAH: Please run fish --profile /tmp/profile -c '__fish_print_hostnames' on your system, audit that file for anything you consider private and upload it. If you don't want to go through the entire file, run sort -nk2 /tmp/profile | tail -n 20 and upload that instead.

@ThomasAH

This comment has been minimized.

Show comment
Hide comment
@ThomasAH

ThomasAH Sep 5, 2017

Interesting ...

Full profile attached. There is nothing private in it, the only information that some people might consider private is the location of my home directory and that I use fish's vi mode :)

Edit: It seems I can't attach the profile here. Here is the output of sort -nk2 /tmp/profile | tail -n 20:

11057	11057	------> cat $config ^/dev/null \
164	11221	-----> set paths $paths (cat $config ^/dev/null \
28	11249	----> for config in $argv...
154	11472	---> _recursive $ssh_config
221	11814	--> _ssh_include $ssh_config
47	12998	--> for mode in insert default visual...
172	17395	-> set -l ssh_configs (_ssh_include /etc/ssh/ssh_config) (_ssh_include $ssh_config)
2231	26722	-> fish_vi_key_bindings
1481	28328	> builtin source /home/thomas/.config/fish/config.fish
51064	51064	------> nroff -c -man $mfish -t $rLL "$__fish_datadir/man/man1/$item.1" ^/dev/null
1165	52229	-----> set help (nroff -c -man $mfish -t $rLL "$__fish_datadir/man/man1/$item.1" ^/dev/null)
12	52275	----> if test -e "$__fish_datadir/man/man1/$item.1"...
59852	59852	--> and string replace -r '[#@| ].*' '' <$file \
4192	76949	----> for line in $help
99654	99654	--> and string replace -r '[#@| ].*' '' <$file \
855	133394	---> __fish_print_help commandline
1342	134736	--> commandline -cpo
243	135802	-> set -l ssh_command (functions ssh | string split ' ') (commandline -cpo)
175	160966	-> for file in $known_hosts...
1409	329553	> __fish_print_hostnames

ThomasAH commented Sep 5, 2017

Interesting ...

Full profile attached. There is nothing private in it, the only information that some people might consider private is the location of my home directory and that I use fish's vi mode :)

Edit: It seems I can't attach the profile here. Here is the output of sort -nk2 /tmp/profile | tail -n 20:

11057	11057	------> cat $config ^/dev/null \
164	11221	-----> set paths $paths (cat $config ^/dev/null \
28	11249	----> for config in $argv...
154	11472	---> _recursive $ssh_config
221	11814	--> _ssh_include $ssh_config
47	12998	--> for mode in insert default visual...
172	17395	-> set -l ssh_configs (_ssh_include /etc/ssh/ssh_config) (_ssh_include $ssh_config)
2231	26722	-> fish_vi_key_bindings
1481	28328	> builtin source /home/thomas/.config/fish/config.fish
51064	51064	------> nroff -c -man $mfish -t $rLL "$__fish_datadir/man/man1/$item.1" ^/dev/null
1165	52229	-----> set help (nroff -c -man $mfish -t $rLL "$__fish_datadir/man/man1/$item.1" ^/dev/null)
12	52275	----> if test -e "$__fish_datadir/man/man1/$item.1"...
59852	59852	--> and string replace -r '[#@| ].*' '' <$file \
4192	76949	----> for line in $help
99654	99654	--> and string replace -r '[#@| ].*' '' <$file \
855	133394	---> __fish_print_help commandline
1342	134736	--> commandline -cpo
243	135802	-> set -l ssh_command (functions ssh | string split ' ') (commandline -cpo)
175	160966	-> for file in $known_hosts...
1409	329553	> __fish_print_hostnames
@faho

This comment has been minimized.

Show comment
Hide comment
@faho

faho Sep 5, 2017

Member

Okay, so:

  • You are using the version from fish 2.6.0

  • The majority of your time actually is spent in the known_hosts stuff

  • You might have a ssh function - can you upload that? functions ssh | xsel --clipboard should add it to your clipboard (if you have xsel installed and are running this on X11)

Anyway, I'm going to open a PR any minute now that could possibly improve stuff somewhat.

Member

faho commented Sep 5, 2017

Okay, so:

  • You are using the version from fish 2.6.0

  • The majority of your time actually is spent in the known_hosts stuff

  • You might have a ssh function - can you upload that? functions ssh | xsel --clipboard should add it to your clipboard (if you have xsel installed and are running this on X11)

Anyway, I'm going to open a PR any minute now that could possibly improve stuff somewhat.

@faho faho referenced this issue Sep 5, 2017

Merged

Improve ssh completions/__fish_print_hostnames #4378

0 of 1 task complete
@faho

This comment has been minimized.

Show comment
Hide comment
@faho

faho Sep 5, 2017

Member

@ThomasAH: In case you didn't see it, please test #4378!

Member

faho commented Sep 5, 2017

@ThomasAH: In case you didn't see it, please test #4378!

@ThomasAH

This comment has been minimized.

Show comment
Hide comment
@ThomasAH

ThomasAH Sep 5, 2017

Yes, I'm running fish 2.6.0 (Debian jessie using your OBS packages).
I don't have an ssh function (verified by running functions ssh and receiving an empty output)
And I just tested PR #4378, see my reply there.

ThomasAH commented Sep 5, 2017

Yes, I'm running fish 2.6.0 (Debian jessie using your OBS packages).
I don't have an ssh function (verified by running functions ssh and receiving an empty output)
And I just tested PR #4378, see my reply there.

@mqudsi

This comment has been minimized.

Show comment
Hide comment
@mqudsi

mqudsi Sep 26, 2017

Contributor

I've tested @ThomasAH's awk script from #4378 under FreeBSD with the BSD awk and on Solaris, and it works without a problem. Awesome job. It properly ignores hashed hosts as well (which is another situation where #4344 would come in handy). It handles host names, IPv4 addresses, IPv6 addresses, and the loopback address correctly.

I've merged it into both master and 2.7.0

Contributor

mqudsi commented Sep 26, 2017

I've tested @ThomasAH's awk script from #4378 under FreeBSD with the BSD awk and on Solaris, and it works without a problem. Awesome job. It properly ignores hashed hosts as well (which is another situation where #4344 would come in handy). It handles host names, IPv4 addresses, IPv6 addresses, and the loopback address correctly.

I've merged it into both master and 2.7.0

@mqudsi mqudsi closed this Sep 26, 2017

@faho faho added this to the fish 2.7.0 milestone Sep 26, 2018

@faho faho added the enhancement label Sep 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment