Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

speed up ssh completion #4377

Closed
ThomasAH opened this issue Sep 5, 2017 · 7 comments
Closed

speed up ssh completion #4377

ThomasAH opened this issue Sep 5, 2017 · 7 comments
Labels
Milestone

Comments

@ThomasAH
Copy link

@ThomasAH ThomasAH commented Sep 5, 2017

A wish list entry as discussed in issue #4374:
Completion of ssh hostnames in fish 2.6 and current git master becomes slower when many host names are available in known_hosts (and possibly ~/.ssh/config)

In my case:

$ wc -l /etc/ssh/ssh_known_hosts ~/.ssh/known_hosts
   314 /etc/ssh/ssh_known_hosts
   470 /home/thomas/.ssh/known_hosts
$ __fish_print_hostnames | wc -l
  1510
$ __fish_print_hostnames | grep . | sort | uniq | wc -l
  1282

Most entries have one or more IPs in addition to hostnames, some have multiple hostnames.
Some hosts are listed twice due to multiple public keys (e.g. ssh-rsa and ssh-ed25519).

~/.ssh/config has multiple entries (some with multiple hostnames), too:

$ grep '^Host ' ~/.ssh/config|wc -l
  140
$ grep '^Host ' ~/.ssh/config|cut -d' ' -f2-|tr ' ' '\n'|wc -l
  233

Timing is done with:
time fish -c __fish_print_hostnames >/dev/null

Which yields 0.32s (best of multiple runs) on Intel Xeon CPU E5-2620 @ 2.10GHz
and 0.51s on AMD Opteron 2352.

For comparison: bash performs completion here without any noticeable delay.

@faho
Copy link
Member

@faho faho commented Sep 5, 2017

So.... there's a bunch of stuff we can do here. Some is general and requires a bunch of work, some is specific to the ssh completion script and the __fish_print_hostnames function.

The biggest one is this:

complete -x -c ssh -d Hostname -a "
(__fish_print_hostnames)

(
	# Prepend any username specified in the completion to the hostname
	echo (commandline -ct)|sed -ne 's/\(.*@\).*/\1/p'
)(__fish_print_hostnames)
"

That actually calls __fish_print_hostnames twice! Considering that that function is about 80% of the time spent in competing this, just fixing that saves about 40%.

For more improvement, we need profiling. Fortunately fish has that built-in. Unfortunately it's not accessible from inside a running session, so this will include fish's startup time (and yes, we have issues open about that). It will also include an error since the commandline is wrong, but that shouldn't matter.

On my system, about 90% of the time spent in __fish_print_hostnames is actually spent in the if getent hosts block, so I'm going to need some data from someone with a more involved ssh setup.

@ThomasAH: Please run fish --profile /tmp/profile -c '__fish_print_hostnames' on your system, audit that file for anything you consider private and upload it. If you don't want to go through the entire file, run sort -nk2 /tmp/profile | tail -n 20 and upload that instead.

@ThomasAH
Copy link
Author

@ThomasAH ThomasAH commented Sep 5, 2017

Interesting ...

Full profile attached. There is nothing private in it, the only information that some people might consider private is the location of my home directory and that I use fish's vi mode :)

Edit: It seems I can't attach the profile here. Here is the output of sort -nk2 /tmp/profile | tail -n 20:

11057	11057	------> cat $config ^/dev/null \
164	11221	-----> set paths $paths (cat $config ^/dev/null \
28	11249	----> for config in $argv...
154	11472	---> _recursive $ssh_config
221	11814	--> _ssh_include $ssh_config
47	12998	--> for mode in insert default visual...
172	17395	-> set -l ssh_configs (_ssh_include /etc/ssh/ssh_config) (_ssh_include $ssh_config)
2231	26722	-> fish_vi_key_bindings
1481	28328	> builtin source /home/thomas/.config/fish/config.fish
51064	51064	------> nroff -c -man $mfish -t $rLL "$__fish_datadir/man/man1/$item.1" ^/dev/null
1165	52229	-----> set help (nroff -c -man $mfish -t $rLL "$__fish_datadir/man/man1/$item.1" ^/dev/null)
12	52275	----> if test -e "$__fish_datadir/man/man1/$item.1"...
59852	59852	--> and string replace -r '[#@| ].*' '' <$file \
4192	76949	----> for line in $help
99654	99654	--> and string replace -r '[#@| ].*' '' <$file \
855	133394	---> __fish_print_help commandline
1342	134736	--> commandline -cpo
243	135802	-> set -l ssh_command (functions ssh | string split ' ') (commandline -cpo)
175	160966	-> for file in $known_hosts...
1409	329553	> __fish_print_hostnames
@faho
Copy link
Member

@faho faho commented Sep 5, 2017

Okay, so:

  • You are using the version from fish 2.6.0

  • The majority of your time actually is spent in the known_hosts stuff

  • You might have a ssh function - can you upload that? functions ssh | xsel --clipboard should add it to your clipboard (if you have xsel installed and are running this on X11)

Anyway, I'm going to open a PR any minute now that could possibly improve stuff somewhat.

@faho faho mentioned this issue Sep 5, 2017
0 of 1 task complete
@faho
Copy link
Member

@faho faho commented Sep 5, 2017

@ThomasAH: In case you didn't see it, please test #4378!

@ThomasAH
Copy link
Author

@ThomasAH ThomasAH commented Sep 5, 2017

Yes, I'm running fish 2.6.0 (Debian jessie using your OBS packages).
I don't have an ssh function (verified by running functions ssh and receiving an empty output)
And I just tested PR #4378, see my reply there.

@mqudsi
Copy link
Contributor

@mqudsi mqudsi commented Sep 26, 2017

I've tested @ThomasAH's awk script from #4378 under FreeBSD with the BSD awk and on Solaris, and it works without a problem. Awesome job. It properly ignores hashed hosts as well (which is another situation where #4344 would come in handy). It handles host names, IPv4 addresses, IPv6 addresses, and the loopback address correctly.

I've merged it into both master and 2.7.0

@mqudsi mqudsi closed this Sep 26, 2017
@faho faho added this to the fish 2.7.0 milestone Sep 26, 2018
@faho faho added the enhancement label Sep 26, 2018
@dzelionis
Copy link

@dzelionis dzelionis commented Nov 15, 2019

My solution to problem works like a charm:
crontab is regularly updating /etc/hosts for me using all the different sources (nms, db's, python + rest apis...erc...), so now it's also creating special /etc/hosts_for_fish hosts file for fish, which only contains hostnames and no ip:
domas.zelionis@eng ~> grep lab /etc/hosts_for_fish
lab_sam_dcn
lab_le_310_mn02
lab_cn5305_an01
lab_cn_3940_an02
lab_cn_3930_an03

domas.zelionis@eng ~> cat /etc/hosts_for_fish |wc -l
4682

domas.zelionis@eng ~> date --iso-8601=ns ;__fish_print_hostnames > /dev/null;date --iso-8601=ns;
2019-11-15T01:43:42,308179186+0000
2019-11-15T01:43:42,310263370+0000
domas.zelionis@eng ~> date --iso-8601=ns ;__fish_print_hostnames > /dev/null;date --iso-8601=ns;
2019-11-15T01:43:43,684200266+0000
2019-11-15T01:43:43,686187426+0000
domas.zelionis@eng ~> date --iso-8601=ns ;__fish_print_hostnames > /dev/null;date --iso-8601=ns;
2019-11-15T01:43:44,962180906+0000
2019-11-15T01:43:44,964195533+0000

root@eng:/usr/share/fish/functions$cat /usr/share/fish/functions/__fish_print_hostnames.fish

function __fish_print_hostnames -d "Print a list of known hostnames"
cat /etc/hosts_for_fish
return 0
# HACK: This only deals with ipv4

# Print all hosts from /etc/hosts
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 16, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.