SSH host verification isn't working for me at all (including with clean known_hosts)

[swt30]

Julia Version 1.6.0
Commit f9720dc2eb* (2021-03-24 12:55 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.1.0 (ORCJIT, skylake)

Whenever I attempt to dev a package, including from a blank ~/.ssh/known_hosts file, I get this error:

(@v1.6) pkg> dev Revise
     Cloning git-repo `https://github.com/timholy/Revise.jl.git`
SSH host verification: the identity of the server `github.com` does not match its known hosts record. Someone could be trying to man-in-the-middle your connection. It is also possible that the server has changed its key, in which case you should check with the server administrator and if they confirm that the key has been changed, update your known hosts file.
ERROR: failed to clone from https://github.com/timholy/Revise.jl.git, error: GitError(Code:ERROR, Class:Net, user cancelled hostkey check)

I get the same message if I run ssh-keyscan github.com >> ~/.ssh/known_hosts beforehand, so that the github record is in the file. I've also verified that the record in the file is the right one (matches that published by github, so no MITM going on here).

If I had to guess what the problem is, I think it's probably because I have set up gpg-agent to do SSH, following this guide here. It's caused some problems here and there before.

Any ideas why things might not be working out between Pkg, known_hosts and gpg-agent-doing-SSH?

[fredrikekre]

duplicate of JuliaLang/julia#40297 ?

[swt30]

I don't think so - it looks like that issue is about Gitlab publishing multiple keys, and it is also fixed by wiping known_hosts. Whereas Github publishes only one key (as noted by Stefan) and I get the error even on a blank known_hosts file. Importantly, I don't even get the error telling me to run ssh-keyscan github.com, as is supposed to happen when there is no known host entry.

[StefanKarpinski]

We do ship a fallback known hosts file, which we copy to a temp directory on-demand. You can find it by doing this:

juila> using NetworkOptions

julia> NetworkOptions.ssh_known_hosts_files()
2-element Vector{String}:
 "/Users/stefan/.ssh/known_hosts"
 "/var/folders/4g/b8p546px3nd550b3k288mhp80000gp/T/jl_EjTsCB"

That second file is the bundled known hosts file, which should contain exactly this:

github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf
gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9

(This doesn't cause a problem for gitlab because we only fall back to this file if there's no gitlab entries in the first file at all and we do include all three keys and don't add any IP-specific entries.)

Can you try, in addition to emptying your personal known hosts file, also empyting that file to be empty and seeing what happens? You can do this in a Julia session by starting up and doing this:

julia> using NetworkOptions

shell> cp /dev/null $(NetworkOptions.ssh_known_hosts_files()[end])

Then try the dev command in that session. My expectation is that you should get an unknown host error, but it's good to double check that expectation. If that's not the case then your gpg-agent setup may be bypassing looking at known hosts files altogether.

[swt30]

Yes, that works as you expect, giving an unknown host error:

shell> cp /dev/null /home/sthomas/.ssh/known_hosts

shell> cat /home/sthomas/.ssh/known_hosts

julia> using NetworkOptions

shell> cp /dev/null $(NetworkOptions.ssh_known_hosts_files()[end])

julia> NetworkOptions.ssh_known_hosts_files()
2-element Vector{String}:
 "/home/sthomas/.ssh/known_hosts"
 "/tmp/jl_qenz1T"

shell> cat /tmp/jl_qenz1T

(@v1.6) pkg> dev Revise
     Cloning git-repo `https://github.com/timholy/Revise.jl.git`
SSH host verification: the server `github.com` is not a known host. Please run `ssh-keyscan github.com >> /home/sthomas/.ssh/known_hosts` in order to add the server to your known hosts file and then try again.
ERROR: failed to clone from https://github.com/timholy/Revise.jl.git, error: GitError(Code:ERROR, Class:Net, user cancelled hostkey check)

[StefanKarpinski]

Could you try without the gpg-agent to see if it works without that?

[swt30]

Interestingly, I just tried this on my work Mac, where I have also set up gpg-agent in the same way, and it works as expected. I get the right error when I wipe both versions of known_hosts, and then the clone succeeds when either of them exists. I'll dig into this a bit further and see if I can understand where the difference is. I use identical config files on both machines where possible, so either I haven't set them up as identically as I thought, or it's a Mac/Linux difference.

[swt30]

OK, so I confirmed that
a) there's no difference in gpg-agent setup between the machines
b) it also doesn't work when I switch back to standard ssh-agent on Linux, giving the same "does not match its known hosts record" error
so it isn't gpg-agent causing the issue.

Digging into the code suggests that libssh2 is giving a LIBSSH2_KNOWNHOST_CHECK_MISMATCH here: https://github.com/JuliaLang/julia/blob/dbaca8ba16d406004c53cb211b9eaf8028f6b6be/stdlib/LibGit2/src/callbacks.jl#L475

[StefanKarpinski]

Are you using an official Julia build or is a build with a system libssh2?

[swt30]

Thanks, you've found the problem. I was using the Arch default Julia build in the Community repo (the one you get from pacman -S julia). Switching to a binary (julia-bin in the AUR) fixes things. I saw that Mose had actually recommended the latter for an unrelated issue just a couple days ago.

So you think this is a problem with building against system libssh2? Anything to be done, or is that unsupported and we should close this?

[StefanKarpinski]

I wouldn't mind continuing to debug until we understand what's going on but I'm not sure if we'll be able to do anything about a broken setup with system libraries.

What version of libgit2 and libssh2 are you using?

[swt30]

Sure, happy to help if you think it's worth pursuing.

libssh2 1.9.0-3
libgit2 1:1.1.0-1

[StefanKarpinski]

Those are both quite recent, so oldness of the system libraries doesn't seem likely to be the problem. I'm not entirely sure what else to try to understand what's happening, short of setting up an Arch Linux VM myself and trying to reproduce this, which I'm not really interested in doing.

[swt30]

Yeah, understood. Unfortunately I don't really know how one debugs system libraries, so I can't help any further. I'll close this, noting that a solution was to switch from Arch's provided julia to the AUR julia-bin (i.e. use a prebuilt binary).

[swt30]

Thanks for your help :)