known_hosts SSH verification failing for gitlab.com

[bryannagle]

When using Julia 1.6, running "activate" on a module that has unregistered dependencies that use git ssh urls to private repositories, the update will fail if an entry for that repository host (i.e. gitlab) is present in the users ~/.ssh/known_hosts file. Removing that entry fixes the problem, but then the entry is re-added when the user pushes to git.

Steps to Reproduce:

1. Clone a gitlab project using the SSH url.
2. In a project, using package mode, run "add git@gitlab.com:"
3. Commit your updated toml file to gitlab.
4. Run "activate" from package mode.

Repl Example:

(LiquidIAM) pkg> update
Updating registry at ~/.julia/registries/General
Updating git-repo git@gitlab.com:LiquidAnalytics/opensource/liquiddatautils.jl.git
SSH host verification: the identity of the server gitlab.com:22 does not match its known hosts record. Someone could be trying to man-in-the-middle your connection. It is also possible that the server has changed its key, in which case you should check with the server administrator and if they confirm that the key has been changed, update your known hosts file.
ERROR: failed to fetch from git@gitlab.com:LiquidAnalytics/opensource/liquiddatautils.jl.git, error: GitError(Code:ERROR, Class:Net, user cancelled hostkey check)

(LiquidIAM) pkg>

shell> rm ~/.ssh/known_hosts

(LiquidIAM) pkg> update
Updating registry at ~/.julia/registries/General
Updating git-repo git@gitlab.com:LiquidAnalytics/opensource/liquiddatautils.jl.git
Updating git-repo git@gitlab.com:LiquidAnalytics/opensource/dgraph.jl.git
Updating git-repo git@gitlab.com:LiquidAnalytics/LiquidDecisions/Goals/DistributeGoal.git
Installed AWS ─ v1.33.1
No Changes to ~/Workspace/liquid/liquidiam.jl/Project.toml
Updating ~/Workspace/liquid/liquidiam.jl/Manifest.toml
[fbe9abb3] ↑ AWS v1.32.0 ⇒ v1.33.1
Progress [========================================>] 6/6
6 dependencies successfully precompiled in 22 seconds (68 already precompiled)

(LiquidIAM) pkg>

[mbauman]

Perhaps the most telling thing is that doing a third operation (with the freshly generated known hosts entry) fails host verification again.

[StefanKarpinski]

Here's my analysis of this problem. You may be aware that if you've used git and ssh to clone/push/pull anything from gitlab.com, it saves a host fingerprint to your known hosts file (typically ~/.ssh/known_hosts). GitLab publicizes fingerprints for three different algorithms, which you can see all of with the ssh-keyscan tool:

$ ssh-keyscan gitlab.com
# gitlab.com:22 SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9
# gitlab.com:22 SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
# gitlab.com:22 SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf

Command-line git and ssh both use libssh which uses OpenSSL and have support for all of these fingerprint methods. Julia's libgit2 uses libssh2 built against mbedTLS, which does not support all of these fingerprint methods — it only supports the classic ssh-rsa fingerprints. That mismatch is the root of the problem.

The specific issue is that when you use git or ssh to clone/push/pull from GitLab, it negotiates one fingerprinting method, and it tends to pick the newer, more secure algorithms; specifically ecdsa-sha2-nistp256 is preferred, so that's what's recorded in most people's known hosts files. That would be fine if our libgit2/libssh2/mbedtls understood that method, but it doesn't, so it concludes when trying to verify the identity of gitlab.com that the host identity is invalid since it can't verify the fingerprint. If you had never connected to gitlab.com before, you'd actually get a much better error message telling you to run ssh-keyscan gitlab.com, which would fix the issue since you'd get all of the fingerprints, including the one that libgit2/libssh2/mbedtls understands.

Why was this not a problem before? Because libgit2 defaults to not verifying host identities at all (😬). This is bad, so in 1.6 we turned on host identity verification by default. You can, however, opt out of SSH host identity verification by setting the shell environment variable JULIA_SSH_NO_VERIFY_HOSTS=gitlab.com (or, if you just want to not verify anything, use ** as the value). See https://github.com/JuliaLang/NetworkOptions.jl for all the possibilities.

Why isn't this a problem for GitHub? Because they only advertise one kind of SSH fingerprint and it's ssh-rsa:

$ ssh-keyscan github.com
# github.com:22 SSH-2.0-babeld-0913e7e1
# github.com:22 SSH-2.0-babeld-0913e7e1
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
# github.com:22 SSH-2.0-babeld-0913e7e1

So our libgit2/libssh2/mbedtls stack understands that known hosts entry and everything works fine.

For the user, the short-term secure fix is to make sure that the known hosts file has an ssh-rsa fingerprint for gitlab.com, i.e. this line:

gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9

You also want to make sure this is the only fingerprint for gitlab.com in the file, which is a bit unfortunate, but otherwise you'll still have issues because of the following. When you use git and ssh they will choose the newer methods if those fingerprints are present, and then they will also record entries for specific host IP addresses in addition to the entries for the gitlab.com hostname, but only for the fingerprint method that's chosen. If a newer fingerprint type like ecdsa-sha2-nistp256 exists in known hosts for gitlab.com then git or ssh will use that method and record a corresponding ecdsa-sha2-nistp256 fingerprint for the server IP address. Later, if Julia's libgit2 tries to connect to that same IP address, which is likely, it will fail because it doesn't know how to check that kind of fingerprint.

Longer term, the solution is to make sure that our libgit2/libssh2 stack knows how to compute these other kinds of fingerprints. One solution would be to switch from mbedTLS to OpenSSL. Another would be to add support for the other fingerprint methods with mbedTLS. @staticfloat has opened an issue with the libssh2 project for this.

[iamed2]

For those wondering about the state of libssh2, there hasn't been a release in over two years, even to mitigate active CVEs. The maintainers have not responded to requests for a new release since April 1.

[StefanKarpinski]

Yeah, the libgit2+libssh2 situation is a bit of a trainwreck. Libgit2 has many problems itself (insecure defaults all over the place, inability to do many things that work fine in CLI git); it also depends on libssh2, which, as noted is no longer actively maintained and has unfixed CVEs. My conclusion is that we have to drop both.

[iamed2]

Do you think the best way is to go back to git CLI?

[StefanKarpinski]

Yes, I'm in the middle of writing up an issue on Pkg.jl

[StefanKarpinski]

JuliaLang/Pkg.jl#2679

[iamed2]

This may be able to be improved now that libssh2 1.10 has been tagged with ECDSA support for mbedTLS. Not sure if a corresponding libgit2 release needs to be made or we can just update libssh2 directly.

[timholy]

I got this when I first tried to update the registry for 1.9 (copied over my v1.8 Project.toml file, omitted the Manifest.toml and hoping to let it re-resolve):

(@v1.9) pkg> up
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `git@github.com:JuliaRegistries/General.git`
SSH host verification: the identity of the server `github.com:22` does not match its known hosts record. Someone could be trying to man-in-the-middle your connection. It is also possible that the server has changed its key, in which case you should check with the server administrator and if they confirm that the key has been changed, update your known hosts file.
    Updating registry at `~/.julia/registries/HolyLabRegistry`
    Updating git-repo `git@github.com:HolyLab/HolyLabRegistry.git`
SSH host verification: the identity of the server `github.com:22` does not match its known hosts record. Someone could be trying to man-in-the-middle your connection. It is also possible that the server has changed its key, in which case you should check with the server administrator and if they confirm that the key has been changed, update your known hosts file.
┌ Error: Some registries failed to update:
│     — /home/tim/.julia/registries/General — failed to fetch from repo: failed to fetch from git@github.com:JuliaRegistries/General.git, error: GitError(Code:ERROR, Class:Net, user cancelled hostkey check)
│     — /home/tim/.julia/registries/HolyLabRegistry — failed to fetch from repo: failed to fetch from git@github.com:HolyLab/HolyLabRegistry.git, error: GitError(Code:ERROR, Class:Net, user cancelled hostkey check)
└ @ Pkg.Registry ~/src/julia-master/usr/share/julia/stdlib/v1.9/Pkg/src/Registry/Registry.jl:449
ERROR: expected package `RegistryCompatTools [0f4576a4]` to be registered

This happened perhaps because I had a few devved unregistered packages (e.g., RegistryCompatTools above). Removing the ones I didn't want and manually devving the ones I did fixed the issue, but the warning is obviously very misleading.