failed to clone repository: ssh: handshake failed: knownhosts: key mismatch

[slyons]

This isn't the same as the GitHub key rotation issues, I swear.

I have a Git server running on a NAS device that I would like to use for flux as I don't want my infra up on Github while I'm experimenting with Flux.

When trying to bootstrap (after installing with brew), I get the following:

$ flux bootstrap git --url=ssh://git@192.168.166.10/volume1/Git/flux
► cloning branch "main" from Git repository "ssh://git@192.168.166.10/volume1/Git/flux"
✗ failed to clone repository: ssh: handshake failed: knownhosts: key mismatch

But I can SSH in to the device just fine. I did a bit of research that go-git might not support ssh-ed25519 so I even forced it to be ssh-rsa and it still doesn't working. This host is currently the only one in my known_hosts file so it's not a resolution/old key issue. I can clone with standard Git just fine.

$ flux check
► checking prerequisites
✔ Kubernetes 1.22.3-3+64e63223b19811 >=1.19.0-0
► checking controllers
✔ all checks passed

$ flux --version
flux version 0.23.0

Am I missing something?

[mrafaquat]

@slyons did you ever resolve this, i am having the same issue. many thanks

[kingdonb]

I was reading very quickly and almost accused you of being a duplicate of this issue: #2097 (comment)

Fortunately you pointed out in the very first line that you are not. However, this may be a good search result landing place for people who did have that issue, so I might as well drop the link here anyway. (Link above: GitHub has updated their SSH keys. You may need to take action.)

Did you ever resolve this? You should be able to create Flux keys interactively with flux create secret and they are populated with a known_hosts automatically. This results in a similar outcome to flux bootstrap but has less going on so might be a bit more precise tool for debugging your issue.

https://fluxcd.io/docs/cmd/flux_create_secret_git/

I use flux create secret git along with flux create source git and add the secretRef manually or through --secret-ref when adding secondary gitrepos after bootstrapping, this guidance may help to produce a valid secret and source.

It seems unlikely that the host has more than one SSH key unless it also has, besides a git server, an actual SSH server. Next question: if there are maybe indeed multiple SSH servers, are you certain that there is no custom port number expected in your gitrepository?

[slyons]

So the only way I was able to resolve it was to use a private Github repo as opposed to my own Git server, so something clearly broke between trying to resolve the server name (despite using an IP address).

…

On Thu, Jan 27, 2022 at 11:19 AM Kingdon Barrett ***@***.***> wrote: I was reading very quickly and almost accused you of being a duplicate of this issue: #2097 (comment) <#2097 (comment)> Fortunately you pointed out in the very first line that you are not. However, this may be a good search result landing place for people who did have that issue, so I might as well drop the link here anyway. Did you ever resolve this? You should be able to create Flux keys interactively with flux create secret and they are populated with a known_hosts automatically. This results in a similar outcome to flux bootstrap but has less going on so might be a bit more precise tool for debugging your issue. https://fluxcd.io/docs/cmd/flux_create_secret_git/ I use flux create secret git along with flux create source git and add the secretRef manually. It seems unlikely that the host has more than one SSH key unless it also has, besides a git server, an actual SSH server. Next question: if there are maybe indeed multiple SSH servers, are you certain that there is no custom port number expected in your gitrepository? — Reply to this email directly, view it on GitHub <#2114 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAKDO52OOU4MXVYGAOWJGLUYGLF7ANCNFSM5ISHDLBQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bennyandresen]

I'm also unable to bootstrap on a non-github/gitlab/bitbucket gitrepository (in my case gitea).
Same error message: getting the failed to clone repository: ssh: handshake failed: knownhosts: key mismatch message.

I've already tried flux create secret git and then modifying the secret in the flux-system namespace and adding the working known_hosts string from my private .ssh/known_hosts file that I get a match from when running ssh -v git@host and still no dice.

Trying to immediately use flux create source git after creating a secret also fails, as the cluster isn't yet ready and I get the message

► applying secret with repository credentials
✔ authentication configured
► applying GitRepository source
✗ no matches for kind "GitRepository" in version "source.toolkit.fluxcd.io/v1beta1"

I've also tried to add the full results of ssh-keyscan myhost to the secret which I reference with --secret-name.

Next step would probably to bootstrap using github and then migrate away from github using those manual commands? First time trying flux, here.

[hiddeco]

If the bootstrap process fails at some point, you can theoretically manually modify the known_hosts entry in the Secret it creates, and re-run bootstrap. If the Secret exists, it will not be overwritten but the process will rather assume it contains the correct data.

[mariusrugan]

maybe try this CLI bootstrapping again?

flux install \
  --version=latest \
  --components=source-controller,kustomize-controller,helm-controller,notification-controller \
  --namespace=flux-system \
  --network-policy=true \
  --log-level=info

flux create source git flux-system \
  --git-implementation=go-git \
  --url=ssh://git@git.domain.com:22/user/gitops.git \
  --ssh-key-algorithm=ecdsa \
  --ssh-ecdsa-curve=p521 \
  --branch=main \
  --interval=5m \
  --timeout=5m

flux create kustomization flux-system \
--source=flux-system \
--path="./abracadabra" \
--prune=true \
--interval=5m

[bennyandresen]

@mariusrugan Your manual steps worked! I was able to install and create a source.

As I'm new to flux I don't yet know what is expected in place of ./abracadabra but the svc/source-controller is reconciling, as per the command line parameters and it also pulled the latest sha.

Thanks!

[bennyandresen]

I played around a bit more with it and can create kustomizations. I just don't have the flux-system directory content that one would get from the flux bootstrap command. I'm not going to worry about it for now, but flux basically works for me.
(The boostrap command still fails, even though install and create source work)

Thanks again!

[mariusrugan]

take a look at a wonderful community around k8s & flux,
https://github.com/k8s-at-home/awesome-home-kubernetes

and especially https://github.com/k8s-at-home/template-cluster-k3s
which is a flux2 repo outline which you can study and apply for your infra.

my "abracadabra" is exactly that - the entry point for flux2 managing itself:
cluster/base

[carlbordum]

I think this may be an issue with the Generic Git Server implementation. I'm not sure exactly what solved my issue, but I had similar ssh errors. Two things that helped:

1. following @mariusrugans instructions above, but using --private-key-file=~/.ssh/id_rsa with a keypair already on the git server and,
2. using the FULL path of the git repo on the server, e.g. ssh://git@git.example.com/home/git/myrepo.git.

Hope this helps someone out there :)

[carlbordum]

In fact, with the --private-key-file-flag and the full repo path, my flux bootstrap git ... command works again 🎉

[GZYproduction]

I had the same problem, but just running the flux bootstrap git ... with the --private-key-file option didnt solve it for me.

What did solve the problem is actually a funny bug:

1. When the entry in the .ssh/known_hosts was made by my ssh client the flux bootsrap failed.
2. When the known_hosts entry was made by my git client with like git clone <repo> then the command worked again.

So maybe this will help someone in the future:

1. delete the entry in the .ssh/known_hosts file
2. make a git pull <repo> somewhere and accept the Are you sure you want to continue connecting (yes/no/[fingerprint])?
3. flux bootstrap git ... should work again

[ackers-bud]

I had this as well, what tripped me up, was the additional hosts based on IP. I had a bunch of github.com IP's in the known hosts file and it was using these. Solved by taking them all out.