[GitHub] Handshake failed: knownhosts: key mismatch #490

pkit · 2021-11-16T18:34:08Z

Started getting these errors out of the blue on all clusters.

{"level":"error","ts":"2021-11-16T18:21:07.474Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-system","namespace":"flux-system","error":"unable to clone 'ssh://git@github.com/user/repository', error: ssh: handshake failed: knownhosts: key mismatch"}

Doing find -name known_hosts in the pod produces nothing.
Restarting the pod = same error immediately.
What's going on, where's the known_hosts file?

The text was updated successfully, but these errors were encountered:

stefanprodan · 2021-11-16T18:37:16Z

What's going on, where's the known_hosts file?

The known_hosts file is in the same secret as the SSH key, please see the docs here https://fluxcd.io/docs/components/source/gitrepositories/#ssh-authentication

stefanprodan · 2021-11-16T18:42:28Z

I'm getting the same error on my cluster:

✗ GitRepository reconciliation failed: 'unable to clone 'ssh://git@github.com/stefanprodan/my-demo-fleet': ssh: handshake failed: knownhosts: key mismatch'

Looks like an issue with GitHub host keys.

kmannuz · 2021-11-16T18:42:33Z

I am also seeing this error in the last 30 minutes on 3 clusters that had been previously working fine

kingdonb · 2021-11-16T18:45:14Z

According to: https://github.blog/2021-09-01-improving-git-protocol-security-github/

Today is the day that host keys get rotated at GitHub. There are two new host keys in the blog post, one for ECDSA and another for Ed25519.

stefanprodan · 2021-11-16T18:49:22Z

Ok so rotating the SSH key fixes it.

Before:

$ k -n flux-system get secret flux-system -o json | jq '.data | map_values(@base64d)'
{
  "identity": "-----BEGIN PRIVATE KEY-----\n",
  "identity.pub": "ecdsa-sha2-nistp384 \n",
  "known_hosts": "github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ=="
}

After:

{
  "identity": "-----BEGIN PRIVATE KEY-----\n",
  "identity.pub": "ecdsa-sha2-nistp384 \n",
  "known_hosts": "github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg="
}

pkit · 2021-11-16T18:49:37Z

The known_hosts file is in the same secret as the SSH key, please see the docs here https://fluxcd.io/docs/components/source/gitrepositories/#ssh-authentication

Cool, thanks, but I do see the "old" keys when doing keyscan on the nodes.
Somehow only the pods see the "new" ones.
It makes sense though.

stefanprodan · 2021-11-16T18:55:44Z

GitHub has changed its SSH host keys from DSA to ECDSA!
https://github.blog/2021-09-01-improving-git-protocol-security-github/

To fix the key mismatch error, you have two options:

Update the known_hosts in the flux-system secret with the ecdsa-sha2-nistp25 value:

github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=

Or rotate the SSH keys with flux boostrap like so:

delete the deploy key secret from your cluster kubectl -n flux-system delete secret flux-system
rerun flux bootstrap github with the same arguments as before
Flux will generate the secret with ecdsa-sha2 SSH key and Host key

pkit · 2021-11-16T19:18:23Z

Updated known_hosts in flux-system secret manually everywhere.
Seems to work now.

seh · 2021-11-16T19:21:49Z

If you'd like a short program to do it:

#!/usr/bin/env bash

set -e -u -o pipefail

# NB: The Ed25519-format key does not work with Flux.
for secret_name in flux-system repo-2 repo-3; do
  kubectl --namespace=flux-system \
          patch secret "${secret_name}" \
          --patch='
stringData:
  known_hosts: >
    github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg='
done

kubectl --namespace=flux-system rollout restart deployment source-controller
kubectl --namespace=flux-system rollout status deployment/source-controller --watch

brianpham · 2021-11-16T19:37:18Z

Confirmed. Working for us now as well after deleting the secret and bootstrapping again.

stefanprodan · 2021-11-16T19:58:14Z

@seh the secret is not mounted inside source-controller, instead the controller reads the secret from Kubernetes API before each Git operations. I don't think you need rollout restart.

seh · 2021-11-16T19:59:47Z

I was finding that it sits in what appears to be due to a backed-off timer, such that it won't try again for a while after several consecutive failures, but restarting it caused it to try again immediately.

ellieayla · 2021-11-16T21:54:36Z

Variant on the above script: https://gist.github.com/ellieayla/76352313c4f5939db6d2268fb70b0d48

Then either wait or request each GitRepository to reconcile.

poteat · 2021-11-16T23:25:16Z

Confirm that we are getting this on our cluster as well suddenly.

ellieayla · 2021-11-16T23:30:15Z

Note with libgit2, the reported error is unable to clone: Certificate ala #397 and #433.

Update the documentation and example to use the new SSH public key that GitHub deployed after Nov 16th 2021. See also fluxcd/source-controller#490 Signed-off-by: Alessandro Degano <a.degano@gmail.com>

ghost · 2021-11-17T10:14:51Z

@stefanprodan maybe add to the comment that if you edit the secrets manually, you should restart the source-controller after updating the secret, otherwise source-controller might overwrite the secret with the old values.

We've stopped the source-controller before updating the secrets and then started it again just to be safe:

kubectl scale deploy/source-controller --replicas=0

update the secrets

kubectl scale deploy/source-controller --replicas=1

Edit: the old ssh-rsa value gets added back somehow. Maybe kustomize-controller also needs to be restarted.

stefanprodan · 2021-11-17T10:19:34Z

otherwise source-controller might overwrite the secret with the old values.

source-controller doesn't alter secrets. It can't even do that, our RBAC allows the controller read-only access to secrets.

stefanprodan · 2021-11-17T10:21:18Z

Edit: the old ssh-rsa value gets added back somehow. Maybe kustomize-controller also needs to be restarted.

You clearly don't use bootstrap or you've stored the SSH keys in Git. If so, then update the secret in Git as well.

rtjfarrimond · 2021-11-17T10:32:45Z

Unfortunately, this was a predictable incident. It felt wrong to me, as a Flux user, to be providing a known hosts entry as part of the terraform bootstrap process (from this example) for precisely this reason.

To prevent another incident of similar scale in the future, why not give the source-controller the responsibility of maintaining the known hosts file? Presumably given the urls of the sources it has to reconcile it should be fairly straight forward to use something like ssh-keyscan to keep the file up to date?

stefanprodan · 2021-11-17T10:36:18Z

It felt wrong to me, as a Flux user, to be providing a known hosts entry as part of the bootstrap process for precisely this reason.

Bootstrap does no such thing, Flux itself generates the known_hosts entries. As a Flux user, you are never asked to provide host keys.

sebastian-dyroff · 2021-11-17T10:38:38Z

Are multiple known_hosts with different algorithms supported by the go-git implementation?

rtjfarrimond · 2021-11-17T10:42:33Z

Bootstrap does no such thing, Flux itself generates the known_hosts entries. As a Flux user, you are never asked to provide host keys.
@stefanprodan this example from the flux terraform provider examples certainly does.

stefanprodan · 2021-11-17T10:44:19Z

@rtjfarrimond I was referring to flux bootstrap not Terraform.

rtjfarrimond · 2021-11-17T10:46:51Z

I understand, but to be clear, in my original comment I was referring to the terraform bootstrap process. Updated the original comment to reflect this.

hiddeco · 2021-11-17T10:47:27Z

To prevent another incident of similar scale in the future, why not give the source-controller the responsibility of maintaining the known hosts file?

How can a known_hosts file, that is used as a trust storage, be automatically maintained by a service? That would render the known_hosts useless and allow any MITM-attacks to happen.

ghost · 2021-11-17T10:49:19Z

We have two git sources, flux-system and flux-manifests. We've updated the known_hosts for both but for flux-manifests the known_hosts keeps getting replaced with the ssh-rsa key:

{
  "level": "debug",
  "ts": "2021-11-17T10:28:10.304Z",
  "logger": "events",
  "msg": "Normal",
  "object": {
    "kind": "Kustomization",
    "namespace": "flux-system",
    "name": "flux-system",
    "uid": "138b16f7-ca30-458e-a0b1-811b2900fa2c",
    "apiVersion": "kustomize.toolkit.fluxcd.io/v1beta2",
    "resourceVersion": "189896097"
  },
  "reason": "info",
  "message": "Secret/flux-system/flux-manifests configured"
}

Is known_hosts getting updated by the libgit2 callback ?

ghost · 2021-11-17T10:51:40Z

Sorry, my bad. It looks like we have the secrets for flux-manifests in Git and flux is just reconciling the secrets.

hiddeco · 2021-11-17T10:52:13Z

The Secret files are not managed or written to by any of the controllers, but only used for read operations. If something is overwriting your Secret, it must come from something within your configuration.

rtjfarrimond · 2021-11-17T11:01:59Z

How can a known_hosts file, that is used as a trust storage, be automatically maintained by a service? That would render the known_hosts useless and allow any MITM-attack to happen.

If the some process were to update the known_hosts runs on the same box with the same user that uses the known_hosts file, where would the vector for a MITM be?

hiddeco · 2021-11-17T11:13:39Z

By it automatically accepting the offered keys.

If your network is compromised and hostname.com suddenly starts serving traffic from compromised.com with a different host key, which is then automatically excepted by the controller, checking the host key no longer has any value.

rtjfarrimond · 2021-11-17T11:15:15Z

If your network is compromised and hostname.com suddenly starts serving traffic from compromised.com with a different host key, which is then automatically excepted by the controller, checking the host key no longer has any value.

Yep, that makes sense, I withdraw my bad idea! Thanks :)

rtjfarrimond · 2021-11-17T11:58:37Z

@stefanprodan Here is a PR to update the known_hosts in the terraform example I linked earlier.

seh · 2021-11-17T13:12:29Z

Two things lengthened my fixing of this problem across ~20 clusters:

I forgot that I generate Secret manifests for additional GitRepository objects with kustomize, and deploy them with Flux.
The old known hosts entry was in my VCS, and I forgot that I needed to update it there too, as opposed to just in the Secret objects in the Kubernetes clusters.
Flux got stuck in a "deadly embrace" due to the Kustomization "spec.wait" field being true on some of my Kustomizations.
- A Kustomization was waiting on the health of
  - a Kustomization it creates whose GitRepository needed
    - a Secret to be updated by the first Kustomization which was stuck waiting on the health of
      - ...

I had to patch the top-level Kustomization to set "spec.wait" to false, then force Flux to reconcile it. It took many tries before the health checking timeouts expired and Flux finally both updated and then started using the new Secret "data.known_hosts" field value.

Update the documentation and example to use the new SSH public key that GitHub deployed after Nov 16th 2021. See also fluxcd/source-controller#490 Also update the suggested algorithm for the SSH key to use with GitHub to ensure (compability with libgit2)[https://github.blog/2021-09-01-improving-git-protocol-security-github/#libgit2-and-other-git-clients]. Signed-off-by: Alessandro Degano <a.degano@gmail.com>

devozs · 2021-11-17T16:30:35Z

GitHub has changed its SSH host keys from DSA to ECDSA! https://github.blog/2021-09-01-improving-git-protocol-security-github/

To fix the key mismatch error, you have two options:

Update the known_hosts in the flux-system secret with the ecdsa-sha2-nistp25 value:
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
Or rotate the SSH keys with flux boostrap like so:

delete the deploy key secret from your cluster kubectl -n flux-system delete secret flux-system

rerun flux bootstrap github with the same arguments as before

Flux will generate the secret with ecdsa-sha2 SSH key and Host key

Thanks for the suggestion, in my case i also had to:

deleting the secret was not enough, it was also required to delete the git source
flux delete source git flux-system
who also having additional repo (for example referenced from flux-infra repo): remember to bootstrap this repo as well, update the persisted flux secret yaml and as mentioned above to delete the secret and git source

gautamr · 2021-11-17T17:11:49Z

GitHub has changed its SSH host keys from DSA to ECDSA! https://github.blog/2021-09-01-improving-git-protocol-security-github/

To fix the key mismatch error, you have two options:

Update the known_hosts in the flux-system secret with the ecdsa-sha2-nistp25 value:
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
Or rotate the SSH keys with flux boostrap like so:

delete the deploy key secret from your cluster kubectl -n flux-system delete secret flux-system

rerun flux bootstrap github with the same arguments as before

Flux will generate the secret with ecdsa-sha2 SSH key and Host key

worked for us

oscaromeu · 2021-11-18T07:57:31Z

GitHub has changed its SSH host keys from DSA to ECDSA! https://github.blog/2021-09-01-improving-git-protocol-security-github/

Or rotate the SSH keys with flux boostrap like so:

delete the deploy key secret from your cluster kubectl -n flux-system delete secret flux-system

rerun flux bootstrap github with the same arguments as before

Flux will generate the secret with ecdsa-sha2 SSH key and Host key

Worked for me as well, thanks! 👯

cbyad · 2021-11-18T11:33:56Z

GitHub has changed its SSH host keys from DSA to ECDSA! https://github.blog/2021-09-01-improving-git-protocol-security-github/
To fix the key mismatch error, you have two options:
Update the known_hosts in the flux-system secret with the ecdsa-sha2-nistp25 value:
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
Or rotate the SSH keys with flux boostrap like so:

delete the deploy key secret from your cluster kubectl -n flux-system delete secret flux-system

rerun flux bootstrap github with the same arguments as before

Flux will generate the secret with ecdsa-sha2 SSH key and Host key
worked for us too Thks!

kaaboaye · 2021-11-18T18:45:23Z

In my case bootstrap fails to create new secret

flux bootstrap github --owner=USER --repository=REPO --branch=flux2 --personal --path=clusters/CLUSTER --components-extra=image-reflector-controller,image-automation-controller

► connecting to github.com
► cloning branch "flux2" from Git repository "https://github.com/USER/REPO.git"
✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ component manifests are up to date
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
► generating source secret
✔ public key: ecdsa-sha2-nistp384 key 
✗ multiple errors occurred: 
- POST https://api.github.com/repos/USER/REPO/keys: 404 Not Found []
- the requested resource was not found

Switching from ssh to https helped

stefanprodan · 2021-11-18T23:36:45Z

@kaaboaye your user token doesn’t have permission to create deploy keys, you need to be a repo admin.

ninja9k1 · 2021-11-24T00:52:01Z

I am having a very similar, if not the same, error while setting up gitops on my local kind cluster following this tutorial:
https://docs.gitops.weave.works/docs/getting-started/

{"level":"error","ts":"2021-11-16T18:21:07.474Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-system","namespace":"flux-system","error":"unable to clone 'ssh://git@github.com/user/repository', error: ssh: handshake failed: knownhosts: key mismatch"}

This is a brand new instantiation which I have just fired up a few minutes ago as of this writing. kubectl -n flux-system delete secret flux-system does not work as this is not done through flux bootstrap. Any ideas?

sbernheim · 2022-03-16T02:32:42Z

I am having a very similar, if not the same, error while setting up gitops on my local kind cluster following this tutorial: https://docs.gitops.weave.works/docs/getting-started/
{"level":"error","ts":"2021-11-16T18:21:07.474Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-system","namespace":"flux-system","error":"unable to clone 'ssh://git@github.com/user/repository', error: ssh: handshake failed: knownhosts: key mismatch"}
This is a brand new instantiation which I have just fired up a few minutes ago as of this writing. kubectl -n flux-system delete secret flux-system does not work as this is not done through flux bootstrap. Any ideas?

@ninja9k1 - I assume by now that you've resolved this issue for your local gitops installation, but I'll add a response to this Issue in case anyone else finds it and needs the same solution.

The gitops CLI uses your local user's ~/.ssh/known_hosts file as the source for this key, and this error generally means that you need to remove the old RSA host key and add the new ECDSA host key in that file.

This command should remove the existing key:

ssh-keygen -R github.com

You can then either use this command to insert the new key without actually trying to SSH to GitHub:

ssh-keyscan -t ecdsa github.com >> ~/.ssh/known_hosts

Or start an SSH connection to github.com and let GitHub disconnect you after the connection succeeds:

ssh git@github.com

olivercp3 · 2022-12-23T01:51:05Z

An error is still reported ( Handshake failed: knownhosts: key mismatch ) when a new ecdsa hostkey is generated
my bootstrap command is:
flux bootstrap git --url=ssh://git@XXX.com/DP/k8s-deploy-2.git --private-key-file=/root/.ssh/id_ecdsa --branch dev
Secret generated by bootstrap, why knownhosts: key still mismatch

braadaaay · 2024-01-25T07:50:35Z

I managed to get SFTP working, see here on #2948

stefanprodan changed the title ~~handshake failed: knownhosts: key mismatch~~ [GitHub] Handshake failed: knownhosts: key mismatch Nov 16, 2021

aledegano mentioned this issue Nov 17, 2021

(docs) Update GitHub's host key fluxcd/terraform-provider-flux#225

Merged

rtjfarrimond mentioned this issue Nov 17, 2021

Update GitHub known host key for the GKE example fluxcd/terraform-provider-flux#226

Merged

hannah38 mentioned this issue Nov 17, 2021

image automation fix - github key mismatch hmcts/cnp-flux-config#12767

Merged

v-ctiutiu mentioned this issue Nov 17, 2021

Update FluxCD Terraform module to use new GitHub SSH public key digitalocean/container-blueprints#16

Closed

pjbgf mentioned this issue Jan 25, 2022

Bump libgit2 to 1.3.0 and git2go to V33. #557

Closed

pjbgf mentioned this issue Feb 8, 2022

Upgrade libgit2 to 1.3.0 and git2go to V33 #573

Merged

DanielRejniak mentioned this issue Mar 22, 2022

error adding app: failed to clone application repo: failed cloning user repo: ssh://git@github.com/user/podinfo-deploy.git: ssh: handshake failed: knownhosts: key mismatch weaveworks/weave-gitops#1132

Closed

11 tasks

[GitHub] Handshake failed: knownhosts: key mismatch #490

[GitHub] Handshake failed: knownhosts: key mismatch #490

Comments

pkit commented Nov 16, 2021

stefanprodan commented Nov 16, 2021

stefanprodan commented Nov 16, 2021

kmannuz commented Nov 16, 2021

kingdonb commented Nov 16, 2021

stefanprodan commented Nov 16, 2021 • edited Loading

pkit commented Nov 16, 2021

stefanprodan commented Nov 16, 2021 • edited Loading

pkit commented Nov 16, 2021

seh commented Nov 16, 2021 • edited Loading

brianpham commented Nov 16, 2021

stefanprodan commented Nov 16, 2021

seh commented Nov 16, 2021

ellieayla commented Nov 16, 2021 • edited Loading

poteat commented Nov 16, 2021

ellieayla commented Nov 16, 2021

ghost commented Nov 17, 2021 • edited by ghost Loading

stefanprodan commented Nov 17, 2021

stefanprodan commented Nov 17, 2021

rtjfarrimond commented Nov 17, 2021 • edited Loading

stefanprodan commented Nov 17, 2021 • edited Loading

sebastian-dyroff commented Nov 17, 2021

rtjfarrimond commented Nov 17, 2021

stefanprodan commented Nov 17, 2021

rtjfarrimond commented Nov 17, 2021 • edited Loading

hiddeco commented Nov 17, 2021 • edited Loading

ghost commented Nov 17, 2021

ghost commented Nov 17, 2021

hiddeco commented Nov 17, 2021

rtjfarrimond commented Nov 17, 2021 • edited Loading

hiddeco commented Nov 17, 2021

rtjfarrimond commented Nov 17, 2021 • edited Loading

rtjfarrimond commented Nov 17, 2021

seh commented Nov 17, 2021

devozs commented Nov 17, 2021

gautamr commented Nov 17, 2021

oscaromeu commented Nov 18, 2021 • edited Loading

cbyad commented Nov 18, 2021

kaaboaye commented Nov 18, 2021 • edited Loading

stefanprodan commented Nov 18, 2021 • edited Loading

ninja9k1 commented Nov 24, 2021

sbernheim commented Mar 16, 2022

olivercp3 commented Dec 23, 2022 • edited Loading

braadaaay commented Jan 25, 2024

stefanprodan commented Nov 16, 2021 •

edited

Loading

stefanprodan commented Nov 16, 2021 •

edited

Loading

seh commented Nov 16, 2021 •

edited

Loading

ellieayla commented Nov 16, 2021 •

edited

Loading

ghost commented Nov 17, 2021 •

edited by ghost

Loading

rtjfarrimond commented Nov 17, 2021 •

edited

Loading

stefanprodan commented Nov 17, 2021 •

edited

Loading

rtjfarrimond commented Nov 17, 2021 •

edited

Loading

hiddeco commented Nov 17, 2021 •

edited

Loading

rtjfarrimond commented Nov 17, 2021 •

edited

Loading

rtjfarrimond commented Nov 17, 2021 •

edited

Loading

oscaromeu commented Nov 18, 2021 •

edited

Loading

kaaboaye commented Nov 18, 2021 •

edited

Loading

stefanprodan commented Nov 18, 2021 •

edited

Loading

olivercp3 commented Dec 23, 2022 •

edited

Loading