ArgoCD Repo Server stops pulling git repositories due to Azure Devops Repos current sunset SSH-RSA strategy #17634

deB4SH · 2024-03-27T06:53:41Z

Hi all,
Microsoft published a blogpost Feb 15th 2024 to sunset the ssh-rsa support and wants to migrate to rsa-sha2-256/512.
Blog Post: https://devblogs.microsoft.com/devops/ssh-rsa-deprecation/

Based on their schedule everyone that still uses their service should be in Phase 2 where an throttling/delay is in place and an error is shown with following message.

“ssh-rsa is about to be deprecated and your request has been throttled. Please use rsa-sha2-256 or rsa-sha2-512 instead. Your session will continue automatically. For more details see https://devblogs.microsoft.com/devops/ssh-rsa-deprecation.”

This error is also shown within argo.

After searching a bit - it seems like that golang/crypto already support rsa-sha2-256/512 but sadly starting from v0.21.0.
(golang/crypto@6fad3dfc)

Argo seems to use v0.19.0

argo-cd/go.mod

Line 85 in 1bddee2

golang.org/x/crypto v0.19.0

Also found a relevant issue in this regard: #7600

Checklist:

I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
I've included steps to reproduce the bug.
I've pasted the output of argocd version.

Describe the bug
Argo is currently unable to pull git repositories provided by azure devops repos and stops after receives the delay error.

To Reproduce
Pull a repository from their service with a ssh key.

Expected behavior
Argo pulls changes or state from git repository.

Version

argocd@argocd-core-server-588df95858-5jcc7:~$ argocd version
argocd: v2.9.3+6eba5be
  BuildDate: 2023-12-01T23:05:50Z
  GitCommit: 6eba5be864b7e031871ed7698f5233336dfe75c7
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64

Logs

Logs from repo-server:

Stream closed EOF for argocd/argocd-core-repo-server-6b48b5bc4b-t5xfj (copyutil)
repo-server time="2024-03-27T06:40:28Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2024-03-27T06:40:28Z" grpc.time_ms=0.019 span.kind=server system=grpc
repo-server time="2024-03-27T06:40:34Z" level=error msg="finished unary call with code Unknown" error="unknown error: remote: Command git-upload-pack: ssh-rsa is about to be deprecated and your request has been throttled. Please use rsa-sha2-256 or rsa-sha2-512 instead. Your session will continue automatically. For more details see https://aka.ms/ado-ssh-rsa-deprecation." grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-03-27T06:40:29Z" grpc.time_ms=5616.032 span.kind=server system=grpc

The text was updated successfully, but these errors were encountered:

maxkaemmerer · 2024-04-08T09:07:11Z

We are having the same issue and currently can't sync our applications anymore.

Any progress on this?

mbu147 · 2024-04-08T09:27:57Z

We are having the same issue and currently can't sync our applications anymore.

Any progress on this?

+1

{
    "Version": "v2.10.4+f5d63a5",
    "BuildDate": "2024-03-18T08:09:23Z",
    "GitCommit": "f5d63a5c77d2e804e51ef94bee3db441e0789d00",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.3+gf03cc04",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}

crenshaw-dev · 2024-04-08T19:25:58Z

My understanding is that this isn't an issue with golang's crypto library. Rather it has to do with the hash algorithms ssh accepts in Ubuntu 22.04 (the version Argo CD images are built on).

It looks like Ubuntu 22.04 does support the listed algorithms (based on this guide), but maybe neither algorithm is enabled by default in the handshake. You might have to explicitly enable that host key algorithm.

$ ssh -Q sig 
ssh-ed25519
sk-ssh-ed25519@openssh.com
ssh-rsa
rsa-sha2-256
rsa-sha2-512
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
webauthn-sk-ecdsa-sha2-nistp256@openssh.com

maartengo · 2024-04-11T12:12:07Z

I'm not too familiar with all the internal workings of SSH, but would it be a solution to change the known_hosts entry for azure devops to: ssh.dev.azure.com rsa-sha2-512 instead of ssh.dev.azure.com ssh-rsa?

I've changed this within my setup using configs.ssh.knownHosts. I'm not sure if this helps during the brownout, but it seems to be working at the moment.

maartengo · 2024-04-15T08:44:20Z

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround

Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa

I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.

siawkz · 2024-04-16T07:05:03Z

argocd repo add with --insecure-skip-server-verification flag will bypass the verification which causes the issue.

maartengo · 2024-04-16T07:46:15Z

My previous 'workaround' doesn't work, sadly. Is /home/argocd/.ssh/config not the right place to put the ssh config?

Disabling the server verification doesn't sit well with me either, so I hope there is some other solution.

siawkz · 2024-04-16T07:49:02Z

My previous 'workaround' doesn't work, sadly. Is /home/argocd/.ssh/config not the right place to put the ssh config?

Disabling the server verification doesn't sit well with me either, so I hope there is some other solution.

Do you encounter issue pulling git repositories directly on the host machine?

maartengo · 2024-04-16T08:13:16Z

Do you encounter issue pulling git repositories directly on the host machine?

I haven't tested this, but I can at least say that the config is used with a regular ssh ssh.dev.azure.com call; if I input invalid config (e.g. foobar as HostkeyAlgorithms) then the ssh command fails.

siawkz · 2024-04-16T08:19:48Z

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd.
I had to

Generate new public private key ssh-keygen -t rsa-sha2-512
Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

zamedic · 2024-04-16T08:28:16Z

I can confirm what is being mentioned here. During the microsoft brownouts, we are able to work locally with Azure Devops GIT using SSH keys, however all of our argo instances fail to connect citing the error mentioned above.

AleksuKey · 2024-04-16T08:42:43Z

We just got into this issue right now.
We did the following steps to stop having the "You’re using ssh-rsa that is about to be deprecated and your request has been blocked intentionally." error:

Created a new pair of SSH keys with SHA2 512 with:

ssh-keygen -f key_file_name -t rsa-sha2-512

Replaced the keys in ArgoCD and Azure DevOps.
Changed known hosts values in ArgoCD chart values. The value is configs.ssh.knownHosts. We replaced the default Azure DevOps known hosts with the following ones:

ssh.dev.azure.com rsa-sha2-512 AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H
vs-ssh.visualstudio.com rsa-sha2-512 AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

Hope this helps as a temporal workaround.

maxkaemmerer · 2024-04-16T08:45:09Z

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround
Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa
I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.

This seems to work for us.

Change the SSH key to sha2_512.

Kustomize Patch for the Repo Server:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - mountPath: /home/argocd/.ssh
              name: argo-server-ssh-config
      volumes:
          - name: argo-server-ssh-config
            configMap:
              name: argo-server-ssh-config

ConfigMap for the ssh-config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-server-ssh-config
data:
  config: |
    Host *
        HostkeyAlgorithms -ssh-rsa
        PubkeyAcceptedAlgorithms -ssh-rsa

Note: It is possible that this just worked because the brownout ended.

rouke-broersma · 2024-04-16T09:41:53Z

@maxkaemmerer @AleksuKey How long ago did you test this? Note that the brownout has ended so it is possible that this is the reason it now works, instead of the fixes actually working. Neither of these two fixes worked for us.

@siawkz

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd. I had to

Generate new public private key ssh-keygen -t rsa-sha2-512

Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

This works because you are no longer verifying SSL, so ArgoCD is ignoring SSL errors. This is not an appropriate fix, and I would not even suggest it as a workaround for production systems.

maxkaemmerer · 2024-04-16T09:46:36Z

@maxkaemmerer @AleksuKey How long ago did you test this? Note that the brownout has ended so it is possible that this is the reason it now works, instead of the fixes actually working. Neither of these two fixes worked for us.

@siawkz

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd. I had to

Generate new public private key ssh-keygen -t rsa-sha2-512

Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

This works because you are no longer verifying SSL, so ArgoCD is ignoring SSL errors. This is not an appropriate fix, and I would not even suggest it as a workaround for production systems.

@rouke-broersma

About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)

We are also very hesitant to add the --insecure flag.

This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.

We are on ArgoCD version v2.10.6 btw.

bartoszpyrek · 2024-04-16T09:48:42Z

On the ArgoCD 2.8.4 version rsa-sha2-256 is working, but the important thing to check - is in the k8s secret with sshPrivateKey for Azure DevOps if appropriately put the new lines, because when we put this private key in the Azure Key Vault as a secret then "-----END OPENSSH PRIVATE KEY-----" part was not in the new line but it was as a part of the key. After fixing this, ArgoCD can sync properly with Azure DevOps repos.

patst · 2024-04-16T09:54:49Z

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround
Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa
I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.
This seems to work for us.

Change the SSH key to sha2_512.

Kustomize Patch for the Repo Server:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - mountPath: /home/argocd/.ssh
              name: argo-server-ssh-config
      volumes:
          - name: argo-server-ssh-config
            configMap:
              name: argo-server-ssh-config
ConfigMap for the ssh-config:
apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-server-ssh-config
data:
  config: |
    Host *
        HostkeyAlgorithms -ssh-rsa
        PubkeyAcceptedAlgorithms -ssh-rsa
Note: It is possible that this just worked because the brownout ended.

We have this config since a few weeks ago.
Nevertheless, we observed the clone errors today.

I would expect you will see the errors appearing again during the next brownout period. :-/

rouke-broersma · 2024-04-16T10:03:10Z

@rouke-broersma

About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)

We are also very hesitant to add the --insecure flag.

This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.

We are on ArgoCD version v2.10.6 btw.

For our environments the brownout seems to have ended at about 10AM CEST (2 hours ago) so it could very well be that you unfortunately were testing after the brownout.

I've been looking through the code, it looks to me like the /home/argocd/.ssh/config is not being used, because a different SSH config dir is being set:

argo-cd/common/common.go

Line 75 in 4d527f8

DefaultPathSSHConfig = "/app/config/ssh"

This makes sense, because this path is also where the argocd-ssh-known-hosts-cm is mounted by the argocd helm chart:

- mountPath: /app/config/ssh
     name: ssh-known-hosts

So it seems to me like we should extend the argocd-ssh-known-hosts-cm with the config file, instead of mounting a new configmap, so the config file is placed in the actual used ssh config folder.

AleksuKey · 2024-04-16T10:07:26Z

@rouke-broersma
About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)
We are also very hesitant to add the --insecure flag.
This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.
We are on ArgoCD version v2.10.6 btw.

For our environments the brownout seems to have ended at about 10AM CEST (2 hours ago) so it could very well be that you unfortunately were testing after the brownout.

Probably the same happened to us and that's why we saw it solved...

maxkaemmerer · 2024-04-16T10:12:51Z

@rouke-broersma
About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)
We are also very hesitant to add the --insecure flag.
This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.
We are on ArgoCD version v2.10.6 btw.

For our environments the brownout seems to have ended at about 10AM CEST (2 hours ago) so it could very well be that you unfortunately were testing after the brownout.

I've been looking through the code, it looks to me like the /home/argocd/.ssh/config is not being used, because a different SSH config dir is being set:

argo-cd/common/common.go

Line 75 in 4d527f8

DefaultPathSSHConfig = "/app/config/ssh"

This makes sense, because this path is also where the argocd-ssh-known-hosts-cm is mounted by the argocd helm chart:
- mountPath: /app/config/ssh
     name: ssh-known-hosts
So it seems to me like we should extend the argocd-ssh-known-hosts-cm with the config file, instead of mounting a new configmap, so the config file is placed in the actual used ssh config folder.

We had a syntax error in the ssh-config mounted at /home/argocd/.ssh/config and that error showed up in the Argo UI, so I'd assume it is being used.

Very possible that we tested after the brownout however.

FlorisFeddema · 2024-04-18T09:37:04Z

I did some testing yesterday and believe the issue might be in Argo.

It seems like the git implementation used in argo does some parts with the Go user agent and some with the OpenSSH user agent.
Both processes probably need to use the rsa-sha2-256/512 signature to be compliant with Azure DevOps. It might also explain why changing git config does not solve the issue, maybe not both processes use the same git config location. (maybe if both /home/argocd/.ssh/config and /app/config/ssh are updated it might work?)

To check which of the 2 processes is using the wrong signature I used the reposerver container to manually do a git fetch (uses OpenSSH user agent) with extra verbosity. (GIT_SSH_COMMAND="ssh -vvv -i ./../key -o UserKnownHostsFile=/app/config/ssh/ssh_known_hosts" git fetch).

This results in the following (removed a lot of content that is not ):

...
OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256
...
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: KEX algorithms: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ssh-rsa,rsa-sha2-256,rsa-sha2-512
debug2: ciphers ctos: aes256-cbc,aes192-cbc,aes128-cbc,aes128-ctr,aes256-ctr
debug2: ciphers stoc: aes256-cbc,aes192-cbc,aes128-cbc,aes128-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha2-256,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256,hmac-sha2-512
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: Server host key: ssh-rsa SHA256:ohD8VZEXGWo6Ez8GSEJQ9WpafgLFsOfLOtGGQCQo6Og
debug3: record_hostkey: found key type RSA in file /app/config/ssh/ssh_known_hosts:13
debug3: load_hostkeys_file: loaded 1 keys from ssh.dev.azure.com
debug3: record_hostkey: found key type RSA in file /etc/ssh/ssh_known_hosts:13
debug3: load_hostkeys_file: loaded 1 keys from ssh.dev.azure.com
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'ssh.dev.azure.com' is known and matches the RSA host key.
debug1: Found key in /app/config/ssh/ssh_known_hosts:13
debug1: Will attempt key: ./../key RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo explicit
debug1: kex_input_ext_info: server-sig-algs=<ssh-rsa,rsa-sha2-256,rsa-sha2-512>
debug1: Next authentication method: publickey
debug1: Offering public key: ./../key RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo explicit
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: ./../key RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo explicit
debug3: sign_and_send_pubkey: using publickey with RSA SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:EdOva7WBW6rsAMd5e8pGXNi5X+bqqXBrhAaNupIpkWo
Authenticated to ssh.dev.azure.com ([40.74.28.31]:22) using "publickey".
...

It shows that it uses rsa-sha2-512, and thus should work with Azure DevOps during the brownouts and after the removal of ssh-rsa, but since we have not seen any other brownouts I could not run this during brownout.
Since openSSH uses the correct signature, it probably is the go-git client that does not use the correct signature thus breaking ArgoCD with Azure DevOps.

While for Github the issue was to use an elliptic key, this should be fixed in argo or the go-git client.
It also looks like Flux users also had problems with host key algoritms. They added a flag for this to the project (fluxcd/source-controller#711).

maxkaemmerer · 2024-04-25T12:54:16Z

@maxkaemmerer @AleksuKey How long ago did you test this? Note that the brownout has ended so it is possible that this is the reason it now works, instead of the fixes actually working. Neither of these two fixes worked for us.
@siawkz

For my case, git pull works with ssh ssh.dev.azure.com but didn't work for argocd. I had to

Generate new public private key ssh-keygen -t rsa-sha2-512

Override existing repositories with argocd repo add git@ssh.dev.azure.com:v3/<repo> --ssh-private-key-path ~/.ssh/id_rsa --upsert --insecure-skip-server-verification

Hope it helps.

This works because you are no longer verifying SSL, so ArgoCD is ignoring SSL errors. This is not an appropriate fix, and I would not even suggest it as a workaround for production systems.

@rouke-broersma

About five minutes before I wrote the comment. During the last brownout we tested it with a sha2-512 key and had other errors which do not appear with the approach I commented above. (I do not remember the exact wording on those other errors)

We are also very hesitant to add the --insecure flag.

This is not my area expertise. We just tried the approach with the ssh-config and that seems to have worked, hoping it works for others as well until a proper fix is found.

We are on ArgoCD version v2.10.6 btw.

We are having issues again today, so it seems like this did not infact resolve

I'm now testing with an explicit deny for ssh-rsa in the ssh config of the repo server, the opposite of what is described here; https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/2.1-2.2/#workaround
Host *
    HostkeyAlgorithms -ssh-rsa
    PubkeyAcceptedAlgorithms -ssh-rsa
I suspect that the ssh will work without any config changes once Microsoft is done with the brownout period. It probably currently fails because they still expose the ssh-rsa signature key as 'valid' from their side during brownouts.
This seems to work for us.

Change the SSH key to sha2_512.

Kustomize Patch for the Repo Server:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - mountPath: /home/argocd/.ssh
              name: argo-server-ssh-config
      volumes:
          - name: argo-server-ssh-config
            configMap:
              name: argo-server-ssh-config
ConfigMap for the ssh-config:
apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-server-ssh-config
data:
  config: |
    Host *
        HostkeyAlgorithms -ssh-rsa
        PubkeyAcceptedAlgorithms -ssh-rsa
Note: It is possible that this just worked because the brownout ended.

We are having issues again today so it seems like it just worked for us because the brownout had ended

FlorisFeddema · 2024-04-25T13:19:04Z

Looks like the brownout is over again...

danijam · 2024-04-25T14:38:25Z

Joining this thread as we also just recently regenerated our SSH key as Microsoft advised but then during the next brownout observed the issue. Reading through this I'm not certain we've identified a solution or if we need one (aka once ADO kill ssh-rsa it just works)...

Anyone considered using HTTPS? Any pros or cons I should be aware of over SSH? I'm guess SSH security is considered better asymmetric crypto versus a PAT based http basic auth? Anything else?

patst · 2024-04-25T14:52:03Z

https://devblogs.microsoft.com/devops/ssh-rsa-deprecation/

According to their schedule the brownout times and lengths will increase to 8 and 12 hours per day in the next week(s). That will be very annoying to sit out and hope it will take the correct algorightm afterwards

I just took a look at the code and have no idea how everything is wired together, but these parts look interesting:

https://github.com/argoproj/argo-cd/blob/master/util/git/client.go#L267-L279 where the git ssh client is constructed
+
https://github.com/argoproj/argo-cd/blob/master/common/common.go#L74-L77 where the default paths are defined

I took a look in our reposerver with cat /app/config/ssh/ssh_known_hosts and the result is:

ssh.dev.azure.com rsa-sha2-512 ....

which is what we defined as custom config.

Nevertheless, we are getting the errors. Maybe we need a way to configure custom algorithms here? https://github.com/argoproj/argo-cd/blob/master/util/git/client.go#L267

rouke-broersma · 2024-04-25T15:11:24Z

@patst we are coming to the same conclusions by digging through the code and we hope to test this to confirm and hopefully submit a patch.

aurel4oxand · 2024-04-25T15:34:05Z

On the ArgoCD 2.8.4 version rsa-sha2-256 is working, but the important thing to check - is in the k8s secret with sshPrivateKey for Azure DevOps if appropriately put the new lines, because when we put this private key in the Azure Key Vault as a secret then "-----END OPENSSH PRIVATE KEY-----" part was not in the new line but it was as a part of the key. After fixing this, ArgoCD can sync properly with Azure DevOps repos.

Hi, I'm using ArgoCD v2.7.11

Thanks to @bartoszpyrek I managed to get this working by :

generating a new RSA key with rsa-sha2-512 algorithm : ssh-keygen -t rsa-sha2-512
adding generated public key to my Azure DevOps profile

Public key is of form :

ssh-rsa AAAAB3NzaC1y..... user@host

doing a simple test on ArgoCD Ui

Note the new line in yellow after -----END OPENSSH PRIVATE KEY-----

This worked fine 👌

Also, it seems there's no need to update know-hosts configuration, nor skip server verification

Finally, I updated my secret which contains my SSH private key with the new one, also with the newline at the end, and restarted argocd-repo-server pods

I manually deleted repositories which uses the old SSH key and let ArgoCD re-create them (ArgoCD manages itself its configuration, using App of App pattern)

Now everything is fine, let's see if this works during next days...

Good luck 🤞

rouke-broersma · 2024-04-25T15:40:03Z

@aurel4oxand let me guess you tested this somewhere in the last 40 minutes? The brownout is over :)

zamedic · 2024-04-29T14:00:23Z

In out stack, we use Argo CD to manage Argo CD. When I implemented the fix, Argo CD was failing with the main error, so I could not bump the version. I had to log onto k8s and manually bump the repo server version, after which, stuff appeared to work. I do however see today, that this appears to have been purely coincidental. I am continuing to try and resolve this, however today the brownouts appeared to be even more inconsistent with some fetches working and others not.

Also, I am a complete noob when it comes to how GIT/SSH works when it comes to the transport, so learning alot.

cveld · 2024-04-29T16:44:32Z

How do I reproduce the command that ArgoCD fires at Azure DevOps? Is git-upload-pack easily executed from the commandline?

I tried to reproduce a git operation that uses the ssh-rsa signature but I failed. It keeps defaulting to rsa-sha2-512.

$env:GIT_SSH_COMMAND="ssh -vvv -i keyfile -o IdentitiesOnly=yes"
git pull

results in a verbose ssh log:

...
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
...
debug2: host key algorithms: ssh-rsa,rsa-sha2-256,rsa-sha2-512
...
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
...

Additionally I can confirm that any RSA key works irrespective of the -t parameter. The key generated with ssh-keygen -t ssh-rsa -f ssh-rsa-keyfile also just works. At least assuming the brown out is active 😅

rouke-broersma · 2024-04-29T18:16:57Z

@cveld enabling ssh-rsa in the ssh user config file should be more or less similar to what argocd does. Ie the inverse of this doc: https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops#q-ssh-cannot-establish-a-connection-what-should-i-do

You can't reproduce it by default with an ssh or git client because they disabled ssh-rsa in openssh 1-2 years go.

It's not an exact reproduction because argocd does this in go code instead of using the native client but the approximation is close enough I think.

cveld · 2024-04-29T19:05:20Z

@rouke-broersma thanks! I was able to reproduce the brown out with the following in ~/.ssh/config:

Host ssh.dev.azure.com vs-ssh.visualstudio.com
  HostKeyAlgorithms ssh-rsa

zamedic · 2024-04-30T08:52:46Z

I have multiple Argo CD clusters, on 1 cluster I am running some small changes, that 1 appears to not be experiencing brownouts while the other clusters are experiencing brownouts.

The difference is the go-get version. I ran
go get -u github.com/go-git/go-git/v5 which bumped the module up to v5.12.0

Will need to see whats in that change and deploy my fix to all environments to confirm...

blakepettersson · 2024-04-30T09:04:23Z

So this is a no-go. So the only fix, I think, would be to modify

argo-cd/util/git/ssh.go

Line 54 in 0f11dfb

kexAlgos = DefaultSSHKeyExchangeAlgorithms

to either have a hardcoded list of supported algorithms or to read the supported algorithms from ~/.ssh/config.

@rouke-broersma IMO I think initializing SupportedSSHKeyExchangeAlgorithms from an environment variable would be the way to go. The default would be the same as today. Perhaps it could also be worthwhile having a new configmap to set the environment variable from.

zamedic · 2024-04-30T09:11:35Z

Ok, I believe I have traced down the root cause...
By bumping the go-git, it also bumped https://github.com/skeema/knownhosts to version 1.2.2
this version contains the changes for the new algorithms: skeema/knownhosts@bd8e67e

pimjansen · 2024-04-30T09:31:19Z

Ok, I believe I have traced down the root cause...

By bumping the go-git, it also bumped https://github.com/skeema/knownhosts to version 1.2.2

this version contains the changes for the new algorithms: skeema/knownhosts@bd8e67e

Seems solid. It would break old implementations though but i assume that is ok since ssh-rsa is deprecated and dropped to start with.

But only the ArgoCD maintainers can decide this if it not hits a BC or if they allow it and so on

rouke-broersma · 2024-04-30T09:45:32Z

So this is a no-go. So the only fix, I think, would be to modify

argo-cd/util/git/ssh.go

Line 54 in 0f11dfb

kexAlgos = DefaultSSHKeyExchangeAlgorithms

to either have a hardcoded list of supported algorithms or to read the supported algorithms from ~/.ssh/config.

@rouke-broersma IMO I think initializing SupportedSSHKeyExchangeAlgorithms from an environment variable would be the way to go. The default would be the same as today. Perhaps it could also be worthwhile having a new configmap to set the environment variable from.

The issue is with HostKeyAlgorithms not KeyExchangeAlgoritms.

The issue with an environment variable is that the nativeclient which is used for 90% of argocd git interaction would read from ~/.ssh/config so you would need to add the config to both the ssh client config and the environment variable. Ideally both clients would read from the same source, so this difference between the two clients is removed.

See confusion here for example: #17634 (comment)

I think short term the update of the knownhosts library will fix this issue, but long term the differences between go-git and nativeclient need to be resolved because it seems like argocd maintainers also forget about these differences (understandable, but dangerous since it involves crypto and authentication).

jannfis · 2024-04-30T13:17:59Z

Thanks @zamedic - I've merged your PR.

Before cherry-picking this change into supported release branches and putting out a fix release, can somebody confirm that this actually fixes the issue? For example, by running an image that's been built off the latest code (once the image has been built)? I do not have access to an Azure DevOps Git repository, unfortunately.

jannfis · 2024-04-30T13:35:23Z

Anyone inclined to test a build off the master branch with the fix included, the image is ghcr.io/argoproj/argo-cd/argocd:2.11.0-a63068d0

rouke-broersma · 2024-04-30T13:37:34Z

Anyone inclined to test a build off the master branch with the fix included, the image is ghcr.io/argoproj/argo-cd/argocd:2.11.0-a63068d0

We're preparing our dev environment for testing this change.

Unfortunately there is no brownout at the moment, so we will have to wait until Microsoft randomly turns on the brownout somewhere within the next 20 minutes to 2 hours.

razvangoga · 2024-04-30T13:37:58Z

is this image also available as a helm chart? i could test it but my clusters have argo via helm chart installed

rouke-broersma · 2024-04-30T13:40:43Z

is this image also available as a helm chart? i could test it but my clusters have argo via helm chart installed

You can overwrite the image in the helm chart: https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/values.yaml#L56

jgwest · 2024-04-30T13:51:10Z

@jannfis I can confirm the fix addresses the issue for my reproduction case.

I built the image from the PR branch, and was able to test it just before the most recent brown out ended. I had 2 separate namespace-scoped Argo CD instances targeting the same private Azure repo...

one Argo CD instance based on v2.10 which was consistently failing w/ this error
one Argo CD instance based on @zamedic's PR branch, which was consistently passing
during the same time period.

So presuming the brown out period is shared between them (no reason to think otherwise), I can no longer reproduce.

maartengo · 2024-04-30T14:41:35Z

Anyone inclined to test a build off the master branch with the fix included, the image is ghcr.io/argoproj/argo-cd/argocd:2.11.0-a63068d0

We're preparing our dev environment for testing this change.

Unfortunately there is no brownout at the moment, so we will have to wait until Microsoft randomly turns on the brownout somewhere within the next 20 minutes to 2 hours.

The brownout is active... and we are no longer impacted 🎉

We're looking forward to the next Argo release!

jannfis · 2024-04-30T14:51:30Z

Thanks for testing, folks. I'll cut releases today.

jannfis · 2024-04-30T17:40:42Z

2.10.9 is out, 2.9.14 and 2.8.18 are being built and on their way.

Please let us know if those new releases do not fix the issue for you.

Fixes argoproj#17634 Signed-off-by: Enno Boland <g@s01.de>

evanmcneal · 2024-04-30T18:35:58Z

We are waiting on a brownout to test in our dev environment (2.10.9) then move forward to qa and prod... I'll post an update as soon as I know if it works!

We appreciate the hard work!

jannfis · 2024-04-30T19:10:22Z

We appreciate the hard work!

All credits actually go to the people in this thread discussing their observations and potential causes, @zamedic who finally found the root cause and contributed a fix, and all those fine folks who tested it.

The Argo community is just awesome ❤️

aurel4oxand · 2024-05-02T06:53:32Z

That's amazing, GG !

Any chance to get this fix applied to previous releases ?
Currently I'm using Argocd v2.7 which is impacted by the brownout
Upgrading ArgoCD to more recent version is planned but not immediately actually..

Thanks for this 🙏

rouke-broersma · 2024-05-02T07:11:30Z

That's amazing, GG !

Any chance to get this fix applied to previous releases ? Currently I'm using Argocd v2.7 which is impacted by the brownout Upgrading ArgoCD to more recent version is planned but not immediately actually..

Thanks for this 🙏

You're running an EOL version of ArgoCD and will not receive this patch, you should change your upgrade processes to reflect this: https://argo-cd.readthedocs.io/en/stable/developer-guide/release-process-and-cadence/#patch-releases-eg-25x

aurel4oxand · 2024-05-02T07:17:30Z

That's amazing, GG !
Any chance to get this fix applied to previous releases ? Currently I'm using Argocd v2.7 which is impacted by the brownout Upgrading ArgoCD to more recent version is planned but not immediately actually..
Thanks for this 🙏

You're running an EOL version of ArgoCD and will not receive this patch, you should change your upgrade processes to reflect this: https://argo-cd.readthedocs.io/en/stable/developer-guide/release-process-and-cadence/#patch-releases-eg-25x

Thx, I should've check that 👍

jannfis · 2024-05-02T13:08:51Z

Since I've not heard from folks still affected after upgrading, I'm going to close this issue for now.

If anybody running 2.10.9, 2.9.14 or 2.8.18 (and higher) still encounter RSA-related issues with Azure DevOps, feel free to ping on this issue to have it re-opened.

RichardRosso · 2024-06-07T12:33:43Z

I had version v2.1.5 installed and couldn't resolve the issue despite following the recommendations provided here. Eventually, I upgraded to version v2.9.0+9cf0c69, but not before performing a backup and restore. For now, the problem has been resolved.

deB4SH added the bug Something isn't working label Mar 27, 2024

jgwest added the component:git Interaction with GitHub, Gitlab etc label Apr 17, 2024

zamedic mentioned this issue Apr 30, 2024

fix: enable sha256 and sha512 for git ssh #18028

Merged

14 tasks

jannfis closed this as completed in #18028 Apr 30, 2024

jannfis reopened this Apr 30, 2024

Gottox added a commit to Gottox/argo-cd that referenced this issue Apr 30, 2024

add support for defining a custom list of ssh key exchange algorithms.

0bc4f28

Fixes argoproj#17634 Signed-off-by: Enno Boland <g@s01.de>

jannfis closed this as completed May 2, 2024

ArgoCD Repo Server stops pulling git repositories due to Azure Devops Repos current sunset SSH-RSA strategy #17634

ArgoCD Repo Server stops pulling git repositories due to Azure Devops Repos current sunset SSH-RSA strategy #17634

Comments

deB4SH commented Mar 27, 2024

maxkaemmerer commented Apr 8, 2024

mbu147 commented Apr 8, 2024 • edited Loading

crenshaw-dev commented Apr 8, 2024

maartengo commented Apr 11, 2024 • edited Loading

maartengo commented Apr 15, 2024

siawkz commented Apr 16, 2024

maartengo commented Apr 16, 2024

siawkz commented Apr 16, 2024

maartengo commented Apr 16, 2024

siawkz commented Apr 16, 2024

zamedic commented Apr 16, 2024

AleksuKey commented Apr 16, 2024

maxkaemmerer commented Apr 16, 2024 • edited Loading

rouke-broersma commented Apr 16, 2024 • edited Loading

maxkaemmerer commented Apr 16, 2024 • edited Loading

bartoszpyrek commented Apr 16, 2024

patst commented Apr 16, 2024

rouke-broersma commented Apr 16, 2024 • edited Loading

AleksuKey commented Apr 16, 2024

maxkaemmerer commented Apr 16, 2024 • edited Loading

FlorisFeddema commented Apr 18, 2024 • edited Loading

maxkaemmerer commented Apr 25, 2024

FlorisFeddema commented Apr 25, 2024

danijam commented Apr 25, 2024 • edited Loading

patst commented Apr 25, 2024

rouke-broersma commented Apr 25, 2024

aurel4oxand commented Apr 25, 2024 • edited Loading

rouke-broersma commented Apr 25, 2024

zamedic commented Apr 29, 2024 • edited Loading

cveld commented Apr 29, 2024 • edited Loading

rouke-broersma commented Apr 29, 2024

cveld commented Apr 29, 2024

zamedic commented Apr 30, 2024

blakepettersson commented Apr 30, 2024

zamedic commented Apr 30, 2024

pimjansen commented Apr 30, 2024

rouke-broersma commented Apr 30, 2024 • edited Loading

jannfis commented Apr 30, 2024

jannfis commented Apr 30, 2024

rouke-broersma commented Apr 30, 2024 • edited Loading

razvangoga commented Apr 30, 2024

rouke-broersma commented Apr 30, 2024

jgwest commented Apr 30, 2024 • edited Loading

maartengo commented Apr 30, 2024

jannfis commented Apr 30, 2024

jannfis commented Apr 30, 2024

evanmcneal commented Apr 30, 2024

jannfis commented Apr 30, 2024

aurel4oxand commented May 2, 2024

rouke-broersma commented May 2, 2024

aurel4oxand commented May 2, 2024

jannfis commented May 2, 2024

RichardRosso commented Jun 7, 2024

mbu147 commented Apr 8, 2024 •

edited

Loading

maartengo commented Apr 11, 2024 •

edited

Loading

maxkaemmerer commented Apr 16, 2024 •

edited

Loading

rouke-broersma commented Apr 16, 2024 •

edited

Loading

maxkaemmerer commented Apr 16, 2024 •

edited

Loading

rouke-broersma commented Apr 16, 2024 •

edited

Loading

maxkaemmerer commented Apr 16, 2024 •

edited

Loading

FlorisFeddema commented Apr 18, 2024 •

edited

Loading

danijam commented Apr 25, 2024 •

edited

Loading

aurel4oxand commented Apr 25, 2024 •

edited

Loading

zamedic commented Apr 29, 2024 •

edited

Loading

cveld commented Apr 29, 2024 •

edited

Loading

rouke-broersma commented Apr 30, 2024 •

edited

Loading

rouke-broersma commented Apr 30, 2024 •

edited

Loading

jgwest commented Apr 30, 2024 •

edited

Loading