latest openssh security update breaks chacha20-poly1305

[ksyblast]

Hi team!

Oracle Linux 8.9
5.4.17-2136.326.6.el8uek.x86_64
OpenSSH_8.0p1, OpenSSL 1.1.1k

After upgrading to
Name : openssh
Version : 8.0p1
Release : 19.el8_9.2

cannot ssh anymore using chacha20-poly1305 cipher getting Bad packet length and Connection corrupted errors. Everything works using, for example, AES256.

Openssh rollback to the previous version fixes the issue.

Could it be a bug? Thanks

[scoter-oracle]

Can you post logs/evidences on the issue you're facing ? Please share also the client details and OS! Thanks

[ksyblast]

Yes sure
Client openssh-9.0p1-18.fc38.x86_64
openssl version OpenSSL 3.0.9 30 May 2023 (Library: OpenSSL 3.0.9 30 May 2023)
Client OS: Fedora release 38 (Thirty Eight)

ssh -vvv -o Ciphers=chacha20-poly1305@openssh.com 1.2.3.4
OpenSSH_9.0p1, OpenSSL 3.0.9 30 May 2023
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 19: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host 1.2.3.4 originally 1.2.3.4
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final'
debug2: match not found
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]
debug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]
debug1: configuration requests final Match pass
debug2: resolve_canonicalize: hostname 1.2.3.4 is address
debug1: re-parsing configuration
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 19: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host 1.2.3.4 originally 1.2.3.4
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched 'final'
debug2: match found
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]
debug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug1: auto-mux: Trying existing master
debug1: Control socket "/home/user/.ssh/user@1.2.3.4-22" does not exist
debug3: ssh_connect_direct: entering
debug1: Connecting to 1.2.3.4 [1.2.3.4] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 0
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519 type 3
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/user/.ssh/id_xmss type -1
debug1: identity file /home/user/.ssh/id_xmss-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: compat_banner: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 1.2.3.4:22 as 'user'
debug3: record_hostkey: found key type ED25519 in file /home/user/.ssh/known_hosts:1555
debug3: load_hostkeys_file: loaded 1 keys from 1.2.3.4
debug1: load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 ...
debug3: record_hostkey: found key type ED25519 in file /home/user/.ssh/known_hosts:1555
debug3: load_hostkeys_file: loaded 1 keys from 1.2.3.4
debug1: load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '1.2.3.4' is known and matches the ED25519 host key.
debug1: Found key in /home/user/.ssh/known_hosts:1555
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 4 keys
debug1: Will attempt key: /home/user/.ssh/id_rsa
debug1: Will attempt key: /home/user/.ssh/id_ed25519
debug1: Will attempt key: /home/user/.ssh/id_ecdsa 
debug1: Will attempt key: /home/user/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/user/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/user/.ssh/id_xmss 
debug1: Will attempt key: /home/user/.ssh/id_dsa 
debug2: pubkey_prepare: done
debug3: send packet: type 5
Bad packet length 3053936917.
debug2: sshpkt_disconnect: sending SSH2_MSG_DISCONNECT: Packet corrupt
debug3: send packet: type 1
ssh_dispatch_run_fatal: Connection to 1.2.3.4 port 22: Connection corrupted

When I rollback ssh to the previous version at the server everything works fine.
Please tell me which details I should add to help you to investigate

[ksyblast]

P.S. with Ubuntu clients this issue is present by default (looks like it prefers chacha20-poly1305 by default)
Another affected client example:
OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022
Ubuntu 22.04.3 LTS

[gzuaps]

openssh update perfomed on Feb 2 2024 breaks login with default ciphers.
aes256-ctr,aes192-ctr,aes128-ctr all fail with

Corrupted MAC on input.
ssh_dispatch_run_fatal: Connection to 1.2.3.4 port 22: message authentication code incorrect

Client: Centos9 Stream

[loopiv]

Seeing the same issue from Ubuntu to OL8 systems. From a bug report at Ubuntu:

I believe this issue is caused by a bad backport in Oracle's 8.0p1-19.el8_9.2 package. I think their fix for CVE-2023-48795 isn't properly adding kex-strict-s-v00@openssh.com to their KEX. Downgrading the Ubuntu package works around the problem as that prevents the client from offering kex-strict-c-v00@openssh.com.

[b-a-t]

Just to add me too. We also experienced exactly the same problems with SSH interconnectivity since the last update.

[ties]

It would help my situation if there is a new release that machines with the faulty package will update to.

[mboehm21]

We experience the same behaviour:

Server

  Operating System: Oracle Linux Server 8.9
       CPE OS Name: cpe:/o:oracle:linux:8:9:server
            Kernel: Linux 5.15.0-202.135.2.el8uek.x86_64
      Architecture: x86-64

libssh.x86_64                        0.9.6-13.el8_9                              @oraclelinux8-x86_64          
libssh-config.noarch                 0.9.6-13.el8_9                              @oraclelinux8-x86_64          
openssh.x86_64                       8.0p1-19.el8_9.2                            @oraclelinux8-x86_64          
openssh-clients.x86_64               8.0p1-19.el8_9.2                            @oraclelinux8-x86_64          
openssh-server.x86_64                8.0p1-19.el8_9.2                            @oraclelinux8-x86_64

Client

Operating System: Ubuntu 22.04.3 LTS              
          Kernel: Linux 6.5.0-15-generic
    Architecture: x86-64

ii  libssh-4:amd64                             0.9.6-2ubuntu0.22.04.3                  amd64        tiny C SSH library (OpenSSL flavor)
ii  libssh-gcrypt-4:amd64                      0.9.6-2ubuntu0.22.04.3                  amd64        tiny C SSH library (gcrypt flavor)
ii  libssh2-1:amd64                            1.10.0-3                                amd64        SSH2 client-side library
ii  openssh-client                             1:8.9p1-3ubuntu0.6                      amd64        secure shell (SSH) client, for secure access to remote machines

Error

ssh admin@server
Bad packet length 3842318532.
ssh_dispatch_run_fatal: Connection to 1.2.3.4 port 22: Connection corrupted

Workaround

ssh -c aes256-gcm@openssh.com admin@server                                                                                                                         
[admin@server ~]$

[8uachaille]

Same problem with 8.0p1-19.el8_9.2.x86_64 on a few machines here

[pa-jberanek]

Affects connections from PuTTY 0.80 (latest stable release) too.

[romario74]

arch/manjaro SSH client has the same problem:
$ ssh -v
OpenSSH_9.6p1, OpenSSL 3.2.0 23 Nov 2023

Any ETA please?

[tvierling]

Thank you very much for the report and the detailed reproduction instructions. A fix for this was released, and announced here: https://linux.oracle.com/errata/ELSA-2024-12164.html