openssh 9.3p1_1 segfaults when connecting to Azure DevOps repos #135200

ffloimair · 2023-06-28T09:39:33Z

`brew gist-logs <formula>` link OR `brew config` AND `brew doctor` output

brew gist-logs openssh
Error: No logs.

brew config

HOMEBREW_VERSION: 4.0.26-2-gc5cbe64
ORIGIN: https://github.com/Homebrew/brew
HEAD: c5cbe6406b8b9d040c5d1850b5f041af4ddef83c
Last commit: 16 hours ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 9b987d9af4bded53d235cdc0429d0f1027330032
Core tap last commit: 5 months ago
Core tap branch: master
Core tap JSON: 28 Jun 09:17 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: nano
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 8
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: octa-core 64-bit icelake
Clang: 14.0.3 build 1403
Git: 2.41.0 => /usr/local/bin/git
Curl: 7.88.1 => /usr/bin/curl
macOS: 13.4.1-x86_64
CLT: 14.3.1.0.1.1683849156
Xcode: N/A

brew doctor

Your system is ready to brew.

Verification

My "brew doctor output" says Your system is ready to brew. and am still able to reproduce my issue.
I ran brew update and am still able to reproduce my issue.
I have resolved all warnings from brew doctor and that did not fix my problem.
I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.

What were you trying to do (and why)?

git fetch from an Azure DevOps Repository

What happened (include all command output)?

fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

What did you expect to happen?

data can be fetched from remote repository

However this fails

Doing a cross-test with ssh directly i was able to reproduce what is the problem: openssh segfaults

ssh -Tv git@ssh.dev.azure.com

OpenSSH_9.3p1, OpenSSL 3.1.1 30 May 2023
debug1: Reading configuration data /Users/<redacted>ssh /.ssh/config
debug1: /Users/<redacted>/.ssh/config line 33: Applying options for ssh.dev.azure.com
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Connecting to ssh.dev.azure.com [40.74.28.19] port 22.
debug1: Connection established.
debug1: identity file /Users/<redacted>/.ssh/id_rsa type 0
debug1: identity file /Users/<redacted>/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.3
debug1: Remote protocol version 2.0, remote software version SSHBlackbox.10
debug1: compat_banner: no match: SSHBlackbox.10
debug1: Authenticating to ssh.dev.azure.com:22 as 'git'
debug1: load_hostkeys: fopen /Users/<redacted>/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_GROUP received
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: SSH2_MSG_KEX_DH_GEX_REPLY received
debug1: Server host key: ssh-rsa SHA256:ohD8VZEXGWo6Ez8GSEJQ9WpafgLFsOfLOtGGQCQo6Og
debug1: load_hostkeys: fopen /Users/<redacted>/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'ssh.dev.azure.com' is known and matches the RSA host key.
debug1: Found key in /Users/<redacted>/.ssh/known_hosts:191
[1]    30012 segmentation fault  ssh -Tvvv git@ssh.dev.azure.com

If i revert openssh to commit 2848b48 (openssh 9.3p1), install the formula and pin it. Everything works fine (both ssh test and git).

Step-by-step reproduction instructions (by running `brew` commands)

brew install git openssh

git clone
or
git fetch (if the repo is already cloned locally)

will fail

If you do the crosstest with ssh (no Azure DevOps account required) it will segfault.

ssh -Tv git@ssh.dev.azure.com

Unfortunately Azure Repos still uses an old ssh-rsa authentication so you need the following in your ~/.ssh/config (see: https://learn.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops)

Host ssh.dev.azure.com
     HostKeyAlgorithms +ssh-rsa
     PubkeyAcceptedKeyTypes +ssh-rsa

The text was updated successfully, but these errors were encountered:

SMillerDev · 2023-06-28T10:01:53Z

Seems to be similar to openssl/openssl#18420

ffloimair · 2023-06-28T11:14:12Z

Not necessarily. While it is related as Azure DevOps uses an old Host-Key Algorithm, it doesn't explain the segfault. The config override was already necessary in earlier versions of openssh.
This seems to be related to the use of openssl 3.0 as default now. Possibly this openSSL version deprecated this behaviour altogether, but still this should not segfault.

What is even more dubious is the fact that this happens in combination with git, without git giving you even a hint what the problem is. It took me several hours to narrow down that the problem is in fact within openssh.

Whatever this leads to it still should not segfault but instead show an error that is descriptive of the situation.
If the solution to get it working is pinning openssh for those who need it, I am okay with that, but still a segfault is a no-go.

SMillerDev · 2023-06-28T11:18:17Z

Can you run this command with a debugger to see what is happening with the segfault?

ffloimair · 2023-06-28T12:10:28Z

I'll see if I can find time to do so either tonight or tomorrow. Please give me some time.

Bo98 · 2023-06-28T12:36:38Z

Thanks for the detailed steps for reproduction. I've been unable to reproduce the issue so far:

$ ssh -Tv git@ssh.dev.azure.com
OpenSSH_9.3p1, OpenSSL 3.1.1 30 May 2023
debug1: Reading configuration data /Users/bo/.ssh/config
debug1: /Users/bo/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Connecting to ssh.dev.azure.com [51.104.26.0] port 22.
debug1: Connection established.
debug1: identity file /Users/bo/.ssh/id_rsa type -1
debug1: identity file /Users/bo/.ssh/id_rsa-cert type -1
debug1: identity file /Users/bo/.ssh/id_ecdsa type -1
debug1: identity file /Users/bo/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/bo/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/bo/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/bo/.ssh/id_ed25519 type -1
debug1: identity file /Users/bo/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/bo/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/bo/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/bo/.ssh/id_xmss type -1
debug1: identity file /Users/bo/.ssh/id_xmss-cert type -1
debug1: identity file /Users/bo/.ssh/id_dsa type -1
debug1: identity file /Users/bo/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.3
debug1: Remote protocol version 2.0, remote software version SSHBlackbox.10
debug1: compat_banner: no match: SSHBlackbox.10
debug1: Authenticating to ssh.dev.azure.com:22 as 'git'
debug1: load_hostkeys: fopen /Users/bo/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_GROUP received
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: SSH2_MSG_KEX_DH_GEX_REPLY received
debug1: Server host key: ssh-rsa SHA256:ohD8VZEXGWo6Ez8GSEJQ9WpafgLFsOfLOtGGQCQo6Og
debug1: load_hostkeys: fopen /Users/bo/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /usr/local/etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'ssh.dev.azure.com' is known and matches the RSA host key.
debug1: Found key in /Users/bo/.ssh/known_hosts:102
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: get_agent_identities: ssh_fetch_identitylist: agent contains no identities
debug1: Will attempt key: /Users/bo/.ssh/id_rsa 
debug1: Will attempt key: /Users/bo/.ssh/id_ecdsa 
debug1: Will attempt key: /Users/bo/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /Users/bo/.ssh/id_ed25519 
debug1: Will attempt key: /Users/bo/.ssh/id_ed25519_sk 
debug1: Will attempt key: /Users/bo/.ssh/id_xmss 
debug1: Will attempt key: /Users/bo/.ssh/id_dsa 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-rsa,rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: password,publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/bo/.ssh/id_rsa
debug1: Trying private key: /Users/bo/.ssh/id_ecdsa
debug1: Trying private key: /Users/bo/.ssh/id_ecdsa_sk
debug1: Trying private key: /Users/bo/.ssh/id_ed25519
debug1: Trying private key: /Users/bo/.ssh/id_ed25519_sk
debug1: Trying private key: /Users/bo/.ssh/id_xmss
debug1: Trying private key: /Users/bo/.ssh/id_dsa
debug1: Next authentication method: password
git@ssh.dev.azure.com's password:

ffloimair · 2023-06-28T13:22:59Z

interesting

cho-m · 2023-06-28T21:46:18Z

I can reproduce a segfault using x86_64 bottle on Rosetta installation (though I didn't set up RSA key for Azure or SSH configs so can't guarantee exact same situation). I will note that I can't reproduce with ARM bottle installation though with same setup/configs.

Process 11772 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
    frame #0: 0x000000010910019a libcrypto.3.dylib`BN_num_bits + 17
libcrypto.3.dylib`BN_num_bits:
->  0x10910019a <+17>: movslq 0x8(%rdi), %rcx
    0x10910019e <+21>: leal   -0x1(%rcx), %r13d
    0x1091001a2 <+25>: testb  $0x4, 0x14(%rdi)
    0x1091001a6 <+29>: jne    0x1091001c6               ; <+61>
Target 0: (ssh) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x8)
  * frame #0: 0x000000010910019a libcrypto.3.dylib`BN_num_bits + 17
    frame #1: 0x000000010005bbbe ssh`___lldb_unnamed_symbol871 + 78
    frame #2: 0x00000001000e3149 ssh`___lldb_unnamed_symbol1817 + 537
    frame #3: 0x00000001000e4d99 ssh`___lldb_unnamed_symbol1830 + 1209
    frame #4: 0x00000001000a5506 ssh`___lldb_unnamed_symbol1372 + 390
    frame #5: 0x00000001000a566a ssh`___lldb_unnamed_symbol1373 + 42
    frame #6: 0x000000010002fba4 ssh`___lldb_unnamed_symbol588 + 1220
    frame #7: 0x000000010002cf1f ssh`___lldb_unnamed_symbol572 + 335
    frame #8: 0x0000000100009856 ssh`___lldb_unnamed_symbol437 + 21478
    frame #9: 0x000000020014541f dyld`start + 1903

Looking at recompiled symbols, in my case, kexgex_hash is called with NULL values (e.g. prime=0x0000000000000000) at https://github.com/openssh/openssh-portable/blob/master/kexgexc.c#L187-L198

carlocab · 2023-06-29T17:20:38Z

I can reproduce this with the Ventura bottle but not the Monterey one. Using the Monterey bottle for now might be enough of a workaround:

brew fetch --bottle-tag=monterey openssh
brew uninstall openssh
brew install "$(brew --cache --bottle-tag=monterey openssh)"

cho-m · 2023-06-29T23:59:04Z

We could try building with llvm instead.

I think part of issue is OpenSSH's configure.ac Clang version-specific logic, e.g.:

	# clang 15 seems to have a bug in -fzero-call-used-regs=all.  See
	# https://bugzilla.mindrot.org/show_bug.cgi?id=3475 and
	# https://github.com/llvm/llvm-project/issues/59242
	case "$CLANG_VER" in
	15.*) OSSH_CHECK_CFLAG_COMPILE([-fzero-call-used-regs=used]) ;;
	*)    OSSH_CHECK_CFLAG_COMPILE([-fzero-call-used-regs=all]) ;;
	esac

Xcode 14.3.1 (Apple clang 14.0.3) is LLVM 15. Also, the version detection doesn't work with Apple Clang (nor with Homebrew LLVM's Clang) and returns:

checking clang version... version

The "Apple"/"Homebrew" in clang -v causes issues (and wouldn't help detect upstream version for Apple).

cho-m · 2023-06-30T00:22:26Z

Trying out an inreplace hack in #135373. Didn't see any issue when attempted locally but haven't checked if this is the root cause of issue.

ffloimair · 2023-06-30T06:58:41Z

openssh_9.3p1_2 bottle fixes the issue on my machine.
Thanks @cho-m !!!

j0k3r · 2023-06-30T07:08:54Z

Problem fixed for me as well, no more segfault. 👏🏼

ffloimair added the bug Reproducible Homebrew/homebrew-core bug label Jun 28, 2023

cho-m mentioned this issue Jun 30, 2023

openssh 9.3p1_1 spawns zombie process #135197

Closed

4 tasks

carlocab closed this as completed Jun 30, 2023

github-actions bot added the outdated PR was locked due to age label Jul 31, 2023

github-actions bot locked as resolved and limited conversation to collaborators Jul 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

openssh 9.3p1_1 segfaults when connecting to Azure DevOps repos #135200

openssh 9.3p1_1 segfaults when connecting to Azure DevOps repos #135200

ffloimair commented Jun 28, 2023 •

edited

Loading

SMillerDev commented Jun 28, 2023

ffloimair commented Jun 28, 2023

SMillerDev commented Jun 28, 2023

ffloimair commented Jun 28, 2023

Bo98 commented Jun 28, 2023 •

edited

Loading

ffloimair commented Jun 28, 2023

cho-m commented Jun 28, 2023 •

edited

Loading

carlocab commented Jun 29, 2023

cho-m commented Jun 29, 2023 •

edited

Loading

cho-m commented Jun 30, 2023

ffloimair commented Jun 30, 2023

j0k3r commented Jun 30, 2023

openssh 9.3p1_1 segfaults when connecting to Azure DevOps repos #135200

openssh 9.3p1_1 segfaults when connecting to Azure DevOps repos #135200

Comments

ffloimair commented Jun 28, 2023 • edited Loading

brew gist-logs <formula> link OR brew config AND brew doctor output

Verification

What were you trying to do (and why)?

What happened (include all command output)?

What did you expect to happen?

Step-by-step reproduction instructions (by running brew commands)

SMillerDev commented Jun 28, 2023

ffloimair commented Jun 28, 2023

SMillerDev commented Jun 28, 2023

ffloimair commented Jun 28, 2023

Bo98 commented Jun 28, 2023 • edited Loading

ffloimair commented Jun 28, 2023

cho-m commented Jun 28, 2023 • edited Loading

carlocab commented Jun 29, 2023

cho-m commented Jun 29, 2023 • edited Loading

cho-m commented Jun 30, 2023

ffloimair commented Jun 30, 2023

j0k3r commented Jun 30, 2023

ffloimair commented Jun 28, 2023 •

edited

Loading

`brew gist-logs <formula>` link OR `brew config` AND `brew doctor` output

Step-by-step reproduction instructions (by running `brew` commands)

Bo98 commented Jun 28, 2023 •

edited

Loading

cho-m commented Jun 28, 2023 •

edited

Loading

cho-m commented Jun 29, 2023 •

edited

Loading