setopt: CURLOPT_SSH_HOST_PUBLIC_KEY_SHA256 returns CURLE_UNKNOWN_OPTION instead of CURLE_NOT_BUILT_IN on libssh builds

[MegaManSec]

I did this

Hi all,

I'm aware this option has been reported many times on HackerOne as a security issue — the framing is always some variant of "SHA256 pin is silently ignored, MITM possible." That was true and was fixed by b45b6b618d (2023-01-07, "setopt: move the SHA256 opt within #ifdef libssh2"), which moved the case inside #ifdef USE_LIBSSH2 so it now returns an error instead of silently no-opping. This is not that report.

The remaining bug is that the wrong error code comes back. The case falls into default: and returns CURLE_UNKNOWN_OPTION (48) when it should return CURLE_NOT_BUILT_IN (49). The distinction matters: CURLE_UNKNOWN_OPTION means "curl has never heard of this option", which is the signal an app uses to detect it's running against an older libcurl that predates the option. CURLE_NOT_BUILT_IN means "curl knows the option but this build doesn't support it." Those are different failure modes with different correct responses from the caller.

Sibling options handle this correctly — lib/setopt.c:2303,2311 use Curl_ssl_supports to return CURLE_NOT_BUILT_IN for TLS options absent from the current build. The pattern exists, it just wasn't applied here.

lib/setopt.c:2351-2364:

#ifdef USE_LIBSSH2
  case CURLOPT_SSH_HOST_PUBLIC_KEY_SHA256:
    return Curl_setstropt(&s->str[STRING_SSH_HOST_PUBLIC_KEY_SHA256], ptr);
  case CURLOPT_SSH_HOSTKEYDATA:
    s->ssh_hostkeyfunc_userp = ptr;
    break;
#endif /* USE_LIBSSH2 */

When built against libssh, both cases fall to default: → CURLE_UNKNOWN_OPTION. An app that checks setopt returns and treats 48 as "older libcurl, skip this option and continue" will proceed without any pin check. An app that checks for 49 specifically will correctly abort. The reproducer shows the difference:

$ ./api_test_libssh "AAAAAAAA…AAA=" "sftp://127.0.0.1:22222/hello.txt"
=== test build: libcurl 8.21.0-DEV, ssh=libssh/0.11.1/openssl/zlib ===
[A] App ignores setopt return:
  setopt returned 48, proceeding regardless
  perform returned 0 (No error)
  RESULT: connection ACCEPTED with wrong pin

[B] App checks setopt return:
  setopt failed: 48, aborting before perform
  RESULT: aborted before perform (SAFE)

Same source linked against system libcurl (libssh2 backend) returns 60 (CURLE_PEER_FAILED_VERIFICATION) for case A. The curl(1) tool is not affected — src/tool_getparam.c:2770-2775 gates --hostpubsha256 on feature_libssh2.

While in here: feature parity would be straightforward. libssh has exposed ssh_get_publickey_hash(pubkey, SSH_PUBLICKEY_HASH_SHA256, ...) since 0.8.x, directly mirroring the MD5 branch already implemented at lib/vssh/libssh.c:142-169. If SHA256 were implemented there, the wrong-error-code question becomes moot.

I expected the following

correct error code

curl/libcurl version

master

operating system

all