curl 8.10.x with SPNEGO authentication through an HTTP proxy to an HTTP endpoint throws "Failed to find SSL backend for endpoint" error

[stevenpackardblp]

I did this

$ curl -Ssl -u : --negotiate -H "Accept: application/json" --proxy http://<http server proxy address> https://<API server endpoint>
curl: (66) Failed to find SSL backend for endpoint

I expected the following

$ curl -Ssl -u : --negotiate -H "Accept: application/json" --proxy http://<http server proxy address> https://<API server endpoint>
{<json response from the API server>}

curl/libcurl version

curl 8.10.0 and curl 8.10.1

curl --version
curl 8.10.1 (x86_64-unknown-linux-gnu) libcurl/8.10.1 OpenSSL/1.1.1y zlib/1.2.13 libpsl/0.21.5 libssh2/1.9.0 nghttp2/1.61.0
Release-Date: 2024-09-18
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets

operating system

Linux RHEL 7.9 and RHEL 8.8

$ uname -a
Linux <hostname redacted> 4.18.0-477.36.1.el8_8.x86_64 #1 SMP Thu Nov 9 08:12:18 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

[bagder]

Did this work with a previous curl version?

[stevenpackardblp]

It's only when a proxy server is used that the issue occurs. SPNEGO to an HTTPS endpoint without using a proxy server works as expected. I believe 0a5ea09 is the commit that introduced the regression, but I'm not sure why it's causing the issue other than there are no ssl_connect_data with SSL in its name in the linked list when I stepped through a debugger. The only names I saw were HTTPS-CONNECT, SETUP, HTTP-PROXY, H1-PROXY, and HAPPY-EYEBALLS. This the full walkback when the error occurs:

(gdb) where
#0  ossl_get_channel_binding (data=0x467e50, sockindex=0, binding=0x460f60)
    at ../../lib/vtls/openssl.c:5119
#1  0x00007ffff7f7a5d1 in Curl_input_negotiate (data=data@entry=0x467e50,
    conn=conn@entry=0x460a80, proxy=proxy@entry=false, header=<optimized out>,
    header@entry=0x7ffff7fc94bc "Negotiate") at ../../lib/http_negotiate.c:120
#2  0x00007ffff7f7a8d6 in Curl_output_negotiate (data=0x467e50, conn=0x460a80,
    proxy=proxy@entry=false) at ../../lib/http_negotiate.c:191
#3  0x00007ffff7f6b2a9 in output_auth_headers (data=data@entry=0x467e50,
    conn=conn@entry=0x460a80, authstatus=authstatus@entry=0x468c28,
    request=request@entry=0x462960 "CONNECT",
    path=path@entry=0x464e40 "<redacted>",
    proxy=proxy@entry=false) at ../../lib/http.c:601
#4  0x00007ffff7f6bab4 in Curl_http_output_auth (data=data@entry=0x467e50,
    conn=0x460a80, request=0x462960 "CONNECT",
    httpreq=httpreq@entry=HTTPREQ_GET,
    path=0x464e40 "<redacted>",
    proxytunnel=proxytunnel@entry=true) at ../../lib/http.c:772
#5  0x00007ffff7f7b477 in Curl_http_proxy_create_CONNECT (
    preq=preq@entry=0x7fffffffd9f8, cf=cf@entry=0x462870,
    data=data@entry=0x467e50, http_version_major=http_version_major@entry=1)
    at ../../lib/http_proxy.c:114
#6  0x00007ffff7f42951 in start_CONNECT (ts=0x4628a0, data=0x467e50,
    cf=0x462870) at ../../lib/cf-h1-proxy.c:229
#7  H1_CONNECT (ts=0x4628a0, data=0x467e50, cf=0x462870)
    at ../../lib/cf-h1-proxy.c:890
#8  cf_h1_proxy_connect (cf=0x462870, data=0x467e50, blocking=<optimized out>,
    done=0x7fffffffdbab) at ../../lib/cf-h1-proxy.c:1004
#9  0x00007ffff7f7b067 in http_proxy_cf_connect (cf=0x462840, data=0x467e50,
    blocking=<optimized out>, done=0x7fffffffdbab)
    at ../../lib/http_proxy.c:181
#10 0x00007ffff7f4f100 in cf_setup_connect (cf=0x461a30, data=0x467e50,
    blocking=false, done=0x7fffffffdbab) at ../../lib/connect.c:1261
#11 0x00007ffff7f47825 in cf_hc_baller_connect (done=0x7fffffffdbab,
    data=0x467e50, cf=0x4633e0, b=0x462f38) at ../../lib/cf-https-connect.c:151
#12 cf_hc_connect (cf=0x4633e0, data=0x467e50, blocking=<optimized out>,
    done=0x7fffffffdbab) at ../../lib/cf-https-connect.c:298
#13 0x00007ffff7f4c099 in Curl_conn_connect (data=data@entry=0x467e50,
    sockindex=sockindex@entry=0, blocking=blocking@entry=false,
    done=done@entry=0x7fffffffdbab) at ../../lib/cfilters.c:432
#14 0x00007ffff7f88c51 in multi_runsingle (multi=multi@entry=0x4621e0,
    nowp=nowp@entry=0x7fffffffdc40, data=data@entry=0x467e50)
    at ../../lib/multi.c:2030
#15 0x00007ffff7f8a7d9 in curl_multi_perform (multi=multi@entry=0x4621e0,
    running_handles=running_handles@entry=0x7fffffffdd48)
    at ../../lib/multi.c:2653
#16 0x00007ffff7f5b6db in easy_transfer (multi=<optimized out>)
    at ../../lib/easy.c:700
#17 easy_perform (events=false, data=0x467e50) at ../../lib/easy.c:795
#18 curl_easy_perform (data=0x467e50) at ../../lib/easy.c:814
#19 0x000000000041562d in serial_transfers (share=0x462aa0,
    global=0x7fffffffdf00) at ../../src/tool_operate.c:2941
#20 run_all_transfers (result=<optimized out>, share=0x462aa0,
    global=0x7fffffffdf00) at ../../src/tool_operate.c:3129
#21 operate (global=global@entry=0x7fffffffdf00, argc=argc@entry=13,
    argv=argv@entry=0x7fffffffe068) at ../../src/tool_operate.c:3251
#22 0x00000000004039fa in main (argc=13, argv=0x7fffffffe068)
    at ../../src/tool_main.c:271

[stevenpackardblp]

Did this work with a previous curl version?

Yes, it works fine in 8.9.1.

[bagder]

/cc @SGA-max-faxalv

SSL-PROXY perhaps?

[Foorack]

Deeply sorry for the regression. I am currently on holiday without a laptop, and will be back 3rd October.

In the meantime I'm investigating (1) how/if this will be possible, (2) whether the PROXY requires to support SCB as well...

I have an environment at work where I can test this, but I don't know if that proxy supports channel binding.

In the meantime I'm wondering if it would be possible to disable SCB when using proxy, saying it is not supported yet, which would restore previous functionality. Doing so leaves a bad taste, but I also feel deeply sorry for not having access to a computer for so long either to "fix my mess" in a timely manner.

[stevenpackardblp]

We do have other authentication mechanisms besides SPNEGO for the service, but they're not as convenient. We've opted to back out curl to version 8.9.1 until this gets addressed.

[brookheather]

I have the same issue when using 8.10.1 - is there a fix for this in progress - will it be included in the next release (8.10.2?)

[Foorack]

@brookheather Trying to understand how to replicate this. Are you trying to auth with SPNEGO to a HTTPS server, through a HTTP proxy server?

[brookheather]

@brookheather Trying to understand how to replicate this. Are you trying to auth with SPNEGO to a HTTPS server, through a HTTP proxy server?

Correct. I am trying to use libcurl in a program to achieve the same as using the bundled curl.exe 8.8.0 on Windows (which works fine):

curl -x proxy.example.com:8080 --proxy-negotiate https://login.microsoftonline.com/xxxxx/v2.0/.well-known/openid-configuration

[Foorack]

@brookheather I apologize for the inconvenience.

I want to avoid breaking past working functionality, so I have made a proposed PR which - similar to if a peer certificate isn't presented - soft-fails and continues establishing the connection without SecureChannelBinding even if an SSL backend isn't found (such as when using a HTTP proxy).

This makes the behaviour of CURL the same as before then SCB feature was integrated.

And then if someone very specifically needs SecureChannelBinding over HTTP proxy (if that is even technically possible!?) it could be implemented in a future request.

#15410

[brookheather]

Thanks @Foorack I have tested with your change - it no longer shows the original error but I am now getting a "gss_init_sec_context() failed" error during the connection:

CONNECT tunnel: HTTP/1.1 negotiated
allocate connect buffer
gss_init_sec_context() failed: No credentials were supplied, or the credentials were unavailable or inaccessible. Internal credentials cache error.
Proxy auth using Negotiate with user ''
Establish HTTP proxy tunnel to login.microsoftonline.com:443
gss_init_sec_context() failed: No credentials were supplied, or the credentials were unavailable or inaccessible. Internal credentials cache error.
CONNECT tunnel failed, response 407

[stevenpackardblp]

Is there an update here? The issue is still present in version 8.11.1 and is blocking upgrading from 8.9.1. Thanks!

[Foorack]

Been extremely busy, but will try to look into it this week.

@stevenpackardblp @brookheather Do you know what software is used on the proxy?

[stevenpackardblp]

@Foorack I don't know the answer that question, but I'm trying to find out and whether I can share it in this forum. If you're attending FOSDEM this weekend, I'd be happy to sit down with you to do some testing and debugging.

[Foorack]

I've finally had time to sit down and refresh my memory of the code, and there seems to have been done changes since:

• Conditional check around GSS & SSL specific code in http_negotiate.c #14954
• Check connection for SSL use, not handler #16034

Which have made it so CURL specifically no longer crashes if you are connecting through an HTTP proxy. I think you would still crash if you connect through an HTTPS proxy. Oh, and SCB still doesn't work through either type of Proxy.

But that means as long as the backend server doesn't require EPA - which it shouldn't have done before because doing SCB through Proxy has never been possible until now - then you should be able to upgrade to latest version of curl @stevenpackardblp

I will keep looking into if I can get SCB working on connections going through proxy.

And sadly won't be at FOSDEM. Will be at WHY2025 though.

[mtremer]

Hello everybody,

I think I am also running into the same - or at least similar - problem here. My however is: "Failed to find the SSL filter". My full log looks like this:

cURL: Uses proxy env variable https_proxy == 'http://172.28.1.1:800'
cURL:   Trying 172.28.1.1:800...
cURL: CONNECT tunnel: HTTP/1.1 negotiated
cURL: allocate connect buffer
cURL: Failed to find the SSL filter
cURL: Failed sending CONNECT to proxy
cURL: closing connection #0
cURL xfer done: 43 - A libcurl function was given a bad argument
   Error: Failed to find the SSL filter
   Effective URL: https://pakfire.ipfire.org/api/v1/uploads
   Total Time: 0.00s

Once I clear the https_proxy variable, the connection goes through no problem. I am holding a valid Kerberos ticket, but I don't think this even matters as this point since the connection is not even making it past the proxy. Setting or clearing http_proxy(without the s) does not change behaviour.

My expected behaviour is that a connection through the proxy will be established using plain HTTP without authentication and after the SSL connection has been established, the request will be sent using SPEGNO. On the network, the only thing that I can see is a TCP connection being established to the proxy with no further data being sent. The connection is then immediately closed.

I am using a share handle and I am restricting CURLOPT_PROTOCOLS_STR to only HTTP, HTTPS, FTP and FILE. Apart from that I am not doing anything out of the usual I feel. If you are interested in more details, my code is here: https://git.ipfire.org/?p=pakfire.git;a=blob;f=src/pakfire/xfer.c;h=91294117a3ef1dc681988190a9ce9b769f796838;hb=HEAD#l264. I am also happy to answer questions.

@stevenpackardblp @brookheather Do you know what software is used on the proxy?

My proxy is squid, configured without any authentication.

I am running curl from Debian Bookworm Backports in version 8.11.1:

# curl --version
curl 8.11.1 (x86_64-pc-linux-gnu) libcurl/8.11.1 GnuTLS/3.7.9 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 libssh2/1.10.0 nghttp2/1.52.0 ngtcp2/1.9.1 nghttp3/1.6.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2024-12-11, security patched: 8.11.1-1~bpo12+1
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

[icing]

@mtremer thanks for the report. This problem has been fixed in 25b445e. The negotiate code, talking to the http: proxy, was wrongly assuming that SSL was involved.

@stevenpackardblp Since your stracktrace points to the same code path, I believe this is solved by this change as well.

And yes, @Foorack is correct with their analysis.

[Foorack]

@mtremer It looks like 25b445e did not make it into 8.11.1, but will be released as part of the 8.12.0 release in 5th of February. That fix should prevent the error "Failed to find the SSL filter" and basically revert the behaviour to how it was before (working, but no secure-channel-binding).

I am looking into trying to get SCB working over CONNECT proxy as well, but having slow progress due to not that deeply familiar with curl codebase.

[Foorack]

@icing Yes, thanks to that patch, it is my understanding that both people's problem would be solved in the latest release. The question is if this Issue should then be marked as Resolved, and we instead add "GSSAPI SCB over CONNECT proxy" to the TODO list? As that is a feature which has not been implemented in curl before.

[icing]

@icing Yes, thanks to that patch, it is my understanding that both people's problem would be solved in the latest release. The question is if this Issue should then be marked as Resolved, and we instead add "GSSAPI SCB over CONNECT proxy" to the TODO list? As that is a feature which has not been implemented in curl before.

I agree on both. The bug we have fixed. The other part is a missing feature for TODO, I believe, as you suggest.

[stevenpackardblp]

@mtremer It looks like 25b445e did not make it into 8.11.1, but will be released as part of the 8.12.0 release in 5th of February. That fix should prevent the error "Failed to find the SSL filter" and basically revert the behaviour to how it was before (working, but no secure-channel-binding).

I'm happy to report that patching commit 25b445e into the 8.11.1 release resolves the issue I reported here, so I'm ok with closing this issue as fixed in the 8.12.0 release. Thanks for pointing this out!

[jay]

we instead add "GSSAPI SCB over CONNECT proxy" to the TODO list?

@Foorack do you want to submit a PR for this? I assume you mean secure channel binding?

[mtremer]

Thank you very much for everyone looking at this. I am sorry to send my feedback so late, but I can also confirm that this is resolved in 8.12.0.