Building curl against WolfSSL loses protocols and features compared to OpenSSL

[RuurdBeerstra]

I did this

Configure cURL against OpenSSL on Linux. It says in the configure output:

Protocols: DICT FILE FTP FTPS GOPHER HTTP HTTPS IMAP IMAPS POP3 POP3S RTSP SCP SFTP SMB SMBS SMTP SMTPS TELNET TFTP
Features: SSL IPv6 UnixSockets libz NTLM NTLM_WB TLS-SRP HTTPS-proxy

Then, when I use curl_version_info I also get the exact same list of protocols. In "Features" I get the same set of reported features (plus LARGEFILE and UNIX_SOCKETS).

I'm trying to replace OpenSSL by WolfSSL without losing protocols or features.
When I invoke "configure" with the same options as I do for OpenSSL (using --with-wolfssl --without-ssl), I end up with NTLM, NTLM_WB, TLS-SRP and HTTPS-proxy as missing features and SMB and SMBS as missing protocols.

The output of configure is consistent with the report of curl_version_info: Missing protocols and features.

Adding --enable-proxy and --enable-smb to the commandline changes the output of configure (so I thought I had the support enabled successfully), but curl_version_info still reports missing SMB and SMBS as missing protocols, and the NTLM and NTLM_WB features are missing, too.

Digging into the source of smb.c I can see it says it requires NTLM and OpenSSL crypto functions and does not seem to support WolfSSL backend (yet?). So it is not compiled in.

I expected the following

To have WolfSSL support the same protocols and features as OpenSSL.

So there is the minor problem of configure output that says certain features are enabled while in reality they are not, and the more major problem of being unable to get the features working with WolfSSL as a backend.

Is there some way to get this working?
We embed curl in our product in such a way that we cannot predict which features and protocols end up being used, and have to be backward compatible with previous (OpenSSL based) releases.

curl/libcurl version

curl-7.69.1

[curl -V output]
N/A: I just build the library.

operating system

Linux nlbaldev6.infor.com 3.10.0-957.27.2.el7.x86_64 #1 SMP Tue Jul 9 16:53:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[RuurdBeerstra]

Compiled the binary too just now, the -V output is:
rbeerstra@nlbaldev6: ./64bit/curl-7.69.1/build/bin/curl -V
curl 7.69.1 (x86_64-pc-linux-gnu) libcurl/7.69.1 wolfSSL/4.4.0 zlib/1.2.11 libssh2/1.9.0
Release-Date: 2020-03-11
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: IPv6 Largefile libz MultiSSL SSL TLS-SRP UnixSockets

Hope that helps.

[bagder]

This is not a bug, but a known limitation (set).

1. We don't have any NTLM support with wolfSSL so SMB gets disabled.
2. We don't have https-proxy support with wolfSSL

The reason is simply because nobody wrote the necessary code for it.

[RuurdBeerstra]

Odd: You get NTLM and SMB with OpenSSL but not with WolfSSL. WolfSSL has an OpenSSL compatibility layer that I've been using to fool all sorts of software into thinking it is using OpenSSL, while in reality it is linked against WolfSSL. So I thought: How hard can it be?

So I've added a few "defined(USE_OPENSSL) || defined(USE_WOLF_SSL)" where it only used USE_OPENSSL and now it builds and works and supports NTLM + SMB on Wolf.
I've created a patch (attached). I struggled more with git than with curl (must be me being the git here :-( ) . Could not get it to accept my pull request. So I attach the patch here.

0001-Add-WolfSSL-support-to-NTLM.zip

[bagder]

curl doesn't use the OpenSSL compatibility layer - I presume your patch here then requires that wolfSSL is built and provided with that support?

[bagder]

My regular curl-wolfSSL build fails to link with this patch applied since these symbols are missing:

DES_set_odd_parity, DES_set_key, DES_ecb_encrypt

Wouldn't it make more sense to provide a "real" wolfSSL integration for these functions rather than making users also build the OpenSSL API just for these?

Or if we do want to rely on that API, can we make the code check for the existence of that and only use NTLM if this API is present?

[RuurdBeerstra]

Yes – that only requires building Wolf with --enable-openssh, --enable-opensslall and --enable-opensslextra. That enables a whole bunch of #defines to rename OpenSSL functions to be wolfssl_something instead of just “something”. That allows WolfSSL to pretend it is OpenSSL. Your colleagues at WolfSSL added extra OpenSSL compatibility because we (Infor) asked for it. Version 4.4.0 is the one I use now. The quick patch I did “Works for me”, I did not try all builds & tests of cURL. I guess a decent patch would have to test for OpenSSL compatibility in the Wolf libraries. Or just let it fail? The error messages from the compiler about missing OpenSSL functions are clear enough…. My simple patch just equates OpenSSL and WolfSSL for NTLM support… Regards, Ruurd From: Daniel Stenberg <notifications@github.com> Sent: Thursday, 11 June 2020 17:13 To: curl/curl <curl@noreply.github.com> Cc: Ruurd Beerstra <Ruurd.Beerstra@infor.com>; Author <author@noreply.github.com> Subject: Re: [curl/curl] Building curl against WolfSSL loses protocols and features compared to OpenSSL (#5548) Sent by an external sender ------------------------------------ curl doesn't use the OpenSSL compatibility layer - I presume your patch here then requires that wolfSSL is built and provided with that support? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#5548 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKYUE3TXMJUW3KLQA2NFTBLRWDYAHANCNFSM4N2NCHGA>.

[RuurdBeerstra]

That is because you redefine those function based on the OpenSSL version number (in the curl_ntlm_core.c file). 61 # if (OPENSSL_VERSION_NUMBER < 0x00907001L) WolfSSL does not provide that version by default (there is a separate include for it) but Wolf is a new & improved OpenSSL, so I made that: 61 # if (OPENSSL_VERSION_NUMBER < 0x00907001L) && !defined(USE_WOLFSSL) And that fixed it (for me) Ruurd From: Daniel Stenberg <notifications@github.com> Sent: Thursday, 11 June 2020 17:23 To: curl/curl <curl@noreply.github.com> Cc: Ruurd Beerstra <Ruurd.Beerstra@infor.com>; Author <author@noreply.github.com> Subject: Re: [curl/curl] Building curl against WolfSSL loses protocols and features compared to OpenSSL (#5548) Sent by an external sender ------------------------------------ My regular curl-wolfSSL build fails to link with this patch applied since these symbols are missing: DES_set_odd_parity, DES_set_key, DES_ecb_encrypt Wouldn't it make more sense to provide a "real" wolfSSL integration for these functions rather than making users also build the OpenSSL API just for these? Or if we do want to reply on that API, can we make the code check for the existence of that and only use NTLM if this API is present? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#5548 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKYUE3S6DMT3GC3KBRU5WMTRWDZFBANCNFSM4N2NCHGA>.

[bagder]

I'm suggesting that it is adding a dependency on a part of wolfSSL without a very strong motivation and one that users most likely would prefer to avoid since it adds footprint. curl-wolfSSL users haven't needed to enable that before.

Of course the OpenSSL mimicking functions only exist in there if the OpenSSL API is provided, by I figured it is likely that the underlying DES functionality is already there and can be used directly using the wolfSSL provided API - thus avoiding the need for the OpenSSL API that we've worked so hard to avoid so far.

But yes, otherwise we can check for the functions in the configure script and make the NTLM support with wolfSSL rely on that.

[RuurdBeerstra]

I add maximum OpenSSL compatibility flags to Wolf because we need that for our product. There may be a more minimal set of flags that do not increase the WolfSSL footprint (you only need some basic crypto functions and they will be in a minimal WolfSSL build). Using the redefines from the OpenSSL compat layer may not be the most elegant solution, but it *does* make the NTLM module work with either SSL provider. There is something to be said for simplicity, too… But it is your code, your call. It would be nice if a future version of cURL would have NTLM with Wolf, but if not, I can now tweak our automated builds so it works for us… It is *way* past my quitting time here, going to turn this machine off, will be back on Monday. Thanks for the quick replies and support and have a nice weekend, Ruurd From: Daniel Stenberg <notifications@github.com> Sent: Thursday, 11 June 2020 17:35 To: curl/curl <curl@noreply.github.com> Cc: Ruurd Beerstra <Ruurd.Beerstra@infor.com>; Author <author@noreply.github.com> Subject: Re: [curl/curl] Building curl against WolfSSL loses protocols and features compared to OpenSSL (#5548) Sent by an external sender ------------------------------------ I'm suggesting that it is adding a dependency on a part of wolfSSL without a very strong motivation and one that users most likely would prefer to avoid since it adds footprint. curl-wolfSSL users haven't needed to enable that before. Of course the OpenSSL mimicking functions only exist in there if the OpenSSL API is provided, by I figured it is likely that the underlying DES functionality is already there and can be used directly using the wolfSSL provided API - thus avoiding the need for the OpenSSL API that we've worked so hard to avoid so far. But yes, otherwise we can check for the functions in the configure script and make the NTLM support with wolfSSL rely on that. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#5548 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKYUE3UUPRLDJJHSPS6H2BLRWD2TTANCNFSM4N2NCHGA>.

[bagder]

See #5556 for me trying to make a PR out of your patch with my sprinkles on top of it.