CMake: Support for DarwinSSL and mbedTLS + cleanup #1228

Closed
wants to merge 4 commits into
from

Projects

None yet

7 participants

@ligfx
Contributor
ligfx commented Jan 26, 2017

No description provided.

@ligfx, thanks for your PR! By analyzing the history of the files in this pull request, we identified @Lekensteyn, @jzakrzewski and @jgsogo to be potential reviewers.

@jay jay added the cmake label Jan 26, 2017
CMake/FindMbedTLS.cmake
+find_library(MBEDCRYPTO_LIBRARY mbedcrypto)
+
+set(MBEDTLS_INCLUDE_DIRS ${MBEDTLS_INCLUDE_DIR})
+set(MBEDTLS_LIBRARIES ${MBEDTLS_LIBRARY} ${MBEDX509_LIBRARY} ${MBEDCRYPTO_LIBRARY})
Lekensteyn
Lekensteyn Feb 1, 2017 Member

Probably a good idea to quote each of these variables (in case it contains spaces). E.g. "${MBEDTLS_LIBRARY}" "${MBEDX509_LIBRARY}" ...) (same for previous include_dirs line)

ligfx
ligfx Feb 3, 2017 Contributor

Ah, good catch! The quoting rules always trip me up.

CMakeLists.txt
+ if(SECURITY_FRAMEWORK)
+ set(SSL_ENABLED ON)
+ set(USE_DARWINSSL ON)
+ list(APPEND CURL_LIBS "-framework CoreFoundation -framework Security")
Lekensteyn
Lekensteyn Feb 1, 2017 Member

Does this work? Don't you have separate the two options like list(APPEND CURL_LIBS "-framework CoreFoundation" "-framework Security")?

(Specifying -framework should work according to cmake-commands manual (target_link_libraries), hopefully it does (untested).)

ligfx
ligfx Feb 3, 2017 edited Contributor

Any library that begins with - gets passed straight through to the command-line, so it works as two options or as one option. After taking another look at this code, I replaced this with "${COREFOUNDATION_FRAMEWORK}" "${SECURITY_FRAMEWORK}", since it seems more correct (and it already goes to the trouble of finding Security.framework, so might as well...)

@webmaster128

Nice, thanks for pushing macOS+cmake. I'll test this on a Mac soon. The current change does not break my custom OpenSSL configuration on Linux, so 👍

CMakeLists.txt
-if(CMAKE_USE_OPENSSL)
+option(CMAKE_USE_OPENSSL "Use OpenSSL code. Experimental" ON)
+if(CMAKE_USE_OPENSSL AND NOT SSL_ENABLED)
webmaster128
webmaster128 Feb 19, 2017 edited Contributor

This silently prioritizes Secure Channel over OpenSSL when configuring CMAKE_USE_WINSSL=1 CMAKE_USE_OPENSSL=1 on windows.

I think this priority should at least be written down as a comment about TLS at the top, But the safe thing would be to disallow setting multiple TLS providers.

Contributor
webmaster128 commented Feb 20, 2017 edited

When I apply this patch, I get a runtime error when setting CURLOPT_SSL_OPTIONS to CURLSSLOPT_NO_REVOKE. Any idea where this is coming from? It works with cmake build + OpenSSL and buildconf/configure build + DarwnSSL.

Owner
bagder commented Feb 20, 2017

I get a runtime error

Can you elaborate on exactly what you get when you do what?

Contributor

The call of curl_easy_setopt with key CURLOPT_SSL_OPTIONS and value CURLSSLOPT_NO_REVOKE returns the code 48 = CURLE_UNKNOWN_OPTION.

Owner
bagder commented Feb 20, 2017

Oh, that would imply that the USE_SSL define isn't defined, looking at url.c just lines above the case CURLOPT_SSL_OPTIONS: line. That would rather imply that you get a build completely without SSL?

Contributor
webmaster128 commented Feb 20, 2017 edited

I guess USE_SSL is something internal, that CMakeLists.txt should set, right? What is the difference between USE_SSL and SSL_ENABLED?

I cannot really tell what is included in the build. Since my application code fails when doing the above configuration, even my HTTP tests without SSL fail. Is the any useful debugging that I can perform on the .a file to answer your question?

Owner
bagder commented Feb 20, 2017

It is internal but it depends on one of the TLS enabled variables being set. See https://github.com/curl/curl/blob/master/lib/curl_setup.h#L618

Contributor

I guess there is a USE_DARWINSSL block missing in my auto-generated curl_config.h.

CMakeLists.txt
+ if(CMAKE_USE_DARWINSSL AND NOT SSL_ENABLED)
+ find_library(COREFOUNDATION_FRAMEWORK "CoreFoundation")
+ find_library(SECURITY_FRAMEWORK "Security")
+ if(SECURITY_FRAMEWORK)
webmaster128
webmaster128 Feb 20, 2017 Contributor

I think this block does not properly handle the absense of required libraries. What about

    find_library(COREFOUNDATION_FRAMEWORK CoreFoundation)
    if (NOT COREFOUNDATION_FRAMEWORK)
        message(FATAL ERROR "CoreFoundation framework not found")
    endif()

    find_library(SECURITY_FRAMEWORK Foundation)
    if (NOT SECURITY_FRAMEWORK)
        message(FATAL ERROR "Security framework not found")
    endif()

    set(SSL_ENABLED ON)
    set(USE_DARWINSSL ON)
    list(APPEND CURL_LIBS "${COREFOUNDATION_FRAMEWORK}" "${SECURITY_FRAMEWORK}")
ligfx
ligfx Feb 21, 2017 Contributor

You're right (even though in practice they'll never be absent). I added the checks.

@webmaster128

While testing this change, I found that there is DarwinSSL block missing in curl_config.cmake, see https://github.com/curl/curl/pull/1270/files

Contributor
ligfx commented Feb 21, 2017

Thanks so much for testing this out and investigating that issue, @webmaster128. I've cherry-picked your commit from the other PR into this one.

I also added another commit that checks that at most one SSL option is enabled. If multiple options are defined, it raises an error like this:

CMake Error at CMakeLists.txt:307 (message):
  Multiple SSL options specified: CMAKE_USE_DARWINSSL;CMAKE_USE_OPENSSL.
  Please pick at most one and disable the rest.
CMakeLists.txt
+
+ find_library(SECURITY_FRAMEWORK "Security")
+ if(NOT SECURITY_FRAMEWORK)
+ message(FATAL ERROR "Security framework not found")
webmaster128
webmaster128 Feb 21, 2017 Contributor

There is an underscore missing (my fault, I guess). It is FATAL_ERROR, see https://cmake.org/cmake/help/v2.8.8/cmake.html#command%3amessage

Same for the block above.

ligfx
ligfx Feb 21, 2017 Contributor

It does the same thing regardless and aborts CMake, right? 👍 Fixed.

webmaster128
webmaster128 Feb 21, 2017 Contributor

FATAL ERROR was a typo that I copied from my own project where I wrote it the wrong way

CMakeLists.txt
+ message(FATAL ERROR "Security framework not found")
+ endif()
+
+ if(SECURITY_FRAMEWORK)
webmaster128
webmaster128 Feb 21, 2017 Contributor

Now this check is obsolete

ligfx
ligfx Feb 21, 2017 Contributor

Good catch! This code looks much cleaner now, that was a great idea to disallow multiple SSL provider options.

CMakeLists.txt
string(REGEX REPLACE "([a-zA-Z_][a-zA-Z0-9_]*)[\t ]*=[\t ]*([^\n]*)" "SET(\\1 \\2)" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})
- string(REPLACE "§!§" "\n" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})
+ string(REPLACE "�!�" "\n" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})
webmaster128
webmaster128 Feb 21, 2017 Contributor

I think you accidentally changed these lines. See #1271

ligfx
ligfx Feb 21, 2017 Contributor

I responded on your PR.

Contributor

Great! I found some minor other stuff. I'll test again soon

CMakeLists.txt
+ CMAKE_USE_OPENSSL
+ CMAKE_USE_MBEDTLS
+)
+if(enabled_ssl_options MATCHES ";")
webmaster128
webmaster128 Feb 21, 2017 Contributor

Isn't there a way to get the length of a list for this check? Does this work in cmake 2.8? https://cmake.org/cmake/help/v3.0/command/list.html

ligfx
ligfx Feb 21, 2017 Contributor

There is, but that would add another command to call. I'll add a list length output parameter to collect_true.

CMakeLists.txt
+ message(FATAL_ERROR "Security framework not found")
+ endif()
+
+ if(SECURITY_FRAMEWORK)
webmaster128
webmaster128 Feb 21, 2017 Contributor

The obsolete check is still in here

CMakeLists.txt
+option(CMAKE_USE_MBEDTLS "Enable mbedTLS for SSL/TLS" OFF)
+if(CMAKE_USE_MBEDTLS)
+ find_package(MbedTLS)
+ if(MBEDTLS_FOUND)
webmaster128
webmaster128 Feb 21, 2017 Contributor

This should fail hard and early when package MbedTLS was not found (like we discussed for CoreFoundation/Security)

ligfx
ligfx Feb 21, 2017 Contributor

Good catch, I see the pattern now. I'll also do the same thing for OpenSSL.

CMakeLists.txt
string(REGEX REPLACE "([a-zA-Z_][a-zA-Z0-9_]*)[\t ]*=[\t ]*([^\n]*)" "SET(\\1 \\2)" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})
- string(REPLACE "§!§" "\n" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})
+ string(REPLACE "�!�" "\n" MAKEFILE_INC_TEXT ${MAKEFILE_INC_TEXT})
webmaster128
webmaster128 Feb 21, 2017 Contributor

When you replace to "" in 1075, line 1077 should be removed

ligfx
ligfx Feb 21, 2017 Contributor

I don't understand. I'm not changing line 1075/1077. These changes were accidental and I'll remove them from the PR.

webmaster128
webmaster128 Feb 21, 2017 Contributor

Have a look at the full diff and you'll see that there is still a change that does not really make sense: https://github.com/curl/curl/pull/1228/files

webmaster128
webmaster128 Feb 21, 2017 Contributor

... which is gone right now :)

Contributor
webmaster128 commented Feb 21, 2017 edited

My runtime issue is gone now (just verified the current patch).

What now happens is the following:

// apply this PR onto current master and commit

$ git clean -xdf

$ cmake -DCMAKE_USE_DARWINSSL=1 -DHTTP_ONLY=1 -Wno-dev
// long healthy output ending with:
-- Enabled features: DarwinSSL IPv6 unix-sockets libz Largefile NTLM
-- Enabled protocols: HTTP HTTPS
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/me/myorg/curlwrapper/curl

$ cmake -DCMAKE_USE_DARWINSSL=1 -DHTTP_ONLY=1 -Wno-dev
CMake Warning at CMakeLists.txt:49 (message):
  the curl cmake build system is poorly maintained.  Be aware


-- curl version=[7.53.0-DEV]
Found *nroff option: -- -man
CMake Error at CMakeLists.txt:307 (message):
  Multiple SSL options specified: CMAKE_USE_DARWINSSL;CMAKE_USE_OPENSSL.
  Please pick at most one and disable the rest.


-- Configuring incomplete, errors occurred!

When you look at the cache, you see, that there are two CMAKE_USE_*SSL variables set:

//enable Apple OS native SSL/TLS
CMAKE_USE_DARWINSSL:BOOL=ON

//Use GSSAPI implementation (right now only Heimdal is supported
// with CMake build)
CMAKE_USE_GSSAPI:BOOL=OFF

//Use libSSH2
CMAKE_USE_LIBSSH2:BOOL=OFF

//Enable mbedTLS for SSL/TLS
CMAKE_USE_MBEDTLS:BOOL=OFF

//Use OpenSSL code. Experimental
CMAKE_USE_OPENSSL:BOOL=ON

This is because CMAKE_USE_OPENSSL is ON by default. It is not caugth in the first run because the option is set after the multiple ssl check is performed.

I think the most sustainable way to fix this is to default CMAKE_USE_OPENSSL to OFF.

Contributor
ligfx commented Feb 21, 2017

@webmaster128 Thanks for the further testing!

The Autoconf build defaults to enabling OpenSSL. I think enabling some form of SSL by default is a good idea since most users expect curl to be able to make secure connections.

I can move the option definitions to the top of "check for SSL" block, before the multiple SSL check is performed. In some ways that's cleaner than before, and makes it easier to do things like different defaults per-OS (if that's ever wanted). That would look like this:

option(CMAKE_USE_OPENSSL "Use OpenSSL code. Experimental" ON)
option(CMAKE_USE_MBEDTLS "Enable mbedTLS for SSL/TLS" OFF)
if(APPLE)
  option(CMAKE_USE_DARWINSSL "enable Apple OS native SSL/TLS" OFF)
endif()
if(WIN32)
  option(CMAKE_USE_WINSSL "enable Windows native SSL/TLS" OFF)
  cmake_dependent_option(CURL_WINDOWS_SSPI "Use windows libraries to allow NTLM authentication without openssl" ON
    CMAKE_USE_WINSSL OFF)
endif()

collect_true(enabled_ssl_options enabled_ssl_options_count
  CMAKE_USE_WINSSL
  CMAKE_USE_DARWINSSL
  CMAKE_USE_OPENSSL
  CMAKE_USE_MBEDTLS
)
if(enabled_ssl_options_count GREATER 1)
  message(FATAL_ERROR "Multiple SSL options specified: ${enabled_ssl_options}. Please pick at most one and disable the rest.")
endif()

if(CMAKE_USE_WINSSL)
  set(SSL_ENABLED ON)
  set(USE_SCHANNEL ON) # Windows native SSL/TLS support
  set(USE_WINDOWS_SSPI ON) # CMAKE_USE_WINSSL implies CURL_WINDOWS_SSPI
  list(APPEND CURL_LIBS "crypt32")
endif()
if(CURL_WINDOWS_SSPI)
  set(USE_WINDOWS_SSPI ON)
endif()

if(CMAKE_USE_DARWINSSL)
  find_library(COREFOUNDATION_FRAMEWORK "CoreFoundation")

Would that work for you?

Contributor
webmaster128 commented Feb 21, 2017 edited

This means everyone using a different TLS provider has to explicitly disable OpenSSL, right? cmake -DCMAKE_USE_DARWINSSL=1 would not work anymore. I think the default should only be set when no other TLS provider is enabled.

Apart from that, the drafted version looks good to me.

Contributor
ligfx commented Feb 21, 2017

@webmaster128 Okay, I've updated the PR with that logic.

Contributor

All tests run green now :)

The only question I have left is, if there is an equivalent of --without-ca-bundle --without-ca-path from my old build script. Without this parameters I was having the issue that CA files where searched on a user's machine because they were present on the build machine. Is that in the scope of this PR?

Owner
bagder commented Feb 22, 2017

I suggest to take that in a separate PR, as that's rather independent of specific TLS backend.

Contributor

Okay, so I am very happy with is right now.

Note: This PR changes behavior on Windows, where OpenSSL is now enabled by default and will fail with "OpenSSL not found" when OpenSSL is not found unless OpenSSL is disables explicitly. I guess this is okay, but still different to what we have in CMAKE on master.

Contributor
ligfx commented Feb 22, 2017

I didn't realize I changed that behavior. I've changed the PR so it acts the same way as before (doesn't default to OpenSSL on Windows).

@Lekensteyn

Looks mostly good, but add REQUIRED to find_package

CMakeLists.txt
+ find_library(COREFOUNDATION_FRAMEWORK "CoreFoundation")
+ if(NOT COREFOUNDATION_FRAMEWORK)
+ message(FATAL_ERROR "CoreFoundation framework not found")
+ endif()
Lekensteyn
Lekensteyn Feb 24, 2017 Member

These three lines can be removed if you add REQUIRED to find_library (see cmake-commands manpage)

webmaster128
webmaster128 Feb 25, 2017 Contributor

I don't think there is REQUIRED in find_library. At least it is not documented.

Lekensteyn
Lekensteyn Feb 25, 2017 Member

You're right, there is no REQUIRED for find_library

CMakeLists.txt
- set(CURL_WINDOWS_SSPI ON)
+ if(NOT OPENSSL_FOUND)
+ message(FATAL_ERROR "OpenSSL not found")
+ endif()
Lekensteyn
Lekensteyn Feb 24, 2017 Member

These three lines can be removed if you add "REQUIRED" to find_package

ligfx
ligfx Feb 25, 2017 Contributor

Good catch. Fixed both of these.

CMakeLists.txt
+if(CMAKE_USE_MBEDTLS)
+ find_package(MbedTLS)
+ if(NOT MBEDTLS_FOUND)
+ message(FATAL_ERROR "mbed TLS not found")
endif()
Lekensteyn
Lekensteyn Feb 24, 2017 Member

REQUIRED to find_package

Member
Lekensteyn commented Feb 25, 2017 edited

Hmm strange, why is CI failing here: https://travis-ci.org/curl/curl/jobs/205374812#L3906

Maybe it is an intermittent error. Can you try to update the INSTALL.cmake file with the updated state for SSL library support and re-trigger CI?

Contributor
ligfx commented Feb 26, 2017

@Lekensteyn Yeah it must be, none of the files touched in this PR are involved in that build process. I've updated the INSTALL.cmake file and re-pushed.

Owner
jay commented Feb 26, 2017

Hmm strange, why is CI failing here: https://travis-ci.org/curl/curl/jobs/205374812#L3906

From what I understand pipelining there is a timing component in what order the data is received so occasionally they fail.

@Lekensteyn

LGTM, still works on Arch Linux with OpenSSL.

Member

@jay When merging these patches, should only the last commit message to include "Closes ..."? (Is something else needed?)

Contributor

Btw., is CMAKE_* really a naming convention that should be used for the public curl configuration API? As far as I can see, CMAKE_* variables usually come from the CMAKE system (CMAKE_SYSTEM_NAME, CMAKE_C_FLAGS and so on).

The name CMAKE_USE_OPENSSL was introduced in 2009 probably just to avoid a conflict with USE_OPENSSL.

Maybe those should better be prefixed with CURL_*

Contributor
ligfx commented Feb 27, 2017

@webmaster128 I totally agree. That's a good focus for another PR.

Owner
jay commented Feb 27, 2017 edited

When merging these patches, should only the last commit message to include "Closes ..."? (Is something else needed?)

Typically if it were me I would squash all commits or ask the reporter to do it. I see this is a few things in one PR and there are some distinct changes it may be useful leave separate, however this is more work.

'Closes' directive is never needed as a matter of protocol, it could be closed manually. What is important though is that we have some kind of a reference Closes/Fixes/Bug/Ref:, for example

Ref: https://github.com/curl/curl/pull/1228

Ideally something like that would near the end of each of these commit messages, so git blame nobody has to play detective. Off the top of my head I can't easily think of an automated way to append to all the commit messages at once.

Right now there is:

CMake: Add support for DarwinSSL
CMake: Reorganized SSL support, separated WinSSL and SSPI
CMake: Add mbedTLS support
CMake: at most one SSL library may be set
Add missing DarwinSSL block to curl_config.cmake
CMake: update INSTALL about new SSL lib support

At the very least the commit messages should be modified to meet the style format. That would involve changing the subjects to imperative present tense, adding missing area, ref, etc:

CMake: Add support for DarwinSSL
CMake: Reorganize SSL support, separate WinSSL and SSPI
CMake: Add mbedTLS support
CMake: Set at most one SSL library
CMake: Add missing DarwinSSL block to curl_config.cmake
CMake: update INSTALL about new SSL lib support

Then after rebase make sure to confirm that you're the committer on all of them, and the authors are left intact.

If it were me I'd lose at least these squashable 2:

Pick Add missing DarwinSSL block to curl_config.cmake into CMake: Add support for DarwinSSL and add a Assisted-by: Simon Warta line to that commit message.

Squash CMake: update INSTALL about new SSL lib support into CMake: at most one SSL library may be set.

CMake: Add support for DarwinSSL
CMake: Reorganize SSL support, separate WinSSL and SSPI
CMake: Add mbedTLS support
CMake: Set at most one SSL library
Contributor

I totally agree. That's a good focus for another PR.

I can understand that you want this (excellent) piece of work to be done, but my concern is that this introduces two new public variables in a naming format that is at least confusing. As soon as this is in, we need to think about compatibility and I don't think it is worth carrying around compatibility code for a decade just because CMAKE_USE_DARWINSSL and CMAKE_USE_MBEDTLS have been used for one or two weeks, maybe including a release.

Contributor

I think we should do this once and good. The names for the options used in command line should be short and simple - without any prefix. After all the user is building curl and he/she and we know that so why should we force them to type those prefixes?

So I'd say - make a separate PR, cleanup all names, break the compatibility once but do it the right way.

Member

@ligfx Can you squash the commits according to the suggestions from @jay?

@webmaster128 @jzakrzewski There are more inconsistent names (CMAKE_USE_GSSAPI, CMAKE_USE_LIBSSH2), feel free to propose a patch tot rename them (and for backwards compat, the default for the renamed options could use the former).

Maybe it would be even better to introduce a separate CMakeOptions.txt to keep all options in one place. See for example https://github.com/nghttp2/nghttp2/blob/master/CMakeOptions.txt

Contributor
ligfx commented Feb 28, 2017
@Lekensteyn Lekensteyn added a commit that referenced this pull request Mar 5, 2017
@ligfx @Lekensteyn ligfx + Lekensteyn CMake: Reorganize SSL support, separate WinSSL and SSPI
This is closer to how configure.ac does it

Ref: #1228
f85ff14
@Lekensteyn Lekensteyn added a commit that referenced this pull request Mar 5, 2017
@ligfx @Lekensteyn ligfx + Lekensteyn CMake: Add DarwinSSL support
Assisted-by: Simon Warta <simon@kullo.net>
Ref: #1228
eb19e89
@Lekensteyn Lekensteyn added a commit that referenced this pull request Mar 5, 2017
@ligfx @Lekensteyn ligfx + Lekensteyn CMake: Add mbedTLS support
Ref: #1228
c6a9746
@Lekensteyn Lekensteyn added a commit that referenced this pull request Mar 5, 2017
@ligfx @Lekensteyn ligfx + Lekensteyn CMake: Set at most one SSL library
Ref: #1228
813263d
Member

Merged via:
f85ff14 CMake: Reorganize SSL support, separate WinSSL and SSPI
eb19e89 CMake: Add DarwinSSL support
c6a9746 CMake: Add mbedTLS support
813263d CMake: Set at most one SSL library

Thanks for your contribution @ligfx!

@Lekensteyn Lekensteyn closed this Mar 5, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment