cmake: add support for single libcurl compilation pass #11546

vszakats · 2023-07-30T13:08:51Z

Before this patch CMake builds used two separate compilation passes to
build the shared and static libcurl respectively. This patch allows to
reduce that to a single pass if the target platform and build settings
allow it.

This reduces CMake build times when building both static and shared
libcurl at the same time, making these dual builds an almost zero-cost
option.

Enable this feature for Windows builds, where the difference between the
two passes was the use of __declspec(dllexport) attribute for exported
API functions for the shared builds. This patch replaces this method
with the use of libcurl.def at DLL link time.

Also update Makefile.mk to use libcurl.def to export libcurl API
symbols on Windows. This simplifies (or fixes) this build method (e.g.
in curl-for-win, which generated a libcurl.def from .h files using
an elaborate set of transformations).

libcurl.def has the maintenance cost of keeping the list of public
libcurl API symbols up-to-date. This list seldom changes, so the cost
is low.

Closes #11546

curl previously had a libcurl.def placed in the lib directory. I opted for the root directory, because in lib it may be overwritten by .def files generated by the DLL linker command when using certain build methods/options.
single-pass libcurl builds can be freely extended to other platforms, if both shared/static libs have position-independent code enable and symbol exports can be fixed up via a linker option.
it might be an option to make libcurl.def an OS-agnostic raw list of public API symbols (= without the top EXPORTS line) for re-use for other purposes. CMake could handle that with some extra lines. In Makefile.mk less trivially, but also seems doable.

dfandrich · 2023-07-30T18:15:20Z

I thought the whole libcurl.def thing was unneeded because we're using __declspec(dllexport) on all the relevant exports.

dfandrich · 2023-07-30T18:16:45Z

It was removed the last time in commit 9bd7f00

vszakats · 2023-07-30T18:48:16Z

We do use declspec(), yes, and this is what forces separate compile passes for shared and static libs. The goal of this PR is to avoid that double work. Another positive side effect is that Makefile.mk builds can also use this libcurl.def, without having to generate it manually. Makefile.mk always used a single compile pass, so a .def file was always necessary for correct results (on Windows).

The file was deleted because it wasn't used. This time it will be used by two build systems.

Stop dynamically generating `libcurl.def` with the next curl release, which provides it and uses it automatically in GNU Make builds. curl/curl@2ebc74c curl/curl#11546

2ebc74c curl#11546 introduced sharing libcurl objects for shared and static targets. The above automatically enabled for Windows builds, with an option to disable with `SHARE_LIB_OBJECT=OFF`. This patch extend this feature for all platforms as a manual option. You can enable it by setting `SHARE_LIB_OBJECT=ON`. Shared objects will be built in PIC mode, meaning the static lib will also have PIC code. EXPERIMENTAL. Closes #xxxxx

2ebc74c #11546 introduced sharing libcurl objects for shared and static targets. The above automatically enabled for Windows builds, with an option to disable with `SHARE_LIB_OBJECT=OFF`. This patch extend this feature to all platforms as a manual option. You can enable it by setting `SHARE_LIB_OBJECT=ON`. Then shared objects are built in PIC mode, meaning the static lib will also have PIC code. [EXPERIMENTAL] Closes #11627

Before this patch CMake builds used two separate compilation passes to build the shared and static libcurl respectively. This patch allows to reduce that to a single pass if the target platform and build settings allow it. This reduces CMake build times when building both static and shared libcurl at the same time, making these dual builds an almost zero-cost option. Enable this feature for Windows builds, where the difference between the two passes was the use of `__declspec(dllexport)` attribute for exported API functions for the shared builds. This patch replaces this method with the use of `libcurl.def` at DLL link time. Also update `Makefile.mk` to use `libcurl.def` to export libcurl API symbols on Windows. This simplifies (or fixes) this build method (e.g. in curl-for-win, which generated a `libcurl.def` from `.h` files using an elaborate set of transformations). `libcurl.def` has the maintenance cost of keeping the list of public libcurl API symbols up-to-date. This list seldom changes, so the cost is low. Closes curl#11546

2ebc74c curl#11546 introduced sharing libcurl objects for shared and static targets. The above automatically enabled for Windows builds, with an option to disable with `SHARE_LIB_OBJECT=OFF`. This patch extend this feature to all platforms as a manual option. You can enable it by setting `SHARE_LIB_OBJECT=ON`. Then shared objects are built in PIC mode, meaning the static lib will also have PIC code. [EXPERIMENTAL] Closes curl#11627

Switch from `curl-gnumake.sh` to `curl-cmake.sh` with upcoming curl release v8.5.0. cmake builds are now _faster_ for Windows builds than raw gnumake (aka `Makefile.mk`). They also use 'unity' mode, which makes builds run fast with the side-effect of also creating potentially more efficient binaries by allowing better compiler optimizations. This also makes curl-for-win use the same build system for all target platforms (`Makefile.mk` is not available for *nix platforms). Initially on 2022-07-04 cmake was 25% slower than gnumake. By 2022-09-26 this reduced to 20%, by 2023-07-29 to 18% and after the latest round of updates gnumake came out 7% slower than cmake. This is for Windows, for a triple x64, arm64 and x86 build. In absolute times this is 27m22s for cmake and 29m11s for gnumake. (FWIW autotools builds are 52% slower than cmake unity builds now.) Making cmake builds fast was a multi-year effort with these major steps: 1. add support for cmake builds in curl-for-win. 420e73c 2. bring it in-line with gnumake and autotools builds and tidy-up as much as possible. Scattered to a many commits. 3. delete a whole bunch of unused feature detections. curl/curl@4d73854 curl/curl#9044 (and a lot more smaller commits) 4. speed up picky warning option initialization by avoiding brute-force all options. (first in libssh2, then in curl, then applied also ngtcp2, nghttp3, nghttp2) curl/curl@9c543de curl/curl#10973 5. implement single build run for shared and static libcurl + tool (first in libssh2) curl/curl@1199308 curl/curl#11505 53dcd49 6. implement single build pass for shared and static libcurl (for Windows initially) curl/curl@2ebc74c curl/curl#11546 7. extend above to non-Windows (e.g. macOS) curl/curl@fc9bfb1 curl/curl#11627 bafa77d (mac) 1b27304 (linux) 8. implement unity builds: single compiler invocation for libcurl + tool curl/curl@3f8fc25 curl/curl#11095 curl/curl@f42a279 curl/curl#11928 67d7fd3 9. speed up 4x the cmake initialization phase (for Windows) curl/curl@2100d9f curl/curl#12044 10. speed up cmake initialization even more by pre-filling detection results for our well-known mingw-w64 environments. 74dd967 5a43c61 +1: speed up autotools initialization by mitigating a brute-force feature detection on Windows. This reduced total build times by 5 minutes at the time, for the 3 Windows targets in total. curl/curl@6adcff6 curl/curl#9591 Also update build times for cmake-unity and gnumake, based on runs: cmake-unity: https://ci.appveyor.com/project/curlorg/curl-for-win/builds/48357875 gnumake: https://ci.appveyor.com/project/curlorg/curl-for-win/builds/48358005

- 898b012 curl#1288 - 5de6848 curl#10023 - bunch of others that are completed - `NTLM_WB_ENABLED` is implemented in a basic form, and now also scheduled for removal, so a TODO at this point isn't useful. And this 'to-check' item: Q: "The cmake build selected to run gcc with -fPIC on my box while the plain configure script did not." A: With CMake, since 2ebc74c curl#11546 and fc9bfb1 curl#11627, we explicitly enable PIC for libcurl shared lib. Or when building libcurl for shared and static lib in a single pass. We do this by default for Windows or when enabled by the user via `SHARE_LIB_OBJECT`. Otherwise we don't touch this setting. Meaning the default set by CMake (if any) or the toolchain is used. On Debian Bookworm, this means that PIC is disabled for static libs by default. Some platforms (like macOS), has PIC enabled by default. autotools supports the double-pass mode only, and in that case CMake seems to match PIC behaviour now (as tested on Linux with gcc.) Follow-up to 5d5dfdb curl#12500 Closes #xxxxx

- manual completed: 898b012 #1288 - soname completed: 5de6848 #10023 - bunch of others that are completed - `NTLM_WB_ENABLED` is implemented in a basic form, and now also scheduled for removal, so a TODO at this point isn't useful. And this 'to-check' item: Q: "The cmake build selected to run gcc with -fPIC on my box while the plain configure script did not." A: With CMake, since 2ebc74c #11546 and fc9bfb1 #11627, we explicitly enable PIC for libcurl shared lib. Or when building libcurl for shared and static lib in a single pass. We do this by default for Windows or when enabled by the user via `SHARE_LIB_OBJECT`. Otherwise we don't touch this setting. Meaning the default set by CMake (if any) or the toolchain is used. On Debian Bookworm, this means that PIC is disabled for static libs by default. Some platforms (like macOS), has PIC enabled by default. autotools supports the double-pass mode only, and in that case CMake seems to match PIC behaviour now (as tested on Linux with gcc.) Follow-up to 5d5dfdb #12500 Reviewed-by: Jay Satiro Closes #12509

vszakats added build cmake Windows Windows-specific labels Jul 30, 2023

vszakats marked this pull request as draft July 30, 2023 13:08

vszakats force-pushed the build-add-libcurl-def branch from 3e9189a to 022ccf0 Compare July 30, 2023 22:02

vszakats added 3 commits July 30, 2023 22:10

libcurl.def: add

775b7bb

Makefile.mk: use libcurl.def

9800171

cmake: share libcurl objects + use libcurl.def

fdf9386

vszakats force-pushed the build-add-libcurl-def branch from 022ccf0 to fdf9386 Compare July 30, 2023 22:10

vszakats marked this pull request as ready for review July 30, 2023 22:20

vszakats changed the title ~~cmake: add support for single libcurl compilation pass on Windows~~ cmake: add support for single libcurl compilation pass Jul 31, 2023

vszakats closed this in 2ebc74c Aug 1, 2023

vszakats deleted the build-add-libcurl-def branch August 1, 2023 08:15

vszakats added the performance label Aug 1, 2023

vszakats mentioned this pull request Aug 8, 2023

cmake: allow SHARE_LIB_OBJECT=ON on all platforms #11627

Closed

vszakats mentioned this pull request Dec 12, 2023

cmake: delete obsolete TODOs more [ci skip] #12509

Closed

vszakats mentioned this pull request Feb 12, 2024

CMake and Zlib madler/zlib#831

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmake: add support for single libcurl compilation pass #11546

cmake: add support for single libcurl compilation pass #11546

vszakats commented Jul 30, 2023 •

edited

Loading

dfandrich commented Jul 30, 2023 via email

dfandrich commented Jul 30, 2023 via email

vszakats commented Jul 30, 2023 •

edited

Loading

cmake: add support for single libcurl compilation pass #11546

cmake: add support for single libcurl compilation pass #11546

Conversation

vszakats commented Jul 30, 2023 • edited Loading

dfandrich commented Jul 30, 2023 via email

dfandrich commented Jul 30, 2023 via email

vszakats commented Jul 30, 2023 • edited Loading

vszakats commented Jul 30, 2023 •

edited

Loading

vszakats commented Jul 30, 2023 •

edited

Loading