autotools: add support for 'unity' builds, enable in CI by vszakats · Pull Request #14815 · curl/curl

vszakats · 2024-09-06T16:59:07Z

Implement the "unity" builds as known from CMake, but for autotools.
It's limited to lib and src (CMake also supports it in tests).

Enable with: --enable-unity (disabled by default)

Unity builds speed up builds significantly. Cygwin and Windows builds in
particular, but the effect is noticeable on most systems. It also allows
discovering unity issues with autotools, benefitting also CMake when
building the same combination. In CI it makes turnaround times quicker.

This closes build performance with CMake. autotools still lags behind
because it builds shared and static libcurl in two, separate passes.
CMake does it in one. Manpage compilation isn't batched, it is in CMake.
After unity and test bundle support the slowest parts of the build are
the configuration phase (which is effectively a tedious, non-parallel,
compilation and/or linking of 300+ tiny programs. The next bottleneck
is compiling individual examples and finally test servers (only slow
with autotools).

The autotools implementation is slightly less efficient than CMake,
because 3 sources are permanently excluded while in CMake this isn't
necessary and solved more efficiently while building libtests. There is
also no 'unity' support for tests, making them a less efficient also.

Enable it in CI for most configure jobs. Except in GHA/dist (though
it works fine there too), to use the default config there. Also skip for
the Linux AWC-LC job where it made builds time a few seconds longer
(reason undiscovered.)

Autotools test suite builds compared between master → --enable-unity:

GHA/Linux: 32s → 12s
https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
https://github.com/curl/curl/actions/runs/10742978889/job/29796766297
GHA/macOS: 37s → 10s
https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
https://github.com/curl/curl/actions/runs/10742978699/job/29796768875
GHA/FreeBSD: 15m25 → 10m58 (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
https://github.com/curl/curl/actions/runs/10742978937/job/29796766115
GHA/Cygwin: 3m32 → 1m21
https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
https://github.com/curl/curl/actions/runs/10742978645/job/29796756933
GHA/MSYS2: 2m42 → 50s
https://github.com/curl/curl/actions/runs/10705668808/job/29681621166
https://github.com/curl/curl/actions/runs/10742978662/job/29799739289
GHA/mingw-w64: 5m32 → 1m23
https://github.com/curl/curl/actions/runs/10705668808/job/29681628787
https://github.com/curl/curl/actions/runs/10742978662/job/29799741568
Azure MSYS2 mingw64 openssl: 38m55 → 19m30 (all build)
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25521&view=logs&jobId=81bb90a1-709c-5261-a92d-4869708fd1b6&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
Azure Ubuntu default: 2m15 → 1m36 (all build)
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7
https://dev.azure.com/daniel0244/curl/_build/results?buildId=25521&view=logs&jobId=81bb90a1-709c-5261-a92d-4869708fd1b6&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7

rebase on lib: fix unity builds with BearSSL, MSH3, Quiche, OmniOS #14932.
rebase on autotools: tidy-ups in src/Makefile.inc #14933.
disable by default before merge.
enable in CI jobs.
re-test without test bundles and fix issues.

- move `EXTRA_DIST` to the top of file. - move `checksrc` init next to use. - use variable `HUGE` instead of repeating a literal. Cherry-picked from #14815 Closes #14933

- fix MSH3 static symbol clash. - fix Quiche static symbol clash. - fix local macro clash with BearSSL header. - fix local macro clash with OmniOS system header. ``` In file included from ../../lib/urldata.h:197, from ../../lib/altsvc.c:32, from libcurlall.c:2: ../../lib/cf-socket.h:55:25: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token 55 | #define sa_addr _sa_ex_u.addr | ^ In file included from ../../lib/urldata.h:197, from ../../lib/altsvc.c:32, from libcurlall.c:2: ../../lib/cf-socket.h:55:25: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token 55 | #define sa_addr _sa_ex_u.addr | ^ ``` Ref: https://github.com/curl/curl/actions/runs/10738314933/job/29781644299?pr=14772#step:3:6115 Discovered while adding support for "unity" builds for autotools. Required-by: #14922 Cherry-picked from #14815 Closes #14932

Linux event-based: https://github.com/curl/curl/actions/runs/10742237146/job/29794467019?pr=14815#step:39:516 macOS !ssl HTTP-only: https://github.com/curl/curl/actions/runs/10742237143/job/29794466939?pr=14815#step:15:499 macOS OpenSSL torture: https://github.com/curl/curl/actions/runs/10742237143/job/29794472332?pr=14815#step:15:497

vszakats · 2024-09-20T03:27:35Z

This new "unity" option for autotools is disabled by default, except if the env CURL_CI is
set, which enables it by default. This env already had some use in GHA, this PR sets it for
most workflows globally. In a few workflows it's enabled with --enable-unity and in two
workflows: distro and AWC-LC it's intentionally not enabled.

It does work fine in distro, but it's probably best to use the default there.

AWC-LC is a strange case where the build is already faster than elsewhere for some reason,
and unity makes the build a few seconds slower. (Perhaps something obvious, but I don't
know why this is, and I could not see similar results in other jobs.)

This PR is ready for merge.

This avoids surprises when a CI job does't replicate locally with the same options. `CURL_CI` also didn't save much lines as it turns out.

vszakats · 2024-09-20T10:04:08Z

Well, the CURL_CI trick didn't save much editing, while it reduced
transparency and local reproducibility, so went back to just enabling
--enable-unity manually where needed. There is no new "special
case" now, --enable-unity is disabled by default, always.

Unity mode seems to defeat the scanner and miss issues. before, miss: https://github.com/curl/curl/actions/runs/10967056702/job/30456136390 after, OK: https://github.com/curl/curl/actions/runs/10967128744/job/30456330732#step:35:1232 Tested with PR commit: 32854bb Follow-up to 60c3d04 #14815 Closes #14987

Sync it with cmake to: - exclude it from all builds except Windows and Cygwin. - exclude it from unity builds for Cygwin to avoid the included `windows.h` header interfere with the rest of the code. Also: - fix to trim ending spaces from `CSOURCES` for the `tidy` target. The solution requires a non-POSIX `-E` `sed` option. Supported by BSD and GNU implementations. Follow-up to 37523c9 #16480 Follow-up to 60c3d04 #14815 Follow-up to 7860f57 #12408 Closes #16712

- move `EXTRA_DIST` to the top of file. - move `checksrc` init next to use. - use variable `HUGE` instead of repeating a literal. Cherry-picked from curl#14815 Closes curl#14933

- fix MSH3 static symbol clash. - fix Quiche static symbol clash. - fix local macro clash with BearSSL header. - fix local macro clash with OmniOS system header. ``` In file included from ../../lib/urldata.h:197, from ../../lib/altsvc.c:32, from libcurlall.c:2: ../../lib/cf-socket.h:55:25: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token 55 | #define sa_addr _sa_ex_u.addr | ^ In file included from ../../lib/urldata.h:197, from ../../lib/altsvc.c:32, from libcurlall.c:2: ../../lib/cf-socket.h:55:25: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token 55 | #define sa_addr _sa_ex_u.addr | ^ ``` Ref: https://github.com/curl/curl/actions/runs/10738314933/job/29781644299?pr=14772#step:3:6115 Discovered while adding support for "unity" builds for autotools. Required-by: curl#14922 Cherry-picked from curl#14815 Closes curl#14932

Implement the "unity" builds as known from CMake, but for autotools. It's limited to `lib` and `src` (CMake also supports it in `tests`). Enable with: `--enable-unity` (disabled by default) Unity builds speed up builds significantly. Cygwin and Windows builds in particular, but the effect is noticeable on most systems. It also allows discovering unity issues with autotools, benefitting also CMake when building the same combination. In CI it makes turnaround times quicker. This closes build performance with CMake. autotools still lags behind because it builds shared and static libcurl in two, separate passes. CMake does it in one. Manpage compilation isn't batched, it is in CMake. After unity and test bundle support the slowest parts of the build are the configuration phase (which is effectively a tedious, non-parallel, compilation and/or linking of 300+ tiny programs. The next bottleneck is compiling individual examples and finally test servers (only slow with autotools). The autotools implementation is slightly less efficient than CMake, because 3 sources are permanently excluded while in CMake this isn't necessary and solved more efficiently while building libtests. There is also no 'unity' support for tests, making them a less efficient also. Enable it in CI for most `configure` jobs. Except in GHA/dist (though it works fine there too), to use the default config there. Also skip for the Linux AWC-LC job where it made builds time a few seconds longer (reason undiscovered.) Autotools test suite builds compared between master -> `--enable-unity`: - GHA/Linux: 32s -> 12s https://github.com/curl/curl/actions/runs/10705668823/job/29681617374 https://github.com/curl/curl/actions/runs/10742978889/job/29796766297 - GHA/macOS: 37s -> 10s https://github.com/curl/curl/actions/runs/10705668813/job/29681632885 https://github.com/curl/curl/actions/runs/10742978699/job/29796768875 - GHA/FreeBSD: 15m25 -> 10m58 (full workflow time, ~qemu) https://github.com/curl/curl/actions/runs/10705668811/job/29681607915 https://github.com/curl/curl/actions/runs/10742978937/job/29796766115 - GHA/Cygwin: 3m32 -> 1m21 https://github.com/curl/curl/actions/runs/10705668809/job/29681609965 https://github.com/curl/curl/actions/runs/10742978645/job/29796756933 - GHA/MSYS2: 2m42 -> 50s https://github.com/curl/curl/actions/runs/10705668808/job/29681621166 https://github.com/curl/curl/actions/runs/10742978662/job/29799739289 - GHA/mingw-w64: 5m32 -> 1m23 https://github.com/curl/curl/actions/runs/10705668808/job/29681628787 https://github.com/curl/curl/actions/runs/10742978662/job/29799741568 Closes curl#14815

Unity mode seems to defeat the scanner and miss issues. before, miss: https://github.com/curl/curl/actions/runs/10967056702/job/30456136390 after, OK: https://github.com/curl/curl/actions/runs/10967128744/job/30456330732#step:35:1232 Tested with PR commit: curl@32854bb Follow-up to 60c3d04 curl#14815 Closes curl#14987

Sync it with cmake to: - exclude it from all builds except Windows and Cygwin. - exclude it from unity builds for Cygwin to avoid the included `windows.h` header interfere with the rest of the code. Also: - fix to trim ending spaces from `CSOURCES` for the `tidy` target. The solution requires a non-POSIX `-E` `sed` option. Supported by BSD and GNU implementations. Follow-up to 37523c9 curl#16480 Follow-up to 60c3d04 curl#14815 Follow-up to 7860f57 curl#12408 Closes curl#16712

vszakats added build performance labels Sep 6, 2024

vszakats changed the title ~~autotools: add support for "unity" builds~~ autotools: add support for 'unity' builds Sep 6, 2024

github-actions Bot added the cmdline tool label Sep 6, 2024

vszakats force-pushed the am-unity branch from 43de4e8 to 3d6a4f6 Compare September 6, 2024 20:01

This was referenced Sep 15, 2024

GHA/windows: add MSVC vcpkg MSH3 job #14922

Closed

lib: fix unity builds with BearSSL, MSH3, Quiche, OmniOS #14932

Closed

autotools: tidy-ups in src/Makefile.inc #14933

Closed

vszakats added a commit that referenced this pull request Sep 19, 2024

autotools: tidy-ups in src/Makefile.inc

fdb8b40

- move `EXTRA_DIST` to the top of file. - move `checksrc` init next to use. - use variable `HUGE` instead of repeating a literal. Cherry-picked from #14815 Closes #14933

vszakats force-pushed the am-unity branch from 3d6a4f6 to 02f7759 Compare September 19, 2024 14:55

vszakats force-pushed the am-unity branch from 02f7759 to b7a0f13 Compare September 19, 2024 18:29

vszakats mentioned this pull request Sep 19, 2024

tests: speed up builds with single-binary test bundles #14772

Closed

19 tasks

github-actions Bot added the CI Continuous Integration label Sep 20, 2024

vszakats force-pushed the am-unity branch 2 times, most recently from db5f36f to 1fe2b9d Compare September 20, 2024 03:09

vszakats added 7 commits September 20, 2024 05:10

am: add unity support

2090047

disable by default except if CURL_CI env is set

7e237ee

set CURL_CI in jobs that benefit from it to enable unity with autotools

b566ad4

GHA/non-native: enable unity manually

978da1c

disable for AWC-LC, unity makes builds last a couple seconds longer

d7f2f48

enable unity for Circle CI too

ddf1373

vszakats force-pushed the am-unity branch from 1fe2b9d to ddf1373 Compare September 20, 2024 03:15

go traditional and pass --enable-unity directly

c43e9d4

This avoids surprises when a CI job does't replicate locally with the same options. `CURL_CI` also didn't save much lines as it turns out.

vszakats changed the title ~~autotools: add support for 'unity' builds~~ autotools: add support for 'unity' builds, enable in CI Sep 20, 2024

vszakats closed this in 60c3d04 Sep 20, 2024

vszakats deleted the am-unity branch September 20, 2024 21:55

vszakats mentioned this pull request Sep 20, 2024

GHA/linux: disable unity build for fix scanbuild job #14987

Closed

vszakats mentioned this pull request Mar 13, 2025

autotools: fix dllmain.c in unity builds #16712

Closed

Navigation Menu

Uh oh!

Uh oh!

autotools: add support for 'unity' builds, enable in CI - #14815

autotools: add support for 'unity' builds, enable in CI#14815
vszakats wants to merge 8 commits into
curl:masterfrom
vszakats:am-unity

vszakats commented Sep 6, 2024 •

edited

Loading

Uh oh!

vszakats commented Sep 20, 2024 •

edited

Loading

Uh oh!

vszakats commented Sep 20, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

vszakats commented Sep 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vszakats commented Sep 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vszakats commented Sep 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

1 participant

vszakats commented Sep 6, 2024 •

edited

Loading

vszakats commented Sep 20, 2024 •

edited

Loading

vszakats commented Sep 20, 2024 •

edited

Loading