cmake: add support for "unity" builds

[vszakats]

Aka "jumbo" or "amalgamation" builds. It means to compile all sources
per target as a single C source. This is experimental.

You can enable it by passing -DCMAKE_UNITY_BUILD=ON to cmake.
It requires CMake 3.16 or newer.

It makes builds (much) faster, allows for better optimizations and tends
to promote less ambiguous code.

Also add a new AppVeyor CI job and convert an existing one to use
"unity" mode (one MSVC, one MinGW), and enable it for one macOS CI job.

Fix related issues:

• add missing include guard to easy_lock.h.
• rename static variables and functions (and a macro) with names reused
  across sources, or shadowed by local variables.
• add an #undef after use.
• add a missing #undef before use.
• move internal definitions from ftp.h to ftp.c.
• curl_memory.h fixes to make it work when included repeatedly.
• stop building/linking curlx bits twice for a static-mode curl tool.
  These caused doubly defined symbols in unity builds.
• silence missing extern declarations compiler warning for _CRT_glob.
• fix extern declarations for tool_freq and tool_isVistaOrGreater.
• fix colliding static symbols in debug mode: debugtime() and
  statename.
• rename ssl_backend_data structure to unique names for each
  TLS-backend, along with the ssl_connect_data struct member
  referencing them. This required adding casts for each access.
• add workaround for missing [P]UNICODE_STRING types in certain Windows
  builds when compiling lib/ldap.c. To support "unity" builds, we had
  to enable SCHANNEL_USE_BLACKLISTS for Schannel (a Windows
  schannel.h option) globally. This caused an indirect inclusion of
  Windows schannel.h from ldap.c via winldap.h to have it enabled
  as well. This requires [P]UNICODE_STRING types, which is apperantly
  not defined automatically (as seen with both MSVS and mingw-w64).
  This patch includes <subauth.h> to fix it.
  Ref: https://github.com/curl/curl/runs/13987772013
  Ref: https://dev.azure.com/daniel0244/curl/_build/results?buildId=15827&view=logs&jobId=2c9f582d-e278-56b6-4354-f38a4d851906&j=2c9f582d-e278-56b6-4354-f38a4d851906&t=90509b00-34fa-5a81-35d7-5ed9569d331c
• tweak unity builds to compile lib/memdebug.c separately in memory
  trace builds to avoid PP confusion.
• force-disable unity for test programs.
• do not compile and link libcurl sources to libtests twice when libcurl
  is built in static mode.

Ref: libssh2/libssh2#1034
Ref: https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html
Ref: https://en.wikipedia.org/wiki/Unity_build

Closes #11095

[vszakats]

Difficulties with H2, H2-proxy and H1-proxy name collisions, and CF_CTX_CALL_DATA vs. CF_DATA_CURRENT in particular.

The bigger hurdle is MultiSSL, especially ssl_backend_data, which is a name used for both type and variable and some other things I've yet to understand.

("unity" builds require unique symbol names across sources, including statics, types, and macros.)

[jay]

Is this worth the added maintenance and couldn't it introduce bugs? If we redefine a macro in one unit we don't then expect it to be redefined in another. Same thing for includes maybe one include is preferred over another in one unit and maybe the opposite in another unit. Unity might be faster for total rebuilds or clean builds but for every day development the compiler can usually just recompile affected units.

[vszakats]

I think the practice of using unique identifiers across the whole project has its own benefits and generally improves code quality. This applies to curl as well for the necessary updates so far. As for is it worth it, like everything, this has also opponents. I'm personally in favour. curl is built more often as a whole, than incrementally. The optimization benefits can also be considerable, though I'm not yet at the point to make measurements. For sure it makes full builds almost instantaneous (except the still lenghty configuration phase.)

Regarding bugs introduced: CI tests will tell us more once I got around adding one. But first this needs to work in more build combos than a default TLS one.

[MarcelRaad]

FWIW, looking at the first commit, I like the fixes independently of the question if we want to maintain this option or not. Maybe extract them to a separate PR, or at least commit, once they're done.

I find some of the changes in the second commit harder to understand when mixed with other things, like the change from struct ssl_backend_data *backend to void *backend. Maybe even make multiple commits for them to explain them in the commit message and to make them easier to review.

[vszakats]

@MarcelRaad: The only "unity"-specific change is the single new line in CMakeLists.txt. Even that is optional, and only used when unity mode is explicitly enabled with cmake. Rest of the changes is all about avoiding conflicts between source files, and managing PP tricks. (There is probably further opportunities to make names more consistent after these changes in the H1-H2-proxy sources. Changes here so far are limited to what was strictly necessary.)

ssl_backend_data is the name of the struct used in each TLS backend module for the TLS context. These need to be renamed to be unique otherwise they conflict. After renaming them (as done already for OpenSSL and Schannel), struct ssl_backend_data *backend no longer finds this struct under this shared name. I changed that to a void * and added casts for accesses. It's possible there is a superior/simpler/shorter solution. On the first round my goal is to make it build at all. I'm still learning how all these bits are fitting together (esp. in MultiSSL mode).

The next riddle is [UPDATE: might be the case of selective declarations inside warnless.h, depending on BUILDING_WARNLESS_C)]:

lib/warnless.c:370:1: warning: all paths through this function will call itself [-Winfinite-recursion]
{
^
lib/warnless.c:375:1: warning: all paths through this function will call itself [-Winfinite-recursion]
{
^
2 warnings generated.

[eli-schwartz]

The optimization benefits can also be considerable, though I'm not yet at the point to make measurements.

This is usually not meaningful compared to simply using LTO.

[bagder]

This is usually not meaningful compared to simply using LTO.

I suppose this PR can be used to test that exact claim...

[vszakats]

LTO: IME it makes builds slower and also fragile, especially with static builds (where any dependency issue will also be hit).

Remaining issues:

• write():

/my/curl/lib/vquic/curl_ngtcp2.c:292:43: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long long') to 'unsigned int' [-Wshorten-64-to-32]
    ssize_t rc = write(ctx->qlogfd, data, datalen);
                 ~~~~~                    ^~~~~~~
1 warning generated.

• A possibly odd regression when not using "unity" mode?: https://cirrus-ci.com/task/4638440571011072?logs=compile#L321 [FIXED with an hack. Maybe there is a more elegant one.]

Fixed two unrelated Windows issues with missing/misplaced externs.

Also had to add a tweak to CMake, to avoid compiling and linking CURLX sources twice when building a static curl tool.

[vszakats]

One AppVeyor CI job converted to unity, one other added. Both Windows,
the notorious problem-platform, so it will be enough for a start.

Caveats:

• Running tests doesn't work in all cases. This will be an exercise for the future.
• One random build combo (the new AppVeyor CI one with OpenSSL enabled
  alongside Schannel) produced yet another bizarre Windows Schannel PP
  fallout. There is probably more of these, especially on Windows, which is the
  hot mess it usually is. This can be addressed as they come along in real life
  or as interest / free time arises.

At this point I deem this mergable.

[eli-schwartz]

I remain generally skeptical of the values of unity builds, and maintain that if you want better optimization you should be building with LTO.

LTO: IME it makes builds slower and also fragile, especially with static builds (where any dependency issue will also be hit).

So LTO is the fragile thing, despite that this PR introduces "bizarre" fallout and generally has issues running tests? It seems less than ideal to be introducing a compile mode that isn't being thoroughly tested.

I'm curious what the apparent fragility of LTO is. I mean, yes, LTO tends to produce issues where existing UB gets miscompiled with LTO and has benevolent effects without LTO, but that's really no different in concept from building with higher -O settings, and the solution is to always fix the codebase in question to stop doing UB.

I'm also not sure what static builds or dependencies has to do with anything. No one suggested doing a unity build that gathers all dependencies into a single translation unit together with the curl sources -- by the same extension no one is under any obligation to use LTO across both curl and its static dependencies. Although the good news is that unlike unity builds (which will definitely not work with completely external project sources!) LTO is capable of doing that kind of optimization.

Be that as it may, I can happily report that curl builds fine with LTO on my machine (and passes all tests). And it doesn't even need a few thousand lines of changes that incur an ongoing maintenance cost which isn't revealed in the default build mode. :)

[thesamesam]

In addition to what you've said (and I wholeheartedly agree), unity builds in this form introduce variance across the build systems. And even with all of this, tests had to be skipped and weren't solved yet.

Meanwhile, many Linux distributions are building with LTO by default - including building curl, with no complaints.

[vszakats]

This PR doesn't disable the option to build curl with LTO. Nor is this against LTO, or in fact much if anything to do with LTO. It adds unity as a build option for those who might find it useful. It also unearthed some code smells.

[ On Windows (and MinGW), LTO makes build times considerably higher. When combined with static linking you're facing all the LTO bugs of all dependencies, which is sometimes more than one is willing (or able) to fix. LTO objects were (likely still are) compiler version specific with GCC, and had to deal with the fallout after each compiler update. It's a useful feature, but not necessarily for every use-case. ]

Bizarre fallouts are pretty common on Windows, unity builds and LTO aside.

[eli-schwartz]

I'm still confused by this "all the LTO bugs of all dependencies" thing, unless you just mean "C projects always have UB"?

I also don't think you really have to worry about compiler versions. Distros tend to support only a single version, and rebuild their entire stack as needed to ensure their packages are compatible (this is really a static libraries concern though), and outside of distros, people tend to stick to a single compiler version in their project anyway -- because of concerns about C++ compatibility for example.

...

While it's true that LTO can result in increased build times, I think it is generally meritorious. Someone who is seeking ways to increase runtime performance I would assume considers that to be a valuable priority, well worth spending a bit more time compiling releases. For optimizing the development cycle, one would simply turn off LTO because speed of testing loses out to speed of iterative testing... and for the same reason, one would turn off unity during development, because unity builds have a significant negative effect on your incremental build times. Even with a relatively low chunk size, the more you iterate the more you pay the cost for compiling multiple times the amount of code you modified.

So honestly, the motivation for adding support seems weak. And yes, I do understand that this doesn't prevent using LTO anyway. That was never my concern and if I accidentally gave that impression I apologize (although I'm not actually sure how I would have given that impression).

Again: my concerns are that

• it produces cases that are known to be "bizarre", and claiming that it's common to have bizarre issues on Windows seems a poor excuse for merging what is apparently unfinished work,
• it's acknowledged that the result of this is builds that can't be tested, and is thus apparently unfinished work, and
• unity builds in general mean that one is committing to an ongoing maintenance burden of keeping it working, adding additional testing matrixes and then dealing with the fallout of people who successfully tested their code contributions the normal way and then had it fail in CI with obscure errors, because it's now necessary to avoid perfectly reasonable and idiomatic, correct code, because it misbehaves when scoping rules are violated and different translation units see each others' headers and macros and trample all over private local symbols

Maybe I'd feel less weird about it if it was obvious to me that the curl maintainers in general feel it's worth the maintenance burden. That concern was raised above -- it was the first response -- and then not brought up again.

Maybe I'd feel less weird about it if it didn't include big disclaimers about how the PR is incomplete and on the single biggest platform, it has "bizarre" issues -- and those issues appear to be exactly an example of my concern that unity is in fact fragile, flaky and buggy. It gives the impression that curl is okay merging and officially supporting buggy configurations, including configurations that build with a passing return code but are suspect of incorrect runtime behavior.

And this PR is a lot of work to do for something that has such a glaring issue.

[vszakats]

I believe one main difference is that I tried making LTO work on Windows (in the distant past), where things were/are much bumpier than in a Linux env, whereas you seem to have done the same on Linux, where it's most probably just a linker switch away.

This PR is already done, out of my free time. As I said bizarre issues are business as usual on Windows, and all of them here were Windows related. You're welcome to pick up any of the issues and reflect to that, but skimming through weeks of work, picking out 2-3 uncomfortable words in comments, don't seem to be a productive take on this.

Anyhow, I suggest opening a discussion of the merits of LTO. I'm not personally interested in it, and this PR has nothing to do with it.

[AndrewJDR]

@vszakats Thanks for all the effort on this. I'm very sad that this is closed!

For me, what is truly exciting about a working unity / amalgamated libcurl is that it gets libcurl one step closer to this (from a twitter thread):

preconfigured and amalgamated source packages (e.g. "here's a single-source-libcurl preconfigured for Win32 and HTTP to drop right into your project")

@vszakats Do you think this sort of thing would be possible? Basically, if you're trying to ship a project that includes libcurl as source with minimal dependencies -- i.e. without requiring those adopting it to have cmake, etc), one could include a few pre-generated .c files, one for each platform that your project wants to support.

For example, someone supporting windows and linux might have these two files in in their project's repository:
deps/libcurl_win32_schannel.c
deps/libcurl_linux_mbedtls.c

Then, a simple build.bat and build.sh file could be used for the project, rather than burdening the developer with dealing with complex build systems like cmake. This would be a huge simplification for projects where libcurl is the only "complex" dependency that requires a build system.

* Obviously, for the deps/libcurl_linux_mbedtls.c example, mbedtls would need to be included in the repository as well, but mbedtls is actually fairly trivial to ship along with the repo without a build system. Plus that's outside the scope here, which is just getting libcurl to support unity builds.

[vszakats]

@AndrewJDR We always merge manually in curl, leaving PRs show up as "closed". I did merge this, so the feature is live. It's not as extensively tested by the CI as I'd want to though, and recently fixed (#11095) a regression in H2 builds.

What you say is technically possible after the efforts made in this PR, but curl for now leans on CMake to do the "amalgamation" part dynamically, as part of its build. Meaning we have no separate script that converts libcurl sources to a single .c file. CMake internally creates a wrapper .c file that #includes libcurl C sources, so it's not as easy to copy off an amalgamated .c from its build tree as-is either. But, certainly doable with some script foo. Haven't tried, but even a cat command might actually work.

[AndrewJDR]

@vszakats Wow that's great! I will definitely give it a try some time.