Split sources for easier buildsystem integration

[theuni]

This addresses issues like the one in bitcoin/bitcoin#12467, where some of our compiler flags end up being dropped during the subconfigure of Univalue. Specifically, we're still using the compiler-default c++ version rather than forcing c++17.

We can drop the need subconfigure completely in favor of a tighter build integration, where the sources are listed separately from the build recipes, so that they may be included directly by upstream projects. This is similar to the way leveldb build integration works in Core.

A similar split was recently added for minisketch: sipa/minisketch#8

Upstream benefits of this approach include:

• Better caching (for ex. ccache and autoconf)
• No need for a slow subconfigure
• No more missing compile flags
• Compile only the objects needed

There are no benefits to Univalue itself that I can think of. These changes should be a no-op here, and to downstreams as well until they take advantage of the new sources.mk. Once merged, This Core branch is ready-ish for PR, and takes advantage of the features added here.

libsecp256k1 will get the same treatment once this is done.

[fanquake]

Concept ACK

[theuni]

Rebased

[theuni]

Updated the Core implementation here as well: https://github.com/theuni/bitcoin/commits/univalue-split .

[hebasto]

Concept ACK.

What is the reason to keep the lib/univalue_escapes.h auto-generated header in the git repo after the first commit?

[theuni]

@hebasto That's just a convenience Makefile target to regenerate univalue_escapes.h. It's not automatically built in any case. There would be no need for a downstream project to run gen.

That commit is part of this PR because univalue_escapes.h was needlessly used in a recipe, so rather than exposing a variable for it, I eliminated its use entirely.

[jnewbery]

Tested ACK 4a5b0a1

One small question about FORCE inline.

[jnewbery]

Is this FORCE doing anything? As far as I understand, FORCE has no special meaning and by convention is used for a target that doesn't have any prerequisites or recipe, and is therefore always run: https://www.gnu.org/software/make/manual/html_node/Force-Targets.html#Force-Targets. I think the intention is to make sure that $(GENBIN) is always rebuilt, but I have some questions:

• doesn't there need to be a FORCE: target defined for this to work? The various examples I've seen have always included an empty FORCE: target.
• If the intention is to always rebuild $(GENBIN), then shouldn't FORCE be a dependency of $(GENBIN)? If it's a dependency of gen, then gen will get rereun, but it might be using an old version of $(GENBIN).
• Doesn't .INTERMEDIATE: $(GENBIN) mean that $(GENBIN) will be deleted at the end of the make anyway, and will therefore be rebuilt on each make gen?
• Doesn't the fact that gen is PHONY mean that it'll be run every make gen?

[theuni]

doesn't there need to be a FORCE: target defined for this to work?

FORCE is a phony variable here used to make gen always run. (But leaving it undeclared would work too, it would still just always run because it would never complete its goal of generating a file called FORCE)

If the intention is to always rebuild $(GENBIN), then shouldn't FORCE be a dependency of $(GENBIN)?

The intention is for $(GENBIN) to be a real target, and for gen to be a phony convenience target that always rebuilds the header.

Doesn't .INTERMEDIATE: $(GENBIN) mean that $(GENBIN) will be deleted at the end of the make anyway, and will therefore be rebuilt on each make gen?

Yup!

Doesn't the fact that gen is PHONY mean that it'll be run every make gen?

Yup!

The idea here is that "gen" is a convenience target for regenerating the univalue_escapes.h file. It is never run automatically, it is just a way to generate a header that's committed to the repo. As such, when a user does make gen, it should always regenerate the file. The current behavior of depending on its own output is broken.

There's no reason to ever keep $(GENBIN) around, so it's an intermediate. make gen essentially never needs to be used, it's possible that its only use is to demonstrate how the .h was generated.

[jnewbery]

Thanks @theuni! All makes sense.

[fanquake]

ACK 4a5b0a1