[Backport] Serialization framework updates

[furszy]

Decoupled from #2411, built on top of #2359.

Focused on creating the Span class and updating the serialization framework and every object using it up to latest upstream structure (3-4 years ahead of what we currently are in master). We will be up-to-date with them in the area after finishing with #2411 entirely (there are few more updates to the serialization code that comes down #2411 commits line that cannot cherry-pick here).

Adapted the following PRs:

• Introduce Span type and use it instead of FLATDATA bitcoin/bitcoin#12886.
• Introduce BigEndian wrapper and use it for netaddress ports bitcoin/bitcoin#12916.
• Drop unused GetType() from CSizeComputer bitcoin/bitcoin#13558.
• Serialization improvements (minimal initial commits) bitcoin/bitcoin#17850.
• Serialization improvements (step 2) bitcoin/bitcoin#17896.
• [MOVEONLY] Move compressor utility functions out of class bitcoin/bitcoin#12752.
• Serialization improvements step 3 (compression.h) bitcoin/bitcoin#17957.
• Serialization improvements step 4 (undo.h) bitcoin/bitcoin#18021.
• Get rid of VARINT default argument bitcoin/bitcoin#18087.
• Serialization improvements step 5 (blockencodings) bitcoin/bitcoin#18112 (only from 353f376 that we don't support).
• Fix a violation of C++ standard rules where unions are used for type-punning bitcoin/bitcoin#18167.
• Serialization improvements step 6 (all except wallet/gui) bitcoin/bitcoin#18317.
• Serialization improvements: final step bitcoin/bitcoin#19032.

[furszy]

rebased on master, ready to go.

[random-zebra]

This is a great job, really loved it.
Although I think that 068facd contains an error, which causes the warning for which f767d26 was introduced (which, in turn, fyi, results in a bunch of -Wunknown-pragmas warnings on GCC).

The issue here is a mis-use of SER_READ.
This macro is not meant to be used to actually read a value from the stream and save it to the variable passed as argument (like READWRITE). The second argument to SER_READ is a function, to be applied bypassing the const-ness of the object inside SERIALIZE_METHODS.

So, for example, we have something like:

std::vector<unsigned char> bytes;
READWRITE(bytes);
SER_READ(obj, obj.vBits = BytesToBits(bytes));

where, when ser_action.ForRead, READWRITE stores the vector contents inside the bytes local variable, and then SER_READ is used to update the object's member vBits.

When, instead, we do things like:

SER_READ(obj, nMoneySupply);

we are essentially passing this lambda to ::SerRead

[&](Stream& s, typename std::remove_const<Type>::type& obj) { nMoneySupply; }

which is a statement that has no effect on obj.
This is what the -Wunused-value is warning about (not because those variables are dummy containers inside SERIALIZE_METHODS).

And IMO this actually signals an error: as we are not reading anything with those SER_READ calls, we are not skipping the dummy fields, and so the serialization will fail with old DB versions.

random-zebra@3e20242

[furszy]

yeah great, nice catch ☕ . Updating.

[furszy]

Updated per feedback, cherry-picked your commit zebra and removed the warning suppress.

[furszy]

Rebased on master, migrating the recently merged ProUpServPL class to the new serialization form. Preventing a post-merge "fix missing" PR.

[random-zebra]

There is a change in the serialization of CBudgetVote, which makes it impossible to sync proposal votes from nodes that are not running this PR.

[furszy]

Updated per feedback, squashed change in 8a12b0c.
Fast diff check with git range-diff master 6cfd257 d361548.

[random-zebra]

Code review ACK d361548

Will run it live for some time.

[furszy]

Rebased on master, no code changes. So this branch does not reject blocks v10 on mainnet due the recently merged #2434 for v5.2 (requirement signaled by zebra that has been testing it ☕ ☕).

Can easily verify that nothing has changed with git range-diff master d361548 bd4b846d

[random-zebra]

ACK bd4b846

[Fuzzbawls]

ACK bd4b846