Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialization improvements #10785

Closed
wants to merge 26 commits into from
Closed

Serialization improvements #10785

wants to merge 26 commits into from

Conversation

@sipa
Copy link
Member

@sipa sipa commented Jul 10, 2017

This PR improves correctness (removing potentially unsafe const_casts) and flexibility of the serialization code.

The main issue is that use of the current ADD_SERIALIZE_METHODS macro (which is the only way to not duplicate serialization and deserialization code) only expands to a single class method, and thus can only be qualified as either const or non-const - not both. In many cases, serialization needs to work on const objects however, and preferably that is done without casts that could hide const-correctness bugs.

To deal with that, this PR introduces a new approach that includes a SERIALIZE_METHODS(obj) macro, where obj is a variable name. It expands to some boilerplate and a static method to which the object itself is an argument. The advantage is that its type can be templated, and be const when serializing.

Another issue is the various serialization-wrapping macros (VARINT, COMPACTSIZE, FLATDATA and LIMITED_STRING). They all const_cast their argument in order to construct a wrapper object, which supports both serialization and deserialization. This PR makes them templated in the underlying data type (for example, CompactSizeWrapper<uint64_t>). This has the advantage that we can make the template type const when invoked on a const variable (so it would be CompactSizeWrapper<const uint64_t> in that case).

A last issue is the persistent use of the REF macro to deal with temporary expressions being passed in. Since C++11, this is not needed anymore as temporaries are explicitly represented as rvalue references. Thus we can remove REF invocations and instead just make the various classes and helper functions deal correctly with references.

The above changes permit a fully const-correct version of all serialization code. However, it is cumbersome. Any existing ADD_SERIALIZE_METHODS instances in the code that do more than just (conditionally) serializing/deserializing some fields (in particular, it contains branches that assign to some of the variables) need to be split up into an explicit Serialize and Unserialize method instead. In some cases this is inevitable (wallet serializers do some crazy transformations while serializing!), but in many cases it is just annoying duplication.

To improve upon this, a few more primitives that are currently inlined are turned into serialization wrappers:

  • BigEndianWrapper: Serializes/deserializes an integer as big endian rather than little endian (only for 16-bit). This permits the CService serialization to become a oneliner.
  • Uint48Wrapper: Serializes/deserializes only the lower 48 bits of an integer (used in BIP152 code).
  • VectorApplyWrapper: Serializes/deserializes a vector while using a custom serializer for its elements. This simplifies the undo and blockencoding serializers a lot.

Best of all, it removes 147 lines of while code adding a bunch of comments (though the increased use of vararg READWRITE is probably cheating a bit).

The commits are ordered into 3 sections:

  • First, introduce new classes that permit const-correct serialization.
  • Then one by one transform the various files to use the new serializers.
  • Finally, remove the old serializers.

This may be too much to go in at once. I'm happy to split things up as needed.

@gmaxwell
Copy link
Contributor

@gmaxwell gmaxwell commented Jul 10, 2017

Should probably be tested on big endian. :)

@sipa sipa force-pushed the sipa:20170707_noncastserial branch 2 times, most recently Jul 10, 2017
Copy link
Member Author

@sipa sipa left a comment

Here is a self-review in which I point out some of the things reviewers may want to be aware of.

src/blockencodings.h Outdated Show resolved Hide resolved
src/test/serialize_tests.cpp Outdated
READWRITEMANY(intval, boolval, stringval, FLATDATA(charstrval), txval);
SERIALIZE_METHODS(obj)
{
READWRITE(obj.intval, obj.boolval, obj.stringval, FlatData(obj.charstrval), obj.txval);

This comment has been minimized.

@sipa

sipa Jul 10, 2017
Author Member

This whole test is somewhat less valuable now, as both cases use READWRITE.

src/wallet/wallet.h Outdated Show resolved Hide resolved
src/wallet/wallet.h Outdated Show resolved Hide resolved
src/serialize.h Outdated
* V is not required to be an std::vector type. It works for any class that
* exposes a value_type, iteration, and resize method that behave like vectors.
*/
template<template <typename> class W, typename V> class VectorApplyWrapper

This comment has been minimized.

@sipa

sipa Jul 10, 2017
Author Member

Notice the unusual construction of a template that takes a template as parameter here. See "Template template parameter" here: http://en.cppreference.com/w/cpp/language/template_parameters

src/serialize.h Outdated
{
::Serialize(s, std::forward<Arg>(arg));
::Serialize(s, arg);

This comment has been minimized.

@sipa

sipa Jul 10, 2017
Author Member

The reason for removing the std::forward calls here is explained in the commit message (there is no benefit in passing down the rvalue-ness).

@laanwj laanwj requested a review from jonasschnelli Jul 11, 2017
@jonasschnelli
Copy link
Member

@jonasschnelli jonasschnelli commented Jul 11, 2017

Concept ACK.
Binaries: https://bitcoin.jonasschnelli.ch/build/210 (Currently running on a fresh node)
Agree with @gmaxwell that some BE testing would be good.

Will code-review soon.

@sipa sipa force-pushed the sipa:20170707_noncastserial branch Jul 15, 2017
@sipa sipa force-pushed the sipa:20170707_noncastserial branch 2 times, most recently Jul 29, 2017
@sipa
Copy link
Member Author

@sipa sipa commented Jul 30, 2017

Made some changes to reduce the size of the overall diff.

Copy link
Member

@promag promag left a comment

ACK b0652ac. Everything works as expected.

Indeed a lot in one shot :).

src/serialize.h Outdated
}
};
//! Add a LimitedString wrapper around a const string (identity)
template<size_t I> static inline const std::string& LimitedString(const std::string& str) { return str; }
//! Add a LimitedString wrapper around a non-const string

This comment has been minimized.

@promag

promag Aug 6, 2017
Member

Missing periods in comments.

@sipa sipa force-pushed the sipa:20170707_noncastserial branch Aug 15, 2017
@sipa
Copy link
Member Author

@sipa sipa commented Aug 15, 2017

Rebased.

@laanwj laanwj added this to Blockers in High-priority for review Aug 17, 2017
@sipa sipa force-pushed the sipa:20170707_noncastserial branch Aug 25, 2017
@sipa
Copy link
Member Author

@sipa sipa commented Aug 25, 2017

Rebased.

Copy link
Contributor

@ryanofsky ryanofsky left a comment

Reviewed first several commits. Will continue reviewing.

public:
explicit BigEndianWrapper(I& val) : m_val(val)
{
static_assert(S == 2 || S == 4, "Unsupported BigEndian size");

This comment has been minimized.

@ryanofsky

ryanofsky Aug 31, 2017
Contributor

In commit "Add BigEndian serialization wrapper"

Should also static assert sizeof(I) <= S, and std::is_unsigned<I>::value

This comment has been minimized.

@sipa

sipa Sep 1, 2017
Author Member

Fixed.

* Use this wrapper around integer types that are stored in memory in native
* byte order, but serialized in big endian notation.
*
* Onlyy 16-bit types are supported for now.

This comment has been minimized.

@ryanofsky

ryanofsky Aug 31, 2017
Contributor

In commit "Add BigEndian serialization wrapper"

This seems to support 16 and 32 bit types, not just 16.

This comment has been minimized.

@sipa

sipa Sep 1, 2017
Author Member

Fixed.

@@ -493,9 +515,44 @@ class LimitedString
}
};

/** Serialization wrapper class for big-endian integers.
*
* Use this wrapper around integer types that are stored in memory in native

This comment has been minimized.

@ryanofsky

ryanofsky Aug 31, 2017
Contributor

In commit "Add BigEndian serialization wrapper"

Can you add a usage note here on when big endian numbers are actually recommended? Is this only for backwards compatibility with CService? It seems like a serialization format that uses a mix of big endian and little endian numbers would be confusing to work with.

This comment has been minimized.

@sipa

sipa Sep 1, 2017
Author Member

Done.

void Unserialize(Stream& s) {
n = ReadCompactSize<Stream>(s);
void Serialize(Stream& s) const {
WriteCompactSize<Stream>(s, n);

This comment has been minimized.

@ryanofsky

ryanofsky Aug 31, 2017
Contributor

Generalize CompactSize wrapper

Probably should add static assert to check std::is_unsigned<I>, or raise exception if n is less than 0.

Could also check against numeric_limits<int64_t>::max() at runtime or compile time.

(It also seems weird, though not relevant to this wrapper, that it is an error to read a compact int greater than MAX_SIZE but not to write one.)

This comment has been minimized.

@sipa

sipa Sep 1, 2017
Author Member

Fixed.

void Serialize(Stream &s) const {
WriteCompactSize<Stream>(s, n);
void Unserialize(Stream& s) {
n = ReadCompactSize<Stream>(s);

This comment has been minimized.

@ryanofsky

ryanofsky Aug 31, 2017
Contributor

In commit "Generalize CompactSize wrapper"

Ideally, this would throw an exception if return value is greater than numeric_limits<I>::max()

This comment has been minimized.

@sipa

sipa Sep 1, 2017
Author Member

Fixed.

//! Construct a FlatRange wrapper around a non-const POD and array types.
template<typename T> static inline FlatRangeWrapper<char> FlatDataInner(T* t, size_t len) { return FlatRangeWrapper<char>((char*)t, ((char*)t) + len); }
//! Helper macro to easily serialize POD types.
#define FLATDATA(x) FlatDataInner(&(x), sizeof(x))

This comment has been minimized.

@ryanofsky

ryanofsky Aug 31, 2017
Contributor

In commit "Generalize FlatData wrapper"

Though these changes don't make FLATDATA more dangerous than it was previously, the lack of type safety relying on C casts and sizeof here is a little scary. I experimented a little, and it seems this could be cleaned up with some simple changes I posted here: 6ef78bc. Could you take a look, and maybe incorporate these into the PR?

This comment has been minimized.

@sipa

sipa Sep 1, 2017
Author Member

I've incorporated part of your changes, but gone further and just added native support for serializing char arrays (without any wrapper). I think this is much cleaner now.

@sipa sipa force-pushed the sipa:20170707_noncastserial branch Sep 1, 2017
src/serialize.h Outdated
template<typename Stream>
void Unserialize(Stream& s)
{
if (S == 2) m_val = ser_readdata16be(s);

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Add BigEndian serialization wrapper"

Would be good to throw exception if deserialized value is greater than numeric_limits<I>::max(). Or alternately, change the sizeof(I) <= S requirement to sizeof(I) == S to prevent this being possible.

This comment has been minimized.

@sipa

sipa Sep 1, 2017
Author Member

Done.

src/serialize.h Outdated
n = ReadCompactSize<Stream>(s);
void Serialize(Stream& s) const
{
WriteCompactSize<Stream>(s, m_n);

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Generalize CompactSize wrapper"

Would be good to throw exception if m_n is greater than numeric_limits<uint64_t>::max(). Or alternately, require numeric_limits<I>::max() < numeric_limits<int64_t>::max() with static assert.

This comment has been minimized.

@sipa

sipa Sep 1, 2017
Author Member

Done.

src/netaddress.h Outdated
@@ -132,7 +132,7 @@ class CSubNet
inline void SerializationOp(Stream& s, Operation ser_action) {
READWRITE(network);
READWRITE(FLATDATA(netmask));
READWRITE(FLATDATA(valid));
READWRITE(valid);

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Overhaul FLATDATA"

Might be worth splitting this change out into separate commit, or noting in commit message here that that this change is not backwards compatible on platforms where sizeof(bool) is not 1.

This comment has been minimized.

@sipa

sipa Sep 1, 2017
Author Member

Done.

src/serialize.h Outdated
* Wrapper for serializing arrays and POD.
*/
class CFlatData
/** Wrapper for serializing arrays and POD. */

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Overhaul FLATDATA"

Probably more accurate to say "wrapper for serializing char arrays". (Though in principle this could work with stream classes with read/write methods not taking char pointers.)

This comment has been minimized.

@sipa

sipa Sep 1, 2017
Author Member

Fixed.

src/serialize.h Outdated
}
};
//! Construct a FlatRange wrapper around a const char vector.
template<typename T> static inline const FlatRangeWrapper<const char> FlatVector(const T& t) { return FlatRangeWrapper<const char>(CharCast(t.data()), CharCast(t.data() + t.size())); }

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Overhaul FLATDATA"

Maybe call it CharVector or CharArray instead of FlatVector. FlatVector is kind of redundant because any vectors should be flat. But also the vector part is limiting because these functions can work for other types (std::array, std::string, SecureString, std::basic_string_view, etc)

This comment has been minimized.

@sipa

sipa Sep 1, 2017
Author Member

Done.

@sipa sipa force-pushed the sipa:20170707_noncastserial branch Sep 1, 2017
Copy link
Contributor

@ryanofsky ryanofsky left a comment

More review comments. I made it up to the "Convert blockencodings to new serialization" commit this time.

* code. Adding "ADD_SERIALIZE_METHODS" in the body of the class causes these wrappers to be
* added as members.
*/
#define ADD_SERIALIZE_METHODS \

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Remove old serialization primitives"

There are still two references to ADD_SERIALIZE_METHODS in comments.

This comment has been minimized.

@sipa

sipa Sep 2, 2017
Author Member

Done.

src/serialize.h Outdated
* thus allows a single implementation that sees the object as const for serializing
* and non-const for deserializing, without casts.
*/
#define SERIALIZE_METHODS(obj) \

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Introduce new serialization macros without casts"

It would be nice if the SERIALIZE_METHODS macro took a class_name argument. I'd like this so it'd be possible to add deserializing constructors here (like CTransaction has), so there could be a uniform way to deserialize objects without assuming they support default construction. But also a class_name argument would make the macro more flexible and future proof, so it'd be easy to do things like:

  • Adding stricter type checking (e.g. an is_same<class_name, remove_const<Type>> assert in SerializationOps to prevent usage errors or template bloat.
  • Logging or debugging with #class_name
  • Adding static or friend functions that reference class_name.

This comment has been minimized.

@sipa

sipa Sep 2, 2017
Author Member

Done!

src/serialize.h Outdated
@@ -148,9 +148,21 @@ enum
SER_GETHASH = (1 << 2),
};

// Convert the reference base type to X, without changing constness or reference type.

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Add READWRITEAS, a macro to serialize safely as a different type"

Maybe use //!

This comment has been minimized.

@sipa

sipa Sep 2, 2017
Author Member

Done.

src/serialize.h Outdated
}
};
//! Automatically construct a CompactSize wrapper around the argument.
template<typename I> static inline CompactSizeWrapper<I> COMPACTSIZE(I& i) { return CompactSizeWrapper<I>(i); }

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Generalize CompactSize wrapper"

Might be good to add const I& i overload so it's possible to serialize rvalues.

This comment has been minimized.

@sipa

sipa Sep 2, 2017
Author Member

Done (and for other wrappers).

src/serialize.h Outdated
}
};
//! Automatically construct a VarInt wrapper around the argument.
template<typename I> static inline VarIntWrapper<typename std::remove_reference<I>::type> VARINT(I&& i) { return VarIntWrapper<typename std::remove_reference<I>::type>(i); }

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Generalize VarInt wrappers"

This might be easier to understand written with overloads instead of rvalue references:

template<typename I> static inline VarIntWrapper<I> VARINT(I& i) { return VarIntWrapper<I>(i); }
template<typename I> static inline VarIntWrapper<const I> VARINT(const I& i) { return VarIntWrapper<const I>(i); }

Also would make it more consistent with other wrappers.

This comment has been minimized.

@sipa

sipa Sep 2, 2017
Author Member

The problem is that VARINT is called with temporaries as arguments, which is not true for the other ones. Either it's written as 4 cases, or using std::remove_refence.

EDIT: Oh, you're right. An lvalue reference parameter binds to rvalue reference argument, so all good.

This comment has been minimized.

@sipa

sipa Sep 2, 2017
Author Member

Done.

src/serialize.h Outdated
s.read((char*)string.data(), size);
m_string.resize(size);
if (size != 0) {
s.read((char*)m_string.data(), size);

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Generalize LimitedString wrapper"

Maybe use &m_string[0] avoid writing to a const pointer.

This comment has been minimized.

@sipa

sipa Sep 2, 2017
Author Member

Done.

src/serialize.h Outdated
unsigned int nSize = ReadCompactSize(s);
unsigned int nMid = 0;
while (nMid < nSize) {
nMid += 5000000 / sizeof(value_type);

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Add custom vector-element serialization wrapper"

This could use a comment. I don't understand it at all. Wouldn't it be simpler and more efficient to just resize and fill the vector once instead of resizing it multiple times?

This comment has been minimized.

@sipa

sipa Sep 2, 2017
Author Member

Done. The code was also totally broken, so I've rewritten it.

src/serialize.h Outdated
}
m_vector.resize(nMid);
for (value_type& x : m_vector) {
s >> W<value_type>(x);

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Add custom vector-element serialization wrapper"

Doesn't this overwrite elements in the front of the vector each time through the while loop?

This comment has been minimized.

@sipa

sipa Sep 2, 2017
Author Member

Yes, this was bogus. Thanks for pointing that out; fixed by rewriting,

src/blockencodings.h Outdated
}
};

template<typename T>

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Convert blockencodings to new serialization"

Maybe move this up closer to TransactionCompressWrapper class

This comment has been minimized.

@sipa

sipa Sep 2, 2017
Author Member

Done.

{
static_assert(SHORTTXIDS_LENGTH == 6, "shorttxids serialization assumes 6-byte shorttxids");
s >> header >> nonce >> VectorApply<Uint48Wrapper>(shorttxids) >> prefilledtxn;
FillShortTxIDSelector();

This comment has been minimized.

@ryanofsky

ryanofsky Sep 1, 2017
Contributor

In commit "Convert blockencodings to new serialization"

Would you be opposed to adding mutable object access to SERIALIZE_METHODS so Serialize and Unserialize methods don't need to be split up? I could think of a number of ways to do this. Maybe easiest would be to stick a mutable pointer inside ser_action:

SERIALIZE_METHODS(obj) {
  READWRITE(obj.header, obj.nonce, VectorApply<Uint48Wrapper>(obj.shorttxids), obj.prefilledtxn);
  if (ser_action.ForRead()) ser_action.MutableObj()->FillShortTxId();
}

The pointer would be null when serializing.

This comment has been minimized.

@sipa

sipa Sep 3, 2017
Author Member

That's a neat trick. It seems a bit ugly to need to fake returning a mutable object nullptr. I have an alternative, but I'm not sure it's any better.:

template <typename T, typename F>
void IfUnserializer(T& obj, CSerActionUnserialize ser_action, const F& fn) { fn(obj); }
template <typename T, typename F>
void IfUnserialize(const T& obj, CSerActionSerialize ser_action, const F& fn) {}
#define IF_UNSERIALIZE(typ, obj, code) (::IfUnserialize<typ>(obj, ser_action, [&](typ& obj)code))

Which you'd then invoke using

     SERIALIZE_METHODS(SomeType, obj)
     {
          READWRITE(obj.member);
          IF_UNSERIALIZE(SomeType, obj, {obj.FillShortTxid();});
     }

This comment has been minimized.

@ryanofsky

ryanofsky Sep 6, 2017
Contributor

Thread #10785 (comment)

That looks good to me. I was actually going to suggest this same approach before I noticed there was a ser_action.ForRead method. ser_action could also have a method returning a reference instead of a pointer. I think any approach that would avoid duplicating serialization & deserialization would be good, though.

For IF_UNSERIALIZE, maybe consider getting of all the obj macro arguments:

     SERIALIZE_METHODS(SomeType)
     {
          READWRITE(obj.member);
          IF_UNSERIALIZE({obj.FillShortTxid();});
     }

I think the obj arguments maybe help make serialize methods resemble normal methods, but don't actually add real utility.

This comment has been minimized.

@sipa

sipa Sep 6, 2017
Author Member

@ryanofsky I don't see how to get rid of passing in the type to IF_UNSERIALIZE.

This comment has been minimized.

@ryanofsky

ryanofsky Sep 6, 2017
Contributor

@ryanofsky I don't see how to get rid of passing in the type to IF_UNSERIALIZE.

You could pass it as a template parameter to CSerActionSerialize / CSerActionUnserialize and access it through ser_action.

@sipa sipa force-pushed the sipa:20170707_noncastserial branch 2 times, most recently Sep 2, 2017
@sipa sipa removed this from Blockers in High-priority for review Sep 5, 2017
Copy link
Contributor

@ryanofsky ryanofsky left a comment

utACK b49e84ea6dd7e2c0aacb8546c8d8c2d99d1d8214. Finally reviewed the full change.

src/serialize.h Outdated
{
pbegin = (char*)v.data();
pend = (char*)(v.data() + v.size());
static_assert(sizeof(C) == 1, "CharArrayWrapper only works for char types");

This comment has been minimized.

@ryanofsky

ryanofsky Sep 6, 2017
Contributor

I'm not sure that wchar_t has a well-defined in-memory representation.

Since this class is no longer casting any pointers, I don't think that's a problem. Existing c++ type checking will make sure pointers passed to stream read & write methods are compatible, so I don't think there is a reason for this class to be interjecting and adding extra type requirements.

src/serialize.h Outdated Show resolved Hide resolved
src/blockencodings.h Outdated Show resolved Hide resolved
src/blockencodings.h Outdated Show resolved Hide resolved
{
static_assert(SHORTTXIDS_LENGTH == 6, "shorttxids serialization assumes 6-byte shorttxids");
s >> header >> nonce >> VectorApply<Uint48Wrapper>(shorttxids) >> prefilledtxn;
FillShortTxIDSelector();

This comment has been minimized.

@ryanofsky

ryanofsky Sep 6, 2017
Contributor

Thread #10785 (comment)

That looks good to me. I was actually going to suggest this same approach before I noticed there was a ser_action.ForRead method. ser_action could also have a method returning a reference instead of a pointer. I think any approach that would avoid duplicating serialization & deserialization would be good, though.

For IF_UNSERIALIZE, maybe consider getting of all the obj macro arguments:

     SERIALIZE_METHODS(SomeType)
     {
          READWRITE(obj.member);
          IF_UNSERIALIZE({obj.FillShortTxid();});
     }

I think the obj arguments maybe help make serialize methods resemble normal methods, but don't actually add real utility.

// TODO: avoid reimplementing vector deserializer
uint64_t count = 0;
::Unserialize(s, COMPACTSIZE(count));
if (count > MAX_INPUTS_PER_BLOCK) {

This comment has been minimized.

@ryanofsky

ryanofsky Sep 6, 2017
Contributor

Maybe mention in commit message if behavior is changing here. I guess the limit is higher now.

This comment has been minimized.

@sipa

sipa Sep 7, 2017
Author Member

It could use slightly more memory when deserializing an otherwise invalid object, but that shouldn't change behaviour otherwise - if the number of transaction undo objects doesn't match the number of transaction in the block, it's invalid anyway.

src/qt/recentrequeststablemodel.h Outdated
{
unsigned int nDate;
s >> this->nVersion >> id >> nDate >> recipient;
date = QDateTime::fromTime_t(nDate);

This comment has been minimized.

@ryanofsky

ryanofsky Sep 6, 2017
Contributor

In commit "Convert Qt to new serialization"

I guess this is another place that could use IF_UNSERIALIZE if you decide to go this route.

This comment has been minimized.

@sipa

sipa Sep 7, 2017
Author Member

Introduced a wrapper for QDateTime instead.

src/qt/walletmodel.h Outdated
}
authenticatedMerchant = QString::fromStdString(sAuthenticatedMerchant);

This comment has been minimized.

@ryanofsky

ryanofsky Sep 6, 2017
Contributor

In commit "Convert Qt to new serialization"

It seems like if there were serialization wrappers from QString and proto types, the serialize and deserialize methods could be combined again.

This comment has been minimized.

@sipa

sipa Sep 7, 2017
Author Member

Done using wrappers for QString and proto.

src/wallet/wallet.h Outdated
s >> nTime >> vchPubKey;
try {
s >> fInternal;
} catch (std::ios_base::failure&) {

This comment has been minimized.

@ryanofsky

ryanofsky Sep 6, 2017
Contributor

In commit "Convert wallet/walletdb/crypter to new serialization

Maybe another place to use IF_UNSERIALIZE. Or maybe there could be a wrapper that ignores ios_base errors on deserialization.

src/wallet/wallet.h Outdated Show resolved Hide resolved
@sipa sipa force-pushed the sipa:20170707_noncastserial branch Sep 6, 2017
Copy link
Contributor

@ryanofsky ryanofsky left a comment

utACK a9770fbe6ef6d7d9bd5b61e4c51531af7cc2a1fc. Still looks good. Only minor changes since previous review, described in comments above.

@laanwj
Copy link
Member

@laanwj laanwj commented Sep 6, 2017

Big concept ACK, happy to get rid of FLATDATA and similar ugly macros. This is a lot to review/test though, and reasonably high-risk.

sidhujag added a commit to syscoin/syscoin that referenced this pull request Mar 5, 2020
353f376 Convert blockencodings.h to new serialization framework (Pieter Wuille)
e574fff Add CustomUintFormatter (Pieter Wuille)
1063339 Add DifferenceFormatter (Russell Yanofsky)
56dd9f0 Make VectorFormatter support stateful formatters (Russell Yanofsky)
3ca574c Convert CCompactSize to proper formatter (Pieter Wuille)

Pull request description:

  This is probably the most involved change in the sequence of changes extracted from bitcoin#10785.

  In order to implement the differential encoding of BIP152, this change changes `VectorFormatter` to permit a stateful sub-formatter, which is then used by `DifferenceFormatter`. A `CustomUintFormatter` is added as well to do the 48-bit serialization of short ids.

ACKs for top commit:
  laanwj:
    ACK 353f376, nice change
  ryanofsky:
    Code review ACK 353f376. Only changes since last review are suggested assert change and MASK->MAX rename

Tree-SHA512: 976618991a8be62ba0738725b7cfa166a56cde998ebf1031ba6f28557032f1577b666ac7ae25cd498c0e1e740108c3c56a342620b724df41d6cc9d8bdafac037
MarcoFalke added a commit that referenced this pull request May 20, 2020
f9ee0f3 Add comments to CustomUintFormatter (Pieter Wuille)
4eb5643 Convert everything except wallet/qt to new serialization (Pieter Wuille)
2b1f85e Convert blockencodings_tests to new serialization (Pieter Wuille)
73747af Convert merkleblock to new serialization (Pieter Wuille)
d06fedd Add SER_READ and SER_WRITE for read/write-dependent statements (Russell Yanofsky)
6f9a1e5 Extend CustomUintFormatter to support enums (Russell Yanofsky)
769ee5f Merge BigEndian functionality into CustomUintFormatter (Pieter Wuille)

Pull request description:

  The next step of changes from #10785.

  This:
  * Adds support for enum serialization to `CustomUintFormatter`, used in `CAddress` for service flags.
  * Merges `BigEndian` into `CustomUintFormatter`, used in `CNetAddr` for port numbers.
  * Converts everything (except wallet and gui) to use the new serialization framework.

ACKs for top commit:
  MarcoFalke:
    re-ACK f9ee0f3, only change is new documentation commit for CustomUintFormatter 📂
  ryanofsky:
    Code review ACK f9ee0f3. Just new commit adding comment since last review
  jonatack:
    Code review re-ACK f9ee0f3 only change since last review is an additional commit adding Doxygen documentation for `CustomUintFormatter`.

Tree-SHA512: e7a0a36afae592d5a4ff8c81ae04d858ac409388e361f2bc197d9a78abca45134218497ab2dfd6d031e0cce0ca586cf857077b7c6ce17fccf67e2d367c1b6cd4
sidhujag added a commit to syscoin/syscoin that referenced this pull request May 20, 2020
…llet/gui)

f9ee0f3 Add comments to CustomUintFormatter (Pieter Wuille)
4eb5643 Convert everything except wallet/qt to new serialization (Pieter Wuille)
2b1f85e Convert blockencodings_tests to new serialization (Pieter Wuille)
73747af Convert merkleblock to new serialization (Pieter Wuille)
d06fedd Add SER_READ and SER_WRITE for read/write-dependent statements (Russell Yanofsky)
6f9a1e5 Extend CustomUintFormatter to support enums (Russell Yanofsky)
769ee5f Merge BigEndian functionality into CustomUintFormatter (Pieter Wuille)

Pull request description:

  The next step of changes from bitcoin#10785.

  This:
  * Adds support for enum serialization to `CustomUintFormatter`, used in `CAddress` for service flags.
  * Merges `BigEndian` into `CustomUintFormatter`, used in `CNetAddr` for port numbers.
  * Converts everything (except wallet and gui) to use the new serialization framework.

ACKs for top commit:
  MarcoFalke:
    re-ACK f9ee0f3, only change is new documentation commit for CustomUintFormatter 📂
  ryanofsky:
    Code review ACK f9ee0f3. Just new commit adding comment since last review
  jonatack:
    Code review re-ACK f9ee0f3 only change since last review is an additional commit adding Doxygen documentation for `CustomUintFormatter`.

Tree-SHA512: e7a0a36afae592d5a4ff8c81ae04d858ac409388e361f2bc197d9a78abca45134218497ab2dfd6d031e0cce0ca586cf857077b7c6ce17fccf67e2d367c1b6cd4
laanwj added a commit that referenced this pull request May 26, 2020
71f016c Remove old serialization primitives (Pieter Wuille)
92beff1 Convert LimitedString to formatter (Pieter Wuille)
ef17c03 Convert wallet to new serialization (Pieter Wuille)
65c589e Convert Qt to new serialization (Pieter Wuille)

Pull request description:

  This is the final step 🥳 of the serialization improvements extracted from #10785.

  It converts the LimitedString wrapper to a new-style formatter, and updates the wallet and Qt code to use the new serialization framework. Finally all remaining old primitives are removed.

ACKs for top commit:
  jonatack:
    ACK 71f016c reviewed diff, builds/tests/re-fuzzed.
  laanwj:
    Code review ACK 71f016c

Tree-SHA512: d952194bc73259f6510bd4ab1348a1febbbf9862af30f905991812fb0e1f23f15948cdb3fc662be54d648e8f6d95b11060055d2e7a8c2cb5bf008224870b1ea1
sidhujag added a commit to syscoin/syscoin that referenced this pull request May 27, 2020
71f016c Remove old serialization primitives (Pieter Wuille)
92beff1 Convert LimitedString to formatter (Pieter Wuille)
ef17c03 Convert wallet to new serialization (Pieter Wuille)
65c589e Convert Qt to new serialization (Pieter Wuille)

Pull request description:

  This is the final step 🥳 of the serialization improvements extracted from bitcoin#10785.

  It converts the LimitedString wrapper to a new-style formatter, and updates the wallet and Qt code to use the new serialization framework. Finally all remaining old primitives are removed.

ACKs for top commit:
  jonatack:
    ACK 71f016c reviewed diff, builds/tests/re-fuzzed.
  laanwj:
    Code review ACK 71f016c

Tree-SHA512: d952194bc73259f6510bd4ab1348a1febbbf9862af30f905991812fb0e1f23f15948cdb3fc662be54d648e8f6d95b11060055d2e7a8c2cb5bf008224870b1ea1
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this pull request Jun 9, 2020
42343c7 Split up and sanitize CAccountingEntry serialization (Pieter Wuille)
029ecac Split up and sanitize CWalletTx serialization (Pieter Wuille)

Pull request description:

  This is a small subset of changes taken from bitcoin#10785, fixing a few of the craziest constness violations in the serialization code.

  `CWalletTx` currently serializes some of its fields by embedding them in a key-value `mapValue`, which is modified (and then fixed up) even from the `Serialize` method (for which `mapValue` is const). `CAccountingEntry` goes even further in that it stores such a map by appending it into `strComment` after a null char, which is again later fixed up again.

  Fix this by splitting the serialization and deserialization code, and making the serialization act on a copy of `mapValue` / `strComment`.

Tree-SHA512: 487e04996dea6aba5b9b8bdaf2c4e680808f111a15afc557b8d078e14b01e4f40f8ef27588869be62f9a87052117c17e0a0c26c59150f83472a9076936af035e
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this pull request Jun 9, 2020
42343c7 Split up and sanitize CAccountingEntry serialization (Pieter Wuille)
029ecac Split up and sanitize CWalletTx serialization (Pieter Wuille)

Pull request description:

  This is a small subset of changes taken from bitcoin#10785, fixing a few of the craziest constness violations in the serialization code.

  `CWalletTx` currently serializes some of its fields by embedding them in a key-value `mapValue`, which is modified (and then fixed up) even from the `Serialize` method (for which `mapValue` is const). `CAccountingEntry` goes even further in that it stores such a map by appending it into `strComment` after a null char, which is again later fixed up again.

  Fix this by splitting the serialization and deserialization code, and making the serialization act on a copy of `mapValue` / `strComment`.

Tree-SHA512: 487e04996dea6aba5b9b8bdaf2c4e680808f111a15afc557b8d078e14b01e4f40f8ef27588869be62f9a87052117c17e0a0c26c59150f83472a9076936af035e
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this pull request Jun 10, 2020
42343c7 Split up and sanitize CAccountingEntry serialization (Pieter Wuille)
029ecac Split up and sanitize CWalletTx serialization (Pieter Wuille)

Pull request description:

  This is a small subset of changes taken from bitcoin#10785, fixing a few of the craziest constness violations in the serialization code.

  `CWalletTx` currently serializes some of its fields by embedding them in a key-value `mapValue`, which is modified (and then fixed up) even from the `Serialize` method (for which `mapValue` is const). `CAccountingEntry` goes even further in that it stores such a map by appending it into `strComment` after a null char, which is again later fixed up again.

  Fix this by splitting the serialization and deserialization code, and making the serialization act on a copy of `mapValue` / `strComment`.

Tree-SHA512: 487e04996dea6aba5b9b8bdaf2c4e680808f111a15afc557b8d078e14b01e4f40f8ef27588869be62f9a87052117c17e0a0c26c59150f83472a9076936af035e
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this pull request Jun 11, 2020
42343c7 Split up and sanitize CAccountingEntry serialization (Pieter Wuille)
029ecac Split up and sanitize CWalletTx serialization (Pieter Wuille)

Pull request description:

  This is a small subset of changes taken from bitcoin#10785, fixing a few of the craziest constness violations in the serialization code.

  `CWalletTx` currently serializes some of its fields by embedding them in a key-value `mapValue`, which is modified (and then fixed up) even from the `Serialize` method (for which `mapValue` is const). `CAccountingEntry` goes even further in that it stores such a map by appending it into `strComment` after a null char, which is again later fixed up again.

  Fix this by splitting the serialization and deserialization code, and making the serialization act on a copy of `mapValue` / `strComment`.

Tree-SHA512: 487e04996dea6aba5b9b8bdaf2c4e680808f111a15afc557b8d078e14b01e4f40f8ef27588869be62f9a87052117c17e0a0c26c59150f83472a9076936af035e
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this pull request Jun 11, 2020
42343c7 Split up and sanitize CAccountingEntry serialization (Pieter Wuille)
029ecac Split up and sanitize CWalletTx serialization (Pieter Wuille)

Pull request description:

  This is a small subset of changes taken from bitcoin#10785, fixing a few of the craziest constness violations in the serialization code.

  `CWalletTx` currently serializes some of its fields by embedding them in a key-value `mapValue`, which is modified (and then fixed up) even from the `Serialize` method (for which `mapValue` is const). `CAccountingEntry` goes even further in that it stores such a map by appending it into `strComment` after a null char, which is again later fixed up again.

  Fix this by splitting the serialization and deserialization code, and making the serialization act on a copy of `mapValue` / `strComment`.

Tree-SHA512: 487e04996dea6aba5b9b8bdaf2c4e680808f111a15afc557b8d078e14b01e4f40f8ef27588869be62f9a87052117c17e0a0c26c59150f83472a9076936af035e
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this pull request Jun 11, 2020
42343c7 Split up and sanitize CAccountingEntry serialization (Pieter Wuille)
029ecac Split up and sanitize CWalletTx serialization (Pieter Wuille)

Pull request description:

  This is a small subset of changes taken from bitcoin#10785, fixing a few of the craziest constness violations in the serialization code.

  `CWalletTx` currently serializes some of its fields by embedding them in a key-value `mapValue`, which is modified (and then fixed up) even from the `Serialize` method (for which `mapValue` is const). `CAccountingEntry` goes even further in that it stores such a map by appending it into `strComment` after a null char, which is again later fixed up again.

  Fix this by splitting the serialization and deserialization code, and making the serialization act on a copy of `mapValue` / `strComment`.

Tree-SHA512: 487e04996dea6aba5b9b8bdaf2c4e680808f111a15afc557b8d078e14b01e4f40f8ef27588869be62f9a87052117c17e0a0c26c59150f83472a9076936af035e
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this pull request Jun 12, 2020
42343c7 Split up and sanitize CAccountingEntry serialization (Pieter Wuille)
029ecac Split up and sanitize CWalletTx serialization (Pieter Wuille)

Pull request description:

  This is a small subset of changes taken from bitcoin#10785, fixing a few of the craziest constness violations in the serialization code.

  `CWalletTx` currently serializes some of its fields by embedding them in a key-value `mapValue`, which is modified (and then fixed up) even from the `Serialize` method (for which `mapValue` is const). `CAccountingEntry` goes even further in that it stores such a map by appending it into `strComment` after a null char, which is again later fixed up again.

  Fix this by splitting the serialization and deserialization code, and making the serialization act on a copy of `mapValue` / `strComment`.

Tree-SHA512: 487e04996dea6aba5b9b8bdaf2c4e680808f111a15afc557b8d078e14b01e4f40f8ef27588869be62f9a87052117c17e0a0c26c59150f83472a9076936af035e
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this pull request Sep 27, 2020
…ithout FLATDATA

a7c45bc Add native support for serializing char arrays without FLATDATA (Pieter Wuille)

Pull request description:

  Support is added to serialize arrays of type `char` or `unsigned char` directly, without any wrappers. All invocations of the `FLATDATA` wrappers that are obsoleted by this are removed.

  This includes a patch by @ryanofsky to make `char` casting type safe.

  The serialization of `CSubNet` is changed to serialize a `bool` directly rather than though `FLATDATA`. This makes the serialization independent of the size of the bool type (and will use 1 byte everywhere).

  This is a small change taken from bitcoin#10785.

Tree-SHA512: a41f61ca5fdc2fadb2d0e1702351a58a23841d551f505292a9542602cdb19f90d8944b8df14b872810a56bd201648fa4c0e958f3e9427fe829886284e85b9bfd
Signed-off-by: pasta <pasta@dashboost.org>

# Conflicts:
#	src/test/serialize_tests.cpp
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this pull request Oct 22, 2020
…ithout FLATDATA

a7c45bc Add native support for serializing char arrays without FLATDATA (Pieter Wuille)

Pull request description:

  Support is added to serialize arrays of type `char` or `unsigned char` directly, without any wrappers. All invocations of the `FLATDATA` wrappers that are obsoleted by this are removed.

  This includes a patch by @ryanofsky to make `char` casting type safe.

  The serialization of `CSubNet` is changed to serialize a `bool` directly rather than though `FLATDATA`. This makes the serialization independent of the size of the bool type (and will use 1 byte everywhere).

  This is a small change taken from bitcoin#10785.

Tree-SHA512: a41f61ca5fdc2fadb2d0e1702351a58a23841d551f505292a9542602cdb19f90d8944b8df14b872810a56bd201648fa4c0e958f3e9427fe829886284e85b9bfd
Signed-off-by: pasta <pasta@dashboost.org>

# Conflicts:
#	src/test/serialize_tests.cpp
sidhujag added a commit to syscoin-core/syscoin that referenced this pull request Nov 10, 2020
…its)

9250a08 Convert addrdb/addrman to new serialization (Pieter Wuille)
ca33451 Introduce new serialization macros without casts (Pieter Wuille)

Pull request description:

  This is a minimal subset of bitcoin#10785 that still does *something*.

  It adds a new saner serialization macro, which can be used in parallel with the old one. Then the addrdb code is converted to use this new macro.

  I'll add follow-up PRs that add more functionality + converting of other modules as things get merged.

ACKs for top commit:
  jamesob:
    ACK 9250a08 ([`jamesob/ackr/17850.1.sipa.serialization_improvemen`](https://github.com/jamesob/bitcoin/tree/ackr/17850.1.sipa.serialization_improvemen))
  kallewoof:
    ACK 9250a08
  laanwj:
    code review ACK 9250a08

Tree-SHA512: d4f58c7f85d8ada7543ee43159be57d320746abe003af11395508d280d339fac7faa198e707d1a689fb0a775fc36b3945178c3ae1c0cf9ffe685773c6ddc10c1
sidhujag added a commit to syscoin-core/syscoin that referenced this pull request Nov 10, 2020
9b66083 Convert chain to new serialization (Pieter Wuille)
2f1b2f4 Convert VARINT to the formatter/Using approach (Pieter Wuille)
ca62563 Add a generic approach for (de)serialization of objects using code in other classes (Pieter Wuille)

Pull request description:

  This is a second carve-out from bitcoin#10785.

  This introduces a const-correct generic approach for serializing objects using custom serializers (defined separately from the object being serialized), then converts VARINT to use that approach, and then converts chain.h to the new framework (including the new const-correct VARINT macro).

ACKs for top commit:
  jamesob:
    ACK 9b66083 ([`jamesob/ackr/17896.1.sipa.serialization_improvemen`](https://github.com/jamesob/bitcoin/tree/ackr/17896.1.sipa.serialization_improvemen))
  ryanofsky:
    Code review ACK 9b66083. Only change since last review is suggested lvalue reference tweak

Tree-SHA512: 2da4af1754699cb223d6beae44c587555e39ef6951448488a04783c92e2dfd4a305934f71cc3a75d06faf6d722723d8cdbd5ccb12039783f8d62039b83987bb8
sidhujag added a commit to syscoin-core/syscoin that referenced this pull request Nov 10, 2020
4de934b Convert compression.h to new serialization framework (Pieter Wuille)
ca34c5c Add FORMATTER_METHODS, similar to SERIALIZE_METHODS, but for formatters (Pieter Wuille)

Pull request description:

  This is the next piece of the puzzle from bitcoin#10785. It includes:
  * The `FORMATTER_METHODS` macro, similar to `SERIALIZE_METHODS`, for defining a formatter with a unified serialization/deserialization implementation.
  * Updating `compression.h` to consist of 3 formatters, rather than old-style wrappers (`ScriptCompression`, `AmountCompression`, `TxOutCompression`).

ACKs for top commit:
  laanwj:
    code review ACK 4de934b
  ryanofsky:
    Code review ACK 4de934b. Only change since last review is removing REF usages

Tree-SHA512: d52ca21eb1ce87d9bc3c90d00c905bd4fada522759aaa144c02a58b4d738d5e8647c0558b8ce393c707f6e3c4d20bf93781a2dcc1e1dcbd276d9b5ffd0e02cd6
sidhujag added a commit to syscoin-core/syscoin that referenced this pull request Nov 10, 2020
3c94b00 Convert undo.h to new serialization framework (Pieter Wuille)
3cd8ab9 Make std::vector and prevector reuse the VectorFormatter logic (Pieter Wuille)
abf8624 Add custom vector-element formatter (Pieter Wuille)
37d800b Add a constant for the maximum vector allocation (5 Mbyte) (Pieter Wuille)

Pull request description:

  The next step of changes from bitcoin#10785.

  This one adds:
  * A meta-formatter for vectors, which serializes the vector elements using another formatter
  * Switch the undo.h code to the new framework, using the above (where undo entries are serialized as a vector, each of which uses a modified serializer for the UTXOs).

ACKs for top commit:
  laanwj:
    code review ACK 3c94b00
  jonatack:
    Qualified ACK 3c94b00
  ryanofsky:
    Code review ACK 3c94b00. Changes since last review: renaming formatter classes, adding suggested static_assert, and removing temporary in VectorFormatter

Tree-SHA512: 44eebf51a303f6adbbc1ca2b9d043e8ae7fd37e06778e026590892f8d09f8253067862a68ba8ca5d733fd2f8e7c84edd255370f5a4b6560259427a65f94632df
sidhujag added a commit to syscoin-core/syscoin that referenced this pull request Nov 10, 2020
353f376 Convert blockencodings.h to new serialization framework (Pieter Wuille)
e574fff Add CustomUintFormatter (Pieter Wuille)
1063339 Add DifferenceFormatter (Russell Yanofsky)
56dd9f0 Make VectorFormatter support stateful formatters (Russell Yanofsky)
3ca574c Convert CCompactSize to proper formatter (Pieter Wuille)

Pull request description:

  This is probably the most involved change in the sequence of changes extracted from bitcoin#10785.

  In order to implement the differential encoding of BIP152, this change changes `VectorFormatter` to permit a stateful sub-formatter, which is then used by `DifferenceFormatter`. A `CustomUintFormatter` is added as well to do the 48-bit serialization of short ids.

ACKs for top commit:
  laanwj:
    ACK 353f376, nice change
  ryanofsky:
    Code review ACK 353f376. Only changes since last review are suggested assert change and MASK->MAX rename

Tree-SHA512: 976618991a8be62ba0738725b7cfa166a56cde998ebf1031ba6f28557032f1577b666ac7ae25cd498c0e1e740108c3c56a342620b724df41d6cc9d8bdafac037
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this pull request Dec 16, 2020
172f5fa Support deserializing into temporaries (Pieter Wuille)
2761bca Merge READWRITEMANY into READWRITE (Pieter Wuille)

Pull request description:

  This is another fragment of improvements from bitcoin#10785.

  The current serialization code does not support serializing/deserializing from/to temporaries (like `s >> CFlatData(script)`). As a result, there are many invocations of the `REF` macro which in addition to changing the reference type also changes the constness. This is unnecessary in C++11 as we can use rvalue references now instead.

  The first commit is an extra simplification we can make that removes the duplication of code between `READWRITE` and `READWRITEMANY` (and related functions).

Tree-SHA512: babfa9cb268cc3bc39917e4f0a90e4651c33d85032161e16547a07f3b257b7ca7940e0cbfd69f09439d26fafbb1a6cf6359101043407e2c7aeececf7f20b6eed
UdjinM6 added a commit to UdjinM6/dash that referenced this pull request Dec 17, 2020
172f5fa Support deserializing into temporaries (Pieter Wuille)
2761bca Merge READWRITEMANY into READWRITE (Pieter Wuille)

Pull request description:

  This is another fragment of improvements from bitcoin#10785.

  The current serialization code does not support serializing/deserializing from/to temporaries (like `s >> CFlatData(script)`). As a result, there are many invocations of the `REF` macro which in addition to changing the reference type also changes the constness. This is unnecessary in C++11 as we can use rvalue references now instead.

  The first commit is an extra simplification we can make that removes the duplication of code between `READWRITE` and `READWRITEMANY` (and related functions).

Tree-SHA512: babfa9cb268cc3bc39917e4f0a90e4651c33d85032161e16547a07f3b257b7ca7940e0cbfd69f09439d26fafbb1a6cf6359101043407e2c7aeececf7f20b6eed
UdjinM6 added a commit to UdjinM6/dash that referenced this pull request Dec 17, 2020
…dress ports

ece88fd Introduce BigEndian wrapper and use it for netaddress ports (Pieter Wuille)

Pull request description:

  This is another small improvement taken from bitcoin#10785.

  Instead of manually converting from/to BE format in the `CService` serializer, provide a generic way in serialize.h to serialize BE data (only 16 bits for now).

Tree-SHA512: bd67cf7eed465dad08551fb62f659e755e0691e4597a9f59d285d2b79975b50e5710d35a34a185b5ad232e1deda9a4946615f9132b1ed7d96ed8087f73ace66b
UdjinM6 added a commit to dashpay/dash that referenced this pull request May 14, 2021
UdjinM6 added a commit to UdjinM6/dash that referenced this pull request May 28, 2021
…xcept wallet/gui)

f9ee0f3 Add comments to CustomUintFormatter (Pieter Wuille)
4eb5643 Convert everything except wallet/qt to new serialization (Pieter Wuille)
2b1f85e Convert blockencodings_tests to new serialization (Pieter Wuille)
73747af Convert merkleblock to new serialization (Pieter Wuille)
d06fedd Add SER_READ and SER_WRITE for read/write-dependent statements (Russell Yanofsky)
6f9a1e5 Extend CustomUintFormatter to support enums (Russell Yanofsky)
769ee5f Merge BigEndian functionality into CustomUintFormatter (Pieter Wuille)

Pull request description:

  The next step of changes from bitcoin#10785.

  This:
  * Adds support for enum serialization to `CustomUintFormatter`, used in `CAddress` for service flags.
  * Merges `BigEndian` into `CustomUintFormatter`, used in `CNetAddr` for port numbers.
  * Converts everything (except wallet and gui) to use the new serialization framework.

ACKs for top commit:
  MarcoFalke:
    re-ACK f9ee0f3, only change is new documentation commit for CustomUintFormatter 📂
  ryanofsky:
    Code review ACK f9ee0f3. Just new commit adding comment since last review
  jonatack:
    Code review re-ACK f9ee0f3 only change since last review is an additional commit adding Doxygen documentation for `CustomUintFormatter`.

Tree-SHA512: e7a0a36afae592d5a4ff8c81ae04d858ac409388e361f2bc197d9a78abca45134218497ab2dfd6d031e0cce0ca586cf857077b7c6ce17fccf67e2d367c1b6cd4
UdjinM6 added a commit to UdjinM6/dash that referenced this pull request May 28, 2021
71f016c Remove old serialization primitives (Pieter Wuille)
92beff1 Convert LimitedString to formatter (Pieter Wuille)
ef17c03 Convert wallet to new serialization (Pieter Wuille)
65c589e Convert Qt to new serialization (Pieter Wuille)

Pull request description:

  This is the final step 🥳 of the serialization improvements extracted from bitcoin#10785.

  It converts the LimitedString wrapper to a new-style formatter, and updates the wallet and Qt code to use the new serialization framework. Finally all remaining old primitives are removed.

ACKs for top commit:
  jonatack:
    ACK 71f016c reviewed diff, builds/tests/re-fuzzed.
  laanwj:
    Code review ACK 71f016c

Tree-SHA512: d952194bc73259f6510bd4ab1348a1febbbf9862af30f905991812fb0e1f23f15948cdb3fc662be54d648e8f6d95b11060055d2e7a8c2cb5bf008224870b1ea1
gades pushed a commit to cosanta/cosanta-core that referenced this pull request Jun 24, 2021
…ithout FLATDATA

a7c45bc Add native support for serializing char arrays without FLATDATA (Pieter Wuille)

Pull request description:

  Support is added to serialize arrays of type `char` or `unsigned char` directly, without any wrappers. All invocations of the `FLATDATA` wrappers that are obsoleted by this are removed.

  This includes a patch by @ryanofsky to make `char` casting type safe.

  The serialization of `CSubNet` is changed to serialize a `bool` directly rather than though `FLATDATA`. This makes the serialization independent of the size of the bool type (and will use 1 byte everywhere).

  This is a small change taken from bitcoin#10785.

Tree-SHA512: a41f61ca5fdc2fadb2d0e1702351a58a23841d551f505292a9542602cdb19f90d8944b8df14b872810a56bd201648fa4c0e958f3e9427fe829886284e85b9bfd
Signed-off-by: pasta <pasta@dashboost.org>

# Conflicts:
#	src/test/serialize_tests.cpp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet