Adopt low level byte serialization #88
Comments
|
D'oh, I should have collected more data before parsing from zero.. :) Anyway, some data with the serialization via 4ef1b9eb0e: It's notable that the tests were done on top of #74, which explains the 4-10x bump for the 2nd+ calls, due to the input transaction cache in It can be seen that the average time to list one transaction is roughly 47 ms uncached, and around 5.2 ms after the first call. The first time it took 4:04 min to list all 5151 Omni wallet transactions (of 5343 wallet transactions in total), and only about 27 seconds to list all transactions, if called a second time. Once the mainnet parsing finished, I'm going to get some numbers to compare for the JSON serialization. |
|
Here we go, without byte serialization: So the average time for one uncached transaction is 127 ms, and around 89 ms for subsequent calls. In total, it took 10:52 min (vs. 4:04 min) to list all 5151 wallet transactions, if called the first time, and 7:37 min (vs. 27 seconds) for cached queries. That's around 17x slower. |
|
This is EPIC - nice work mate!!!!! |
|
Given that these results are better than expected, we should probably convert the other parts as well. And since we're using the serialization of Bitcoin Core, things could get a lot easier, because values can simply be wrapped into Say there is a class to represent recipients of STO transactions: class CSerializableStoRecipients
{
private:
uint256 hashTxid;
uint256 hashBlock;
std::string strSender;
uint32_t propertyId;
int64_t amountFee;
std::set<std::pair<int64_t, std::string> > recipientsSet;
public:
/** Creates a new recipients object */
CSerializableStoRecipients()
{
// Set null or so
}
ADD_SERIALIZE_METHODS;
template <typename Stream, typename Operation>
inline void SerializationOp(Stream& s, Operation ser_action, int nType, int nVersion) {
READWRITE(hashTxid);
READWRITE(hashBlock);
READWRITE(strSender);
READWRITE(propertyId);
READWRITE(amountFee);
READWRITE(recipientsSet);
}
/** Transaction hash */
uint256 getTxid() const { return hashTxid; }
/** Block hash */
uint256 getBlock() const { return hashBlock; }
/** Sender */
std::string getSender() const { return strSender; }
/** Property identifier */
uint32_t getPropertyId() const { return propertyId; }
/** Amount paid as STO fee */
int64_t getFeeAmount() const { return amountFee; }
/** Recipients */
std::set<std::pair<int64_t, std::string> > getRecipients() const { return recipientsSet; }
/** Sum of all recipient values */
int64_t getTotalAmount() const;
}Then it could be stored or retrieved as follows: bool CMPSTOList::StoreRecipients(const CSerializableStoRecipients& recipients)
{
// DB key for entry
CDataStream ssKey(SER_DISK, CLIENT_VERSION);
ssKey << std::make_pair('t', recipients.getTxid());
leveldb::Slice slKey(&ssKey[0], ssKey.size());
// DB value for entry
CDataStream ssValue(SER_DISK, CLIENT_VERSION);
ssValue.reserve(ssValue.GetSerializeSize(recipients));
ssValue << recipients;
leveldb::Slice slValue(&ssValue[0], ssValue.size());
// Persist entry
leveldb::Status status = pdb->Put(writeoptions, slKey, slValue);
return status.ok();
}bool CMPSTOList::GetRecipients(const uint256& hashTxid, CSerializableStoRecipients& retRecipients) const
{
// DB key for entry
CDataStream ssKey(SER_DISK, CLIENT_VERSION);
ssKey << std::make_pair('t', hashTxid);
leveldb::Slice slKey(&ssKey[0], ssKey.size());
// DB value for entry
std::string strValue;
leveldb::Status status = pdb->Get(readoptions, slKey, &strValue);
if (!status.ok()) {
PrintToLog("%s(): ERROR: %s\n", __func__, status.ToString());
return false;
}
// Deserialize entry
try {
CDataStream ssValue(strValue.data(), strValue.data() + strValue.size(), SER_DISK, CLIENT_VERSION);
ssValue >> retRecipients;
} catch (const std::exception& e) {
PrintToLog("%s(): ERROR: %s\n", __func__, e.what());
return false;
}
return true;
}No need to manually parse the list of recipients ... :) |
As described in #85 (comment), one of the most time consuming parts in the processing flow is the serialization and deserialization of smart properties.
This is caused by the conversion of
string -> JSON -> EntryandEntry -> JSON -> string, and not so much by fetching or storing data to the database.I was curious, if we can improve it by adopting low level byte serialization, and the results are very impressive. The follownig data was captures on mainnet, averaged over 500 runs and based this commit:
For SP 4 on mainnet:
For SP 3 on mainnet:
As proof-of-concept I removed the
toJSON()andfromJSON()methods ofCMPSPInfo::Entryand replaced the related parts where entries are stored or retrieved:I'm currently parsing mainnet transactions to get a feeling for the speed improvements in practise. Given that
CMPSPInfo::getSP()is used almost everywhere, it should be notable, especially in the context oflisttransactions_MP, which is primarily slowed down due to the parsing of SP3.I really, really don't want to start yet another feature before the release, but we should seriously consider replacing the critical parts.
If it turns out, as I hope, that
listtransactions_MPgets a speed boost of 50x or more, then I'm going to prepare a clean replacement forgetSP(),putSP()and friends.Otherwise, and regarding the other parts where serialization may play a role, I suggest to adopt byte serialization after the release.
What do you think?
The text was updated successfully, but these errors were encountered: