Avoid persisting redundant tx sets #3501

marta-lokhova · 2022-08-09T17:25:53Z

Resolves #3499

marta-lokhova · 2022-08-09T18:11:19Z

note: I realized some functionality is currently missing from the PR, so please don't review it yet

marta-lokhova · 2022-08-11T19:41:35Z

This is up-to-date now. Note that protocol-next doesn't build at the moment because we need to update XDR upsteam. I'll do it after a round of review though, just in case XDR needs to change.

marta-lokhova · 2022-08-12T20:25:07Z

src/main/PersistentState.cpp

@@ -20,18 +21,52 @@ using namespace std;
 std::string PersistentState::mapping[kLastEntry] = {
    "lastclosedledger", "historyarchivestate", "lastscpdata",
    "databaseschema",   "networkpassphrase",   "ledgerupgrades",
-    "rebuildledger",    "lastscpdataxdr"};
+    "rebuildledger",    "lastscpdataxdr",      "txset"};

 std::string PersistentState::kSQLCreateStatement =
    "CREATE TABLE IF NOT EXISTS storestate ("
    "statename   CHARACTER(32) PRIMARY KEY,"


Interesting, it looks like sqlite (which is what we use for tests) doesn't actually enforce this limit, but it's now too small because we reference tx sets by hash. I got a failure on Postgres though, so am currently fixing this.

UPD: this is fixed now (added extra tests for Postgres as well)

dmkozh · 2022-08-12T20:10:29Z

src/herder/TxSetFrame.h

@@ -57,6 +58,9 @@ class TxSetFrame : public NonMovableOrCopyable
    makeFromWire(Hash const& networkID,
                 GeneralizedTransactionSet const& xdrTxSet);

+    static TxSetFrameConstPtr
+    getTxSetFramePtr(StoredTransactionSet const& storedSet, Application& app);


This is also a factory method, isn't it? I would rather call it makeFromStoredTxSet for consistency with other factory methods. Also, could you please add a short doc comment?

good idea, done.

dmkozh · 2022-08-12T21:21:30Z

src/main/PersistentState.cpp

+            {
+                auto txSetPtr = TxSetFrame::getTxSetFramePtr(txSet, mApp);
+                txSets.emplace(
+                    std::pair(txSetPtr->getContentsHash(),


nit: there is no need to use the pair constructor when you're using emplace

True, just a habit of writing things down explicitly. Updated (in the other place too)

dmkozh · 2022-08-12T21:29:56Z

src/main/ApplicationImpl.cpp

@@ -308,6 +308,9 @@ ApplicationImpl::initialize(bool createNewDB, bool forceRebuild)
        upgradeToCurrentSchemaAndMaybeRebuildLedger(true, forceRebuild);
    }

+    // Subtle: restore persistent state after a potential DB schema upgrade
+    mPersistentState->restoreState();


Just to double-check: this should be safe for restarts without upgrades, right?

Yeah, when a node upgrades its version to include this change, it'll do a schema upgrade, which will replace all of the old-style rows with new ones. If there is no upgrade, it means the schema is already up-to-date, which is what restoreState expects (as it queries PersistentStateV1)

dmkozh · 2022-08-12T21:32:52Z

src/herder/HerderImpl.cpp

        }
+        txSetsToPersist.emplace(std::pair(


nit: no need to use pair constructor with emplace

dmkozh · 2022-08-12T21:38:14Z

src/herder/test/HerderTests.cpp

+                    continue;
+                }
+                auto msgs = herder.getSCP().getLatestMessagesSend(i);
+                if (expectedSCPState.has_value())


nit: if (expectedSCPState) is sufficient, though I suppose you could use the explicit check for expressivity, so up to you

kept it as-is if you don't mind: has_value is used pretty commonly in the codebase, and I do like the explicitness of it

MonsieurNicolas · 2022-08-16T01:24:09Z

src/database/Database.cpp

+        // Sqlite does not enforce size limits
+        if (!isSqlite())
+        {
+            PersistentState::upgradeSizeLimit(mApp.getDatabase());


Can't you make the schema upgrade on sqlite instead of skipping? That way in the future (when we squash sql upgrades) we can use the same code to create tables (and the schema is the same everywhere)

ALTER COLUMN doesn't exist in sqlite. Instead, you have to create a new table and copy its contents from the old table. It doesn't really make sense to do this, given that this upgrade is a no-op on sqlite.

oh I see, so a sad sqlite limitation.
So maybe update the comment to something a little more specific: "sqlite does not enforce size constraints and does not support updating columns".

Also, you should just move this to upgradeSCPDataV1Format: there is no reason to split the schema upgrade into 2 parts.

src/herder/HerderImpl.cpp

MonsieurNicolas · 2022-08-16T01:39:33Z

src/main/PersistentState.cpp

+            xdr::xdr_from_opaque(buffer, scpState);
+            for (auto const& e : scpState.v1().scpEnvelopes)
+            {
+                for (auto const& hash : getTxSetHashes(e))


This is probably not the right place for this code: it's supposed to match logic that lives right now in HerderImpl. That way we let PersistentState be a "dumb" key/value store.

Moved it to Herder.

MonsieurNicolas · 2022-08-16T01:55:05Z

src/main/PersistentState.cpp

+}
+
+void
+PersistentState::startTxSetGCTimer()


I don't really like having this GC timer setup independently of what Herder is doing (at a minimum this should be managed by Herder).
Do we really need to GC using a timer?
It can be done on startup and every X times we store SCP state or maybe it should be based on the number of TxSets that we have in the database (we can estimate that from herder, doest not need to be exact)?

Maybe we should just reuse the code in HerderPersistence for all this (there is quite a bit of overlap, but I understand it may not be worth it)?

I'm not a fan of the suggested approaches, because they both make it harder to reason about what's happening:

we don't know how many times SCP persistence is called per ledger, so we can't really reason about how often tx sets are cleaned up (same thing with the number of tx sets).

It makes testing harder as now we have to into account cases where purging triggers at different times depending on consensus.

I went with the timer, as it seems to be the simplest approach that avoids anything complicated where we have to map Herder's state to the database and try to estimate a good time to do the cleanup.

I suggest we keep the timer approach. I do like the idea of moving it to Herder, this way we don't need extra start/shutdown logic in PersistentState, and it'll be easy to refactor the purging function based on your other suggestion.

actually you're right, I was thinking that we could get arbitrary number of values from other validators, so we should try to GC more often in some situations, but this is only tracking what the current validator is doing.

MonsieurNicolas · 2022-08-16T01:56:30Z

src/herder/HerderImpl.cpp

 }

 void
 HerderImpl::restoreSCPState()
 {
    ZoneScoped;

+    // First, load all known tx sets


this code and GC may be able to share code (and we want to GC before loading all those txsets)

I had the same thought, but ultimately decided to keep them separate: the functions are sufficiently different, so refactoring didn't actually simplify much.

src/main/PersistentState.cpp

MonsieurNicolas · 2022-08-19T00:04:34Z

r+ f13ddfe

lokera666

Yeah

marta-lokhova force-pushed the persistent_state branch 5 times, most recently from 0a06b7c to a8fc982 Compare August 11, 2022 19:38

marta-lokhova commented Aug 12, 2022

View reviewed changes

dmkozh reviewed Aug 12, 2022

View reviewed changes

marta-lokhova added this to In progress in v19.4.0 via automation Aug 12, 2022

marta-lokhova force-pushed the persistent_state branch 2 times, most recently from c29ea2d to 862b15c Compare August 15, 2022 18:20

marta-lokhova requested a review from MonsieurNicolas August 15, 2022 21:50

MonsieurNicolas requested changes Aug 16, 2022

View reviewed changes

marta-lokhova mentioned this pull request Aug 18, 2022

Pick up persistent state changes in core stellar/stellar-xdr#19

Merged

marta-lokhova force-pushed the persistent_state branch from c01b46d to be65ec2 Compare August 18, 2022 20:52

marta-lokhova added 3 commits August 18, 2022 16:17

Avoid persisting redundant tx sets

d808d60

Bump src/protocol-next/xdr version to pickup persistent state changes

487fee0

Fix rename failures

f13ddfe

marta-lokhova force-pushed the persistent_state branch from 3e7d437 to f13ddfe Compare August 18, 2022 23:18

latobarita merged commit 7c6de8b into stellar:master Aug 19, 2022

v19.4.0 automation moved this from In progress to Done Aug 19, 2022

marta-lokhova deleted the persistent_state branch August 19, 2022 17:16

lokera666 reviewed Oct 1, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid persisting redundant tx sets #3501

Avoid persisting redundant tx sets #3501

marta-lokhova commented Aug 9, 2022

marta-lokhova commented Aug 9, 2022

marta-lokhova commented Aug 11, 2022

marta-lokhova Aug 12, 2022 •

edited

dmkozh Aug 12, 2022

marta-lokhova Aug 15, 2022

dmkozh Aug 12, 2022

marta-lokhova Aug 15, 2022

dmkozh Aug 12, 2022

marta-lokhova Aug 12, 2022

dmkozh Aug 12, 2022

dmkozh Aug 12, 2022

marta-lokhova Aug 15, 2022

MonsieurNicolas Aug 16, 2022

marta-lokhova Aug 16, 2022

MonsieurNicolas Aug 16, 2022

MonsieurNicolas Aug 16, 2022

marta-lokhova Aug 16, 2022

MonsieurNicolas Aug 16, 2022

marta-lokhova Aug 16, 2022

MonsieurNicolas Aug 16, 2022

MonsieurNicolas Aug 16, 2022

marta-lokhova Aug 17, 2022

MonsieurNicolas commented Aug 19, 2022

lokera666 left a comment

Avoid persisting redundant tx sets #3501

Avoid persisting redundant tx sets #3501

Conversation

marta-lokhova commented Aug 9, 2022

marta-lokhova commented Aug 9, 2022

marta-lokhova commented Aug 11, 2022

marta-lokhova Aug 12, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MonsieurNicolas commented Aug 19, 2022

lokera666 left a comment

Choose a reason for hiding this comment

marta-lokhova Aug 12, 2022 •

edited