History pruning (fixes #4419) #4445
Conversation
|
draft until status-im/nim-eth#574 is merged |
arnetheduck
force-pushed
the
prune-history
branch
from
15fbe9b
to
a7747b4
Compare
arnetheduck
force-pushed
the
prune-history
branch
2 times, most recently
from
ef556d4
to
98cffd2
Compare
| # rolled back automatically by the error response, then the ROLLBACK command | ||
| # will fail with an error, but no harm is caused by this. | ||
| # | ||
| if isInsideTransaction(db.db): # calls `sqlite3_get_autocommit` |
There was a problem hiding this comment.
Is this nested transaction code newly useful specifically with pruning?
There was a problem hiding this comment.
yes (at least it was, at some stage of the pruning development 😄) - it's a bit of a mess, but the function is available at different abstraction levels - since it's a tool for performance (and not atomicity of operations) it kind of make sense to apply it liberally when there are multiple writes to group, so it's slightly easier to allow nesting because a "super-composed" operation like wiping lots of states will be made up of "composed" operations like wiping a state and its corresponding state root
| legacy: bool = true): bool = | ||
| if db.statesNoVal[fork].contains(key.data).expectDb(): return true | ||
|
|
||
| (legacy and fork == BeaconStateFork.Phase0 and db.v0.containsState(key)) |
There was a problem hiding this comment.
Is support for this v0 database schema still relevant? At the moment, it could still be backfilled, since the network block availability still stretches back to genesis. Another approach might be to cut off this data and let it be recreated, while that's still feasible.
There was a problem hiding this comment.
the v0 data appears in any database that's been with us since genesis and covers the initial months - there's no simple way to "recreate" it that doesn't introduce complexity or performance issues, since it's a non-trivial amount of data whose performance profile is marred by the without_rowid mistake (ie deletes in particular are slow because they cause btree rebalances that rewrite all data).
We generally have never "rewritten" any tables in the database for the simple reason that there's never really a good time to do so - recreating states for example (as in this particular example, because it is a state function) would require reindexing / replay to reach parity, which is slow.
There was a problem hiding this comment.
I'm not that strongly opposed to keeping it, but it's this special case which cuts across much of beacon_chain_db, and there's a relatively finite, albeit not that short, time window where it can be quite naturally reconstructed. Doing so is a one-time cost borne in the background to homogenize the database schema.
There was a problem hiding this comment.
The way forward for "keeping" the data in a pruned world I think is to rely on era files - as such, changing to the current table format is of dubious value - also, we've gone through other changes to the tables such as switching to framed snappy in bellatrix, enabling rowid:s etc - if we did a migration for each of those, we'd be looking at hour-long restart times which certainly would not be popular - or days of background processing for data that ultimately is read only rarely.
Regarding reconstruction, we actually have --reindex - it is now able to read era files and create state table entries - I'm seeing this as the way forward for "upgrading" a database to state diffs as well - part of the "pruning" series of PR: was to ensure that this works with era and can start at any point in time (as opposed to genesis-only) - this allows a user to move between a pruned and a full node with relative ease.
I'm somewhat hesitant however to require that "defaults-everything" users that have used nimbus for a long while have to do this kind of manual work to keep using nimbus, so I find it easier to maintain minimal read-only support in chaindb.
Introduce (optional) pruning of historical data - a pruned node will continue to answer queries for historical data up to `MIN_EPOCHS_FOR_BLOCK_REQUESTS` epochs, or roughly 5 months, capping typical database usage at around 60-70gb. To enable pruning, add `--history=prune` to the command line - on the first start, old data will be cleared (which may take a while) - after that, data is pruned continuously. When pruning an existing database, the database will not shrink - instead, the freed space is recycled as the node continues to run - to free up space, perform a trusted node sync with a fresh database. When switching on archive mode in a pruned node, history is retained from that point onwards. History pruning is scheduled to be enabled by default in a future release. In this PR, `minimal` mode from #4419 is not implemented meaning retention periods for states and blocks are always the same - depending on user demand, a future PR may implement `minimal` as well.
arnetheduck
force-pushed
the
prune-history
branch
from
98cffd2
to
0b901bb
Compare
Introduce (optional) pruning of historical data - a pruned node will continue to answer queries for historical data up to
MIN_EPOCHS_FOR_BLOCK_REQUESTSepochs, or roughly 5 months, capping typical database usage at around 60-70gb.To enable pruning, add
--history=pruneto the command line - on the first start, old data will be cleared (which may take a while) - after that, data is pruned continuously.When pruning an existing database, the database will not shrink - instead, the freed space is recycled as the node continues to run - to free up space, perform a trusted node sync with a fresh database.
When switching on archive mode in a pruned node, history is retained from that point onwards.
History pruning is scheduled to be enabled by default in a future release.
In this PR,
minimalmode from #4419 is not implemented meaning retention periods for states and blocks are always the same - depending on user demand, a future PR may implementminimalas well.