-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(core): support WAL sequencer transaction file partitioned format #4194
Conversation
Conflicts: core/src/test/java/io/questdb/test/ServerMainTest.java
core/src/main/java/io/questdb/cairo/wal/seq/TableTransactionLogFile.java
Show resolved
Hide resolved
|
||
import static io.questdb.cairo.wal.WalUtils.*; | ||
|
||
public class TableTransactionLogV2 implements TableTransactionLogFile { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please document why two implementations are required
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added javadoc
|
||
long getPartNo(); | ||
|
||
// Sets cursor to minimum available position. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
terminology, it might be easier to use txn
instead, such as min txn, vs minimum position etc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
renamed to toMinTxn()
Conflicts: core/src/test/java/io/questdb/test/ServerMainBackupDatabaseTest.java
Conflicts: core/src/main/java/io/questdb/cairo/DatabaseSnapshotAgentImpl.java
Conflicts: core/src/test/java/io/questdb/test/cairo/CairoEngineTest.java core/src/test/java/io/questdb/test/wal/WalTxnDetailsFuzzTest.java
[PR Coverage check]😍 pass : 668 / 724 (92.27%) file detail
|
WAL sequencer file
_txnlog
records all 28 bytes per transaction and currently there is no way to reduce the space taken by this file. This can add up to many GiB of storage over time. This PR introduces new storage of_txnlog
where it is chunked into multiple fixed-size files.The chunks or parts of
_txnlog
are stored in_txn_parts
subdirectory as files with numeric names:When all the transactions from the part are fully applied to the table, the part is deleted by
WalPurgeJob
"Old" non-chunked sequencer files are fully supported. To switch for a new table to use chunked format a configuration option
cairo.default.wal.seq.chunk.txn.count
to be set, for example to the value of 5000.Also, this change adds additional commit details to the sequencer log of the new format: