Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 33 additions & 7 deletions pages/fundamentals/data-durability.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,11 @@ Per-transaction WAL checksums were introduced in Memgraph v3.12.
WAL files written by older versions do not contain checksums and are recovered
without integrity verification. Checksum verification applies only to WAL files
written by Memgraph v3.12 or newer.

Before Memgraph v3.13, a checksum mismatch anywhere in the WAL chain recovered
the readable prefix of that file and then continued with the files after it,
which could bring the database up with an incomplete dataset without reporting
it. Since v3.13, damage in a completed WAL file fails recovery instead.
</Callout>

To guard against silent on-disk corruption, Memgraph protects WAL files with
Expand All @@ -113,16 +118,37 @@ To guard against silent on-disk corruption, Memgraph protects WAL files with
- **Each transaction** is protected by a 4-byte checksum covering the
transaction's bytes (transaction start, deltas and transaction end).

Checksums are verified automatically during recovery. If a transaction's stored
checksum does not match the recomputed value, the WAL is considered corrupted at
that point and recovery stops, so corrupted data is never applied to the
database. A mismatch in the WAL header causes recovery from that file to fail.
Checksums are verified automatically during recovery, so corrupted data is never
applied to the database. A mismatch in the WAL header causes recovery from that
file to fail. What happens on a transaction checksum mismatch depends on whether
the file was completed:

- A **completed (finalized) WAL file** — one Memgraph finished writing and
rotated away from — was flushed to disk before it was closed, so every
transaction in it is durable and was already acknowledged. A mismatch there
means the bytes on disk rotted, and recovery fails rather than continuing with
part of the file. Recovering only a prefix would be unsound, because the WAL
files that follow were written on top of the transactions that went missing.
- A **WAL file that was still being written** when the process stopped may
legitimately have a torn tail. Recovery applies its transactions up to the
last whole one and stops there, which is how an interrupted write degrades
gracefully.

Files that the newest snapshot already covers are skipped without being read, so
corruption in those is harmless. If a WAL file written after the newest snapshot
is damaged, the database fails to recover; use
[`--storage-allow-recovery-failure`](/database-management/configuration) together
with `RECOVER SNAPSHOT` (see [recovery failure
handling](#recovery-failure-handling)), or restore from a backup.

The same checksums protect WAL files that are buffered on disk on a replica
before being applied, so corruption introduced between the main and the replica
is detected before the data is committed. Deltas streamed during the commit
(`PrepareCommitRpc`) are not checksummed because the TCP transport already
provides integrity guarantees.
is detected before the data is committed. When a replica hits damage in a
finalized file it reports the failure and stops applying the rest of the batch
instead of skipping ahead; the in-flight transaction is aborted, main does not
advance its view of the replica, and the transfer is retried later. Deltas
streamed during the commit (`PrepareCommitRpc`) are not checksummed because the
TCP transport already provides integrity guarantees.

<Callout type="info">
Snapshots are not yet protected by checksums.
Expand Down
19 changes: 19 additions & 0 deletions pages/release-notes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,19 @@ guide.
- Fixed a crash when multiple sessions printed query plans at the same time
(for example concurrent `EXPLAIN` / `PROFILE`, or query-plan logging).
[#4489](https://github.com/memgraph/memgraph/pull/4489)
- Fixed silent data loss when a WAL file other than the last one in the chain
was damaged. Recovery used to apply the readable prefix of that file and then
apply the following files in full, coming up with an incomplete dataset
without reporting anything; on a replica the divergence never healed because
the replica reported the later timestamp and main considered it caught up.
Damage in a completed (finalized) WAL file is now fatal: the database fails
recovery instead of starting stale, and a replica stops applying the rest of
the chain. A WAL file that was still being written when the process crashed
is unaffected and is still truncated to its last whole transaction. Recover
such an instance with
[`--storage-allow-recovery-failure`](/database-management/configuration) plus
`RECOVER SNAPSHOT`, or from a backup.
[#4528](https://github.com/memgraph/memgraph/pull/4528)
- `TERMINATE TRANSACTIONS` no longer reports `killed: true` for a transaction
the caller is not authorized to terminate. Such a match now reports
`killed: false`, indistinguishable from an id that does not exist.
Expand Down Expand Up @@ -163,6 +176,12 @@ guide.
statements). Under index-heavy workloads this removes GC-correlated latency
spikes on index and constraint creation.
[#4468](https://github.com/memgraph/memgraph/pull/4468)
- Recovery from WAL files is roughly 35% faster. Each WAL file now records its
timestamp range and transaction count in its header, so deciding which files
to recover from no longer requires parsing every file end to end, and
replaying a file parses it once instead of twice. This also speeds up replica
recovery and the WAL cleanup that runs after every snapshot.
[#4528](https://github.com/memgraph/memgraph/pull/4528)
- Cluster management queries forwarded from a follower coordinator to the leader
now have [explicit RPC
timeouts](/clustering/high-availability/how-high-availability-works#rpc-timeouts),
Expand Down