feat(docs): ADR on the consensus Write-Ahead Log (WAL) by cason · Pull Request #1408 · circlefin/malachite

cason · 2026-01-27T10:37:09Z

Closes #469.

romac · 2026-01-27T16:20:01Z

+So, ideally, it should be the right layer to invoke the WAL primitive for the
+logging of inputs and for replaying WAL entries retrieved from persistent
+storage upon recovery.


It's not clear to me why that is, given all the downsides listed below?

What is the actual upside that you see of making the driver aware of the WAL versus leaving it in core-consensus?

ancazamfir

Very nice doc, reads very well!
We should update the other adrs, especially adr-003 which was written more than 2 years ago.

ancazamfir · 2026-02-02T19:59:31Z

+2. When a consensus message is broadcast, via the `PublishConsensusMsg` effect;
+3. When a value is finalized by the consensus, via the `Decide` effect.
+
+> FIXME: is there any particular reason for item 1, i.e. starting a round?


I can't recall the exact reasoning. StartedRound notifies the host, which might be considered an "output" that could trigger external actions (e.g., proposer starts building a block). Flushing makes sure that the state leading to this round is persisted before the host acts on it. Maybe @romac knows.

It is listed here: #469 (comment), so it was a requirement. But it is not documented why.

Flushing makes sure that the state leading to this round is persisted before the host acts on it.

Yes, I think the reasoning was that StartRound is a state machine output and therefore needs to trigger a WAL flush.

Co-authored-by: Anca Zamfir <ancazamfir@users.noreply.github.com>

cason · 2026-02-03T07:28:13Z

We should update the other adrs, especially adr-003 which was written more than 2 years ago.

It is actually 4 months ago and mostly up-to-date. But I agree that ADR 001 is historical right now.

adizere

thanks Daniel, lgtm!

ancazamfir · 2026-02-03T09:06:06Z

We should update the other adrs, especially adr-003 which was written more than 2 years ago.

It is actually 4 months ago and mostly up-to-date. But I agree that ADR 001 is historical right now.

oh, i meant adr-001 indeed.

cason · 2026-02-03T12:52:44Z

Can someone make the checks to run? They are always canceled for some reason.

nenadmilosevic95

Wonderfull ADR! Great job! Thanks @cason, very well written and extremely useful!!! I left some comments!

nenadmilosevic95 · 2026-02-03T12:03:33Z

 | [003](./adr-003-values-propagation.md)      | Propagation of Proposed Values              | Accepted |
-| [004](./adr-004-coroutine-effect-system.md) | Coroutine-Based Effect System for Consensus | Accepted |
+| [004](./adr-004-coroutine-effect-system.md) | Coroutine-Based Effect System for Consensus | Accepted |
+| [006](./adr-006-write-ahead-log.md)         | Consensus Write-Ahead Log (WAL)             | Accepted |


The number conflicts with PoV ADR, can you use 007 instead?

nenadmilosevic95 · 2026-02-03T12:10:40Z

+> process is the locking mechanism present in multiple consensus algorithms.
+> In Tendermint, a process that emits a `Precommit` for a value `v` also
+> _locks_ `v` at that round.
+> The lock is a promise to not accept a value different than `v` in future


Unless we have L28-30 but I think for the purpose of this example this is precise enough

nenadmilosevic95 · 2026-02-04T08:13:35Z

+> recovered, then the process can receive a proposal for a value `v'` in round
+> `r` and misbehave in two ways:
+>
+> * Amnesia: by "forgetting" about the promise associated to the pre-crash


Amnesia can also be when process doesn't re-propose its valid_value this can cause equivocation.

Yes, there are multiple cases, I am just considering the ignoring lock misbehaviour as an example. This block was not in the initial version, then from some comments I realized that the rationale of replaying was not straightforward as I thought. So I am giving here an example.

I would then just write it slightly different, something like "misbehave in different ways:", so it doesn't seem there are only two ways. Anyway, this is just nitpicking so please ignore if not needed

nenadmilosevic95 · 2026-02-04T08:14:40Z

+>
+> * Amnesia: by "forgetting" about the promise associated to the pre-crash
+>   lock on `v`, accept the proposed value `v' != v`;
+> * Equivocation: for accepting `v'`, to emit a `Prevote` for `id(v')` in


Minor again but to me this is more a double-signing than equivocation

nenadmilosevic95 · 2026-02-04T09:56:50Z

+`process_input` method, which is the same used to process ordinary inputs.
+The `Phase::Recovering` flag is actually only used to block the `WalAppend`
+effect from appending again to the WAL inputs that are being replayed,
+as long as blocking calls to the associated WAL's `flush()` method.


I don't understand this sentence?

nenadmilosevic95 · 2026-02-04T10:00:30Z

+As a result, replayed inputs produce outputs, `Effect`s in the consensus engine
+parlance, in the same way as ordinary inputs, the exception being only the
+effects related to persisting inputs to the WAL.
+In any case, the existence of the `Phase::Recovering` flag allows filtering out
+behaviors, effects, an inputs that are not needed or redundant
+in recovery mode - although it should be used very carefully.
+Once replaying is done, the `Phase::Recovering` flag is cleared and processed
+inputs are appended to the existing WAL, which is not [reset](#reset-1) in this case.


I would polish this part a bit

nenadmilosevic95 · 2026-02-04T10:03:03Z

+externalized, either to the application (1 and 3) or the network (2).
+
+A current limitation of the persistence approach, however, is the fact that
+calls to `wal_append` are also blocking, while they do not have to.


This means that we at the moment we don't do asynchronous writes?

We do, but they are blocking.

nenadmilosevic95 · 2026-02-04T10:09:29Z

+| [004](./adr-004-coroutine-effect-system.md) | Coroutine-Based Effect System for Consensus | Accepted |
+| [007](./adr-007-write-ahead-log.md)         | Consensus Write-Ahead Log (WAL)             | Accepted |


Thanks for changing to 007, I wanted to let you use 006 since you are older :)

romac

Amazing work 🎉

Co-authored-by: nenadmilosevic95 <nenad.milosevic@circle.com>

nenadmilosevic95 · 2026-02-04T13:20:09Z

+> recovered, then the process can receive a proposal for a value `v'` in round
+> `r` and misbehave in two ways:
+>
+> * Amnesia: by "forgetting" about the promise associated to the pre-crash


I would then just write it slightly different, something like "misbehave in different ways:", so it doesn't seem there are only two ways. Anyway, this is just nitpicking so please ignore if not needed

This comment was marked as off-topic.

Sign in to view

github-actions Bot added the need-triage This issue needs to be triaged label Jan 27, 2026

github-actions Bot closed this Jan 27, 2026

cason reopened this Jan 27, 2026

cason removed the need-triage This issue needs to be triaged label Jan 27, 2026

romac reviewed Jan 27, 2026

View reviewed changes

cason added 8 commits February 2, 2026 10:03

docs: ADR 006 for Consensus Write-Ahead Log (WAL)

ecf2d13

docs: ADR 003, intial content

4168077

docs: ADR on WAL, inputs to be persisted

52427c1

docs: ADR on WAL, checkpoints every height

8dfaacf

docs: ADR on WAL, sync vs async persistenc

1244b2a

docs: ADR on WAL, replay impl discussion

5f485df

docs: ADR on WAL, start of Decision session

c50c75d

docs: ADR on WAL, reset, replay, persistence

6c2e979

cason force-pushed the adr-wal branch from 3e6cd58 to 6c2e979 Compare February 2, 2026 09:04

cason added 3 commits February 2, 2026 11:21

docs: WAL, layers reviewed and discussion

09bd168

docs: WAL, error handling, corruption

c248776

docs: ADR WAL, links and references

f621c71

cason marked this pull request as ready for review February 2, 2026 13:09

cason requested a review from ancazamfir as a code owner February 2, 2026 13:09

cason requested a review from nenadmilosevic95 February 2, 2026 14:02

docs: ADR on WAL, applying suggestions from @hvanz

530d75d

ancazamfir reviewed Feb 2, 2026

View reviewed changes

cason and others added 4 commits February 3, 2026 07:59

Apply suggestions from @ancazamfir

fb39168

Co-authored-by: Anca Zamfir <ancazamfir@users.noreply.github.com>

docs: ADR on WAL, replacing Jira link to issue

836c4dc

docs: adding @ancazamfir suggestion

6484d11

docs: ADR on WAL, synchronous writes comment

0a7abb2

adizere approved these changes Feb 3, 2026

View reviewed changes

docs: ADR on WAL, events buffering

ba70e06

cason dismissed ancazamfir’s stale review via ba70e06 February 3, 2026 12:52

cason added 4 commits February 3, 2026 13:54

docs: ADR 006 -> 007 in favor of circlefin#1436

932b2ce

docs: ADR on WAL, removing for reviewers comments

1fc5aaa

docs: ADR on WAL, new issue linked for certificates

e27c8a3

Merge branch 'main' into adr-wal

d235a7b

cason requested review from ancazamfir and romac February 4, 2026 08:36

ancazamfir previously approved these changes Feb 4, 2026

View reviewed changes

nenadmilosevic95 reviewed Feb 4, 2026

View reviewed changes

romac previously approved these changes Feb 4, 2026

View reviewed changes

romac self-requested a review February 4, 2026 11:07

Apply suggestions from @nenadmilosevic95

5dc17ed

Co-authored-by: nenadmilosevic95 <nenad.milosevic@circle.com>

cason dismissed stale reviews from romac and ancazamfir via 5dc17ed February 4, 2026 11:21

Addressing comments from @nenadmilosevic95

3771ad8

cason requested a review from nenadmilosevic95 February 4, 2026 12:56

Merge branch 'main' into adr-wal

04bdcec

nenadmilosevic95 approved these changes Feb 4, 2026

View reviewed changes

cason added 2 commits February 4, 2026 15:07

Addressing a comment from @nenadmilosevic95

9287ce2

docs: ADR on WAL, final version?

57eaabd

romac previously approved these changes Feb 5, 2026

View reviewed changes

Merge branch 'main' into adr-wal

05ed47d

cason dismissed romac’s stale review via 05ed47d February 5, 2026 08:17

docs: ADR's README file renamed

dd33a25

romac approved these changes Feb 5, 2026

View reviewed changes

romac added this pull request to the merge queue Feb 5, 2026

Merged via the queue into circlefin:main with commit 1cab18f Feb 5, 2026
16 checks passed

		\| [004](./adr-004-coroutine-effect-system.md) \| Coroutine-Based Effect System for Consensus \| Accepted \|
		\| [007](./adr-007-write-ahead-log.md) \| Consensus Write-Ahead Log (WAL) \| Accepted \|

Conversation

cason commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as off-topic.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ancazamfir left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cason Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cason commented Feb 3, 2026

Uh oh!

adizere left a comment

Choose a reason for hiding this comment

Uh oh!

ancazamfir commented Feb 3, 2026

Uh oh!

cason commented Feb 3, 2026

Uh oh!

nenadmilosevic95 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

romac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

cason commented Jan 27, 2026 •

edited

Loading

cason Feb 3, 2026 •

edited

Loading