Multiversion: Fix upgrade/downgrade TOCTOU#2758
Merged
Merged
Conversation
Ensure release numbers are displayed as release triples, not just integers.
Previously we asserted the invariant: _if a replica is downgrading, then it must be on the newest release in the binary_.
But this is only kind-of true!
Scenario:
1. Replica starts on release A.
2. Replica detects that its binary has been replaced by B.
It reads the binary of B into a memfd.
3. Replica decides to upgrade to B, so it exec()'s the memfd.
4. (Swap B's binary on disk with C.)
5. Replica starts up, running B's binary.
6. During open, replica reads the binary's header from disk.
But that's C's binary/header, so B is unexpectedly not the latest release in it.
This is a tricky bug! It manifests with the following stack trace:
```
info(replica): 0: release_transition: release=0.16.16..0.16.17 (reason=commit_dispatch)
info(multiversioning): executing current release 0.16.18 (target: 0.16.17) via /opt/tigerbeetle/tigerbeetle...
2025-02-14 14:48:33.574Z info(io): opening "data"...
2025-02-14 14:48:34.053Z info(main): release=0.16.18
2025-02-14 14:48:34.053Z info(main): release_client_min=0.15.3
2025-02-14 14:48:34.053Z info(main): releases_bundled={ 0.16.14, 0.16.16, 0.16.17, 0.16.18, 0.16.19 }
2025-02-14 14:48:34.053Z info(main): git_commit=ac1c58b344c3cc04fbb4211e126553a7c7db1c4f
2025-02-14 14:48:34.866Z info(replica): superblock release=0.16.17
thread 1556 panic: reached unreachable code
tigerbeetle/zig/lib/std/debug.zig:412:14: 0x12d0f7d in assert (tigerbeetle)
tigerbeetle/src/vsr/replica.zig:9335:23: 0x131930f in release_transition (tigerbeetle)
tigerbeetle/src/vsr/replica.zig:629:40: 0x1317e0d in open (tigerbeetle)
tigerbeetle/src/tigerbeetle/main.zig:362:21: 0x131dc85 in start (tigerbeetle)
tigerbeetle/src/tigerbeetle/main.zig:79:30: 0x13b20f1 in main (tigerbeetle)
tigerbeetle/zig/lib/std/start.zig:524:37: 0x12d0d45 in posixCallMainAndExit (tigerbeetle)
tigerbeetle/zig/lib/std/start.zig:266:5: 0x12d0861 in _start (tigerbeetle)
???:?:?: 0x4 in ??? (???)
Unwind information for `???:0x4` was not available, trace may be incomplete
```
...Which is misleading -- it _looks_ like we have somehow transitioned directly from `0.16.16` to `0.16.18` within `0.16.19`'s binary, which shouldn't be possible. And it isn't! We are in `0.16.18`'s binary (via memfd) but when `0.16.18` reads the binary's multiversion header it gets `0.16.19`'s header.
Member
Author
|
A couple more notes:
|
matklad
approved these changes
Feb 20, 2025
matklad
left a comment
Member
There was a problem hiding this comment.
cc @fabioarnold that's another thing that vörtex upgrade test should be able to catch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
Previously we asserted the invariant: if a replica is downgrading, then it must be on the newest release in the binary.
But this is only kind-of true!
Scenario:
It reads the binary of B into a memfd.
But that's C's binary/header, so B is unexpectedly not the latest release in it.
This is a tricky bug! It manifests with the following stack trace:
...Which is misleading -- it looks like we have somehow transitioned directly from
0.16.16to0.16.18within0.16.19's binary, which shouldn't be possible. And it isn't! We are in0.16.18's binary (via memfd) but when0.16.18reads the binary's multiversion header it gets0.16.19's header.Fix
Don't assert the incorrect invariant! If we detect this race, log a warning.
Also in this PR: