epoch system

[matthew-levan]

This PR implements a new format for how piers store their event logs on disk.

Resolves #313.

Design

Existing format:

./zod/.urb/log
├── data.mdb
└── lock.mdb

New format:

./zod/.urb/log
├── 0i0             # epoch dirnames specify the last event of the previous epoch
│   ├── data.mdb    # lmdb file containing events 1-132
│   ├── epoc.txt    # disk format version (this PR starts versioning at 1)
│   ├── lock.mdb    # lmdb lock file
│   └── vere.txt    # binary version this set of events was originally run with
└── 0i132
    ├── data.mdb
    ├── epoc.txt
    ├── lock.mdb
    ├── north.bin   #
    ├── south.bin   # snapshot files (state as of event 132), strictly read-only
    └── vere.txt

The new format introduces epochs, which are simply "slices" or "chunks" of a ship's complete event log. Above, you can see the ship's event log chunked into two epochs: 0i0 and 0i132.

New ships booted with the code in this PR instantiate their log directories with the new format. Existing piers are automatically migrated on boot.

Epoch "rollovers" (when the current epoch is ended and a new, empty epoch is created) occur under three conditions:

1. The pilot uses the new roll subcommand to manually rollover.
2. The pilot runs the chop subcommand.
3. We detect a different running binary version than the one pinned in the current epoch.

Both migrations and epoch rollovers ensure there's a current snapshot before running.

A few TODOs left:

• Iron out small kink in migration behavior for previously chopped piers
• Make sure correct binary version gets pinned to first epoch of migrated piers
• Rollover to new epoch when a new binary version is detected
• Make sure manual migration logic is idempotent
• Update prep command
• Fix chop so it works when there are 3 epochs starting with 0i0
• Reproduce and fix partially-deleted epoch 0i0 after chop
• Pair with someone to run manual GDB testing for migration idempotency and rollover logic
• Take a look at @joemfb's replay code and compare/find overlaps
• Document final system design in this PR
• Correct epoch naming scheme
• Make chop leave the latest two epochs
• Better error handling
• Better cleanup
• Test migration with real ships running on local-networking mode
• Test epoch rollover idempotency
• Test fresh boot
• Handle case where snapshot has been deleted from chk/
• Ensure u3_disk_epoc_good() is implemented and used how we want
• Ensure u3_disk_epoc_init() is implemented and used how we want

[matthew-levan]

Midnight update: I can now boot a fresh fakezod with the epoch format, roll it, and start it up again. Events appear to be processing normally in the dojo from that state, but only the snapshot (not the database) seems to be getting updated. Thus, when shutting down again and trying to reboot, I get a:

...
boot: installed 652 jets
pier: eve_d 146, dun_d 133
Assertion 'god_u->eve_d <= log_u->dun_d' failed in pkg/vere/pier.c:1401
Assertion failed: god_u->eve_d <= log_u->dun_d (pkg/vere/pier.c: _pier_on_lord_live: 1401)

Thoughts welcome, just taking notes here for my future self.

[matthew-levan]

That makes sense, thank you sir.

…

On Sat, Apr 15, 2023 at 11:05 AM Joe Bryan ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In pkg/vere/disk.c <#346 (comment)>: > - if ( c3n == u3_lmdb_gulf(log_u->mdb_u, &fir_d, &log_u->dun_d) ) { - fprintf(stderr, "disk: failed to load latest event from database\r\n"); - c3_free(log_u); - return 0; + if ( 0 == lat_d ) { + c3_d fir_d; + if ( c3n == u3_lmdb_gulf(log_u->mdb_u, &fir_d, &log_u->dun_d) ) { + log_u->dun_d = 0; + } + log_u->sen_d = log_u->dun_d; + } else { + log_u->dun_d = lat_d; + log_u->sen_d = log_u->dun_d; Yeah, they're both event numbers. "commits in flight" would be sen_d - dun_d. — Reply to this email directly, view it on GitHub <#346 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AV2DQRBIDXWQMKTEJIHIZ6LXBK2MDANCNFSM6AAAAAAW4B7T3Y> . You are receiving this because you were assigned.Message ID: ***@***.***>

[matthew-levan]

Another update: got roll working nicely. chop is next on the.. chopping block. prep too. Finally will come the migration code. Stay tuned!

[jalehman]

Thanks for the update! This is exciting. — ~wolref-podlex https://urbit.org ( https://urbit.org/ )

…

On Mon, Apr 17 2023 at 2:15 AM, Matthew LeVan < ***@***.*** > wrote: Another update: got roll working nicely. chop is next on the.. chopping block. prep too. Finally will come the migration code. Stay tuned! — Reply to this email directly, view it on GitHub ( #346 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAEGQQUTMFDFXZEBNS73GA3XBUC2XANCNFSM6AAAAAAW4B7T3Y ). You are receiving this because you are subscribed to this thread. Message ID: <urbit/vere/pull/346/c1510989304 @ github. com>

[matthew-levan]

chop was easy. It's probably not perfect, but it works... Onto prep, then onto the migration code.

[matthew-levan]

Implemented a very minimal migration routine for existing piers-- still need to validate there's a current snapshot, rollover to a new epoch, and delete the backup snapshot (in that order).

_cw_roll is currently just a private function in main.c-- maybe it becomes u3_disk_epoc_roll in disk.c so I can use it in disk.c too. Hm...

[barter-simsum]

u3_disk_epoc_init and several other functions in disk.c should have brace on newline rather than same line

[matthew-levan]

Yes, will do. Thank you, good eye.

…

On Thu, Jun 1, 2023 at 2:23 PM barter-simsum ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In pkg/vere/disk.c <#346 (comment)>: > + // create hard links to data.mdb and lock.mdb in 0i0/ + c3_c dut_c[8193], luk_c[8193]; // old paths + snprintf(dut_c, sizeof(dut_c), "%s/data.mdb", log_u->com_u->pax_c); + snprintf(luk_c, sizeof(luk_c), "%s/lock.mdb", log_u->com_u->pax_c); i.e. at the beginning of this function, dat_c and lok_c store the *old* paths and later store the new paths when dut_c and luk_c store the old paths. Just always store the old paths in dut_c and luk_c. Swapping them is confusing/unnecessary — Reply to this email directly, view it on GitHub <#346 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AV2DQREZV4DV5YRSUS3TV2LXJDM27ANCNFSM6AAAAAAW4B7T3Y> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[barter-simsum]

pushed minor change in f772262. dut_o is pointless, just exit the migration early if data.mdb can't be accessed

[matthew-levan]

Successfully ran a migration on ~mastyr-bottec using local-networking mode:

matthew@domus:~/src/urbit/vere$ ./urbit --local ~/ships/planets/mastyr-bottec/mastyr-bottec/
~
urbit 2.9-c045119fd3
boot: home is /home/matthew/ships/planets/mastyr-bottec/mastyr-bottec
loom: mapped 2048MB
boot: protected loom
live: mapped: GB/1.345.880.064
live: loaded: KB/16.384
boot: installed 657 jets
loom: mapped 2048MB
lite: arvo formula 2a2274c9
lite: core 4bb376f0
lite: final state 4bb376f0
loom: image backup complete
disk: migrated disk to v1 format
disk: loaded epoch 0i219237052
loom: mapped 2048MB
boot: protected loom
live: mapped: GB/1.345.880.064
live: loaded: KB/16.384
boot: installed 657 jets
vere: checking version compatibility
%unlagging
ames: live on 51723 (localhost only)
conn: listening on /home/matthew/ships/planets/mastyr-bottec/mastyr-bottec/.urb/conn.sock
http: web interface live on http://localhost:8080
http: loopback live on http://localhost:12321
pier (219237063): live
~mastyr-bottec:dojo> 

Note that I also had a chop directory in my .urb/log folder-- didn't bother the migration. I'd tested that before with a fakezod too, but nice to see the same behavior on my own planet's pier as well.

[matthew-levan]

Closed in favor of #459, which implements the same and also integrates with urbit play a la #420.