Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement simple replay command #192

Merged
merged 3 commits into from
Feb 6, 2023
Merged

Implement simple replay command #192

merged 3 commits into from
Feb 6, 2023

Conversation

mcevoypeter
Copy link

@mcevoypeter mcevoypeter commented Feb 1, 2023 •
edited
Loading

Description

Resolves #157. As described in #157 (comment), this PR adds support for a play subcommand, which piggybacks off the existing replay implementation. This PR also augments the existing replay implementation with per-batch snapshots and a command line configurable batch size., which makes replaying large event logs a less painful experience for users. Snapshotting after each replay batch of course incurs a performance hit because we're writing to disk more frequently during replay, but replay is already slow, and in practice, users should not expect to see any difference. Additionally, this PR adds command line configurable replay batch size gives users a knob for increasing or decreasing the frequency with which snapshots are taken.

Testing

$ bazel build :urbit
<snip>
$ ./urbit -F zod -u https://bootstrap.urbit.org/urbit-v1.17.pill
<snip>
$ rm zod/.urb/chk/*.bin
$ rm zod/.urb/chk/*.bin
$ ./urbit --replay-to 103 zod
~
urbit 1.20-c98600a5ed
boot: home is /home/tlon/code/gh/urbit/vere/zod
loom: mapped 2048MB
lite: arvo formula 2a2274c9
lite: core 4bb376f0
lite: final state 4bb376f0
loom: mapped 2048MB
boot: protected loom
live: logical boot
boot: installed 652 jets
pier: replay till 103
---------------- playback starting ----------------
pier: replaying events 1-103
<snip>
pier: (103): play: done
---------------- playback complete ----------------
pier: replay barrier reached, cramming
serf (103): saving rock
hash-cons arena:
  root: B/160
  atoms (6939):
    refs: KB/218.920
    data: MB/16.597.260
    dict: KB/206.720
  total: MB/17.022.900
  cells (2598252):
    refs: MB/70.491.560
    dict: MB/174.264.720
  total: MB/244.756.280
total: MB/261.779.340

%group-store: on-peek on path /x/whey
clay: read-at-aeon fail [desk=%base care=%c case=[%da p=~2023.2.1..17.47.34..2cb5] path=/noun/mass]
clay: no files match /mar/mass/hoon
[%error-building-cast %noun %mass]
[%error-building-tube %noun %mass]
peek no tube from noun to mass
serf: measuring memory:

%arvo:
  %hoon:
    %one: KB/29.160
    %two: KB/188.064
    %tri: KB/280.708
    %qua: KB/683.716
    %pen: MB/3.092.604
  --MB/4.274.252
  %hex: KB/562.264
  %pit: B/168
  %lull:
    %dot: MB/1.885.888
    %typ: B/144
  --MB/1.886.032
  %zuse:
    %dot: KB/903.100
    %typ: KB/638.604
  --MB/1.541.704
  %vane:
    %ames:
      %peers-known: KB/9.304
      %peers-alien: B/0
      %dot: MB/3.105.952
      %typ: KB/9.120
      %sac: KB/94.632
    --MB/3.219.008
    %behn:
      %timers: KB/2.988
      %dot: KB/272.284
      %typ: B/840
      %sac: KB/19.848
    --KB/295.960
    %clay:
      %object-store:
        %commits: KB/187.020
        %pages: MB/5.428.184
      --MB/5.615.204
      %domestic:
        %base:
          %mime: B/0
          %flue: MB/17.895.872
          %dojo: KB/1.396
        --MB/17.897.268
        %garden:
          %mime: B/0
          %flue: MB/2.602.580
          %dojo: KB/1.376
        --MB/2.603.956
        %groups:
          %mime: B/0
          %flue: MB/11.052.948
          %dojo: KB/29.144
        --MB/11.082.092
        %kids:
          %mime: B/0
          %flue: KB/17.352
          %dojo: B/720
        --KB/18.072
        %landscape:
          %mime: B/0
          %flue: MB/11.413.300
          %dojo: KB/69.352
        --MB/11.482.652
        %talk:
          %mime: B/0
          %flue: KB/6.816
          %dojo: KB/4.856
        --KB/11.672
        %webterm:
          %mime: B/0
          %flue: KB/3.240
          %dojo: KB/1.280
        --KB/4.520
      --MB/43.100.232
      %foreign: KB/2.592
      %ford-cache: KB/149.016
      %dot: MB/1.521.144
      %typ: KB/1.008
      %sac: KB/127.368
    --MB/50.516.564
    %dill:
      %hey: B/24
      %dug: B/72
      %dot: KB/448.140
      %typ: KB/4.080
      %sac: KB/36.720
    --KB/489.036
    %eyre:
      %bindings: KB/1.732
      %auth: B/0
      %connections: B/0
      %channels: B/0
      %axle: B/404
      %dot: MB/1.943.376
      %typ: KB/2.064
      %sac: KB/29.280
    --MB/1.976.856
    %gall:
      %foreign: B/24
      %blocked:
        %file-server: B/432
      --B/432
      %active:
        %acme: B/900
        %azimuth: B/828
        %chat: KB/1.260
        %chat-cli: KB/1.476
        %chat-hook: B/900
        %chat-store: B/564
        %chat-view: B/564
        %clock: B/828
        %contact-hook: B/564
        %contact-pull-hook: B/996
        %contact-push-hook: B/240
        %contact-store: KB/1.236
        %contact-view: B/564
        %dbug: B/804
        %diary: B/828
        %dm-hook: B/660
        %docket: MB/11.182.220
        %dojo: KB/1.284
        %eth-watcher: B/804
        %graph-pull-hook: B/996
        %graph-push-hook: KB/1.368
        %graph-store:
          %live:
            %dm-inbox: B/48
          --B/48
          %logs:
            %dm-inbox: B/192
          --B/192
          %dot: KB/1.860
        --KB/2.100
        %group-pull-hook: B/996
        %group-push-hook: KB/1.164
        %group-store: KB/2.400
        %group-view: B/828
        %groups: KB/1.428
        %hark: KB/1.332
        %hark-chat-hook: B/564
        %hark-graph-hook: B/804
        %hark-group-hook: B/996
        %hark-invite-hook: B/996
        %hark-store: KB/2.472
        %hark-system-hook: B/828
        %heap: B/828
        %herm: B/828
        %hood: KB/2.520
        %invite-hook: B/660
        %invite-store: KB/1.652
        %invite-view: B/564
        %launch: KB/1.092
        %lens: B/660
        %metadata-hook: B/828
        %metadata-pull-hook: KB/1.404
        %metadata-push-hook: KB/1.308
        %metadata-store: KB/2.012
        %notify: B/948
        %observe-hook: KB/2.796
        %ping: B/732
        %s3-store: B/828
        %sane: B/828
        %settings-store: B/828
        %spider: KB/29.116
        %treaty: KB/1.188
        %weather: B/828
      --MB/11.269.240
      %dot: MB/1.748.544
      %typ: B/840
      %sac: KB/34.848
    --MB/13.053.928
    %iris:
      %nex: B/0
      %outbound: B/160
      %by-id: B/0
      %by-duct: B/0
      %axle: B/132
      %dot: KB/102.300
      %typ: KB/71.796
      %sac: KB/6.528
    --KB/180.916
    %jael:
      %pki: B/664
      %etn: B/0
      %dot: KB/898.208
      %typ: KB/2.520
      %sac: KB/15.720
    --KB/917.112
    %khan:
      %dot: KB/83.932
      %typ: KB/61.664
      %sac: KB/12.624
    --KB/158.220
  --MB/70.807.600
--MB/79.072.020
total userspace: MB/79.072.020
  kernel: KB/11.664
total arvo stuff: KB/11.664
  warm jet state: KB/67.092
  cold jet state: KB/55.188
  hank cache: B/456
  battery hash cache: B/288
  call site cache: B/200
  hot jet state: KB/191.788
total jet stuff: KB/315.012
  bytecode programs: KB/531.328
  bytecode cache: KB/45.860
total nock stuff: KB/577.188
  trace buffer: B/88
  memoization cache: B/288
total road stuff: B/376
space profile: KB/12.000
total marked: MB/79.988.260
free lists: KB/112.836
sweep: MB/79.988.260

pier: cram complete, shutting down
$ ./urbit --replay-to 111 --batch-size 2 zod
~
urbit 1.20-c98600a5ed
boot: home is zod
loom: mapped 2048MB
lite: arvo formula 2a2274c9
lite: core 4bb376f0
lite: final state 4bb376f0
loom: mapped 2048MB
boot: protected loom
live: loaded: MB/79.495.168
boot: installed 652 jets
pier: replay in 2-event batches
pier: replay till 111
---------------- playback starting ----------------
pier: replaying events 104-111
pier: (105): play: done
pier: (107): play: done
pier: (109): play: done
pier: (111): play: done
---------------- playback complete ----------------
pier: replay barrier reached, cramming
serf (111): saving rock
hash-cons arena:
  root: B/160
  atoms (6937):
    refs: KB/218.920
    data: MB/16.597.218
    dict: KB/206.720
  total: MB/17.022.858
  cells (2598268):
    refs: MB/70.491.560
    dict: MB/174.264.720
  total: MB/244.756.280
total: MB/261.779.298

%group-store: on-peek on path /x/whey
clay: read-at-aeon fail [desk=%base care=%c case=[%da p=~2023.2.1..17.47.37..8a00] path=/noun/mass]
clay: no files match /mar/mass/hoon
[%error-building-cast %noun %mass]
[%error-building-tube %noun %mass]
serf: measuring memory:to mass

%arvo:
  %hoon:
    %one: KB/29.160
    %two: KB/188.064
    %tri: KB/280.708
    %qua: KB/683.716
    %pen: MB/3.092.604
  --MB/4.274.252
  %hex: KB/562.264
  %pit: B/168
  %lull:
    %dot: MB/1.885.888
    %typ: B/144
  --MB/1.886.032
  %zuse:
    %dot: KB/903.100
    %typ: KB/638.604
  --MB/1.541.704
  %vane:
    %ames:
      %peers-known: KB/9.304
      %peers-alien: B/0
      %dot: MB/3.105.952
      %typ: KB/9.120
      %sac: KB/94.632
    --MB/3.219.008
    %behn:
      %timers: KB/2.556
      %dot: KB/272.308
      %typ: B/840
      %sac: KB/19.848
    --KB/295.552
    %clay:
      %object-store:
        %commits: KB/187.020
        %pages: MB/5.428.184
      --MB/5.615.204
      %domestic:
        %base:
          %mime: B/0
          %flue: MB/17.895.872
          %dojo: KB/1.396
        --MB/17.897.268
        %garden:
          %mime: B/0
          %flue: MB/2.602.580
          %dojo: KB/1.376
        --MB/2.603.956
        %groups:
          %mime: B/0
          %flue: MB/11.052.948
          %dojo: KB/29.144
        --MB/11.082.092
        %kids:
          %mime: B/0
          %flue: KB/17.352
          %dojo: B/720
        --KB/18.072
        %landscape:
          %mime: B/0
          %flue: MB/11.413.300
          %dojo: KB/69.352
        --MB/11.482.652
        %talk:
          %mime: B/0
          %flue: KB/6.816
          %dojo: KB/4.856
        --KB/11.672
        %webterm:
          %mime: B/0
          %flue: KB/3.240
          %dojo: KB/1.280
        --KB/4.520
      --MB/43.100.232
      %foreign: KB/2.592
      %ford-cache: KB/149.016
      %dot: MB/1.521.144
      %typ: KB/1.008
      %sac: KB/127.368
    --MB/50.516.564
    %dill:
      %hey: B/24
      %dug: B/72
      %dot: KB/448.140
      %typ: KB/4.080
      %sac: KB/36.720
    --KB/489.036
    %eyre:
      %bindings: KB/1.732
      %auth: B/0
      %connections: B/0
      %channels: B/0
      %axle: B/404
      %dot: MB/1.943.376
      %typ: KB/2.064
      %sac: KB/29.280
    --MB/1.976.856
    %gall:
      %foreign: B/24
      %blocked:
        %file-server: B/432
      --B/432
      %active:
        %acme: B/900
        %azimuth: B/828
        %chat: KB/1.260
        %chat-cli: KB/1.476
        %chat-hook: B/900
        %chat-store: B/564
        %chat-view: B/564
        %clock: B/828
        %contact-hook: B/564
        %contact-pull-hook: B/996
        %contact-push-hook: B/240
        %contact-store: KB/1.236
        %contact-view: B/564
        %dbug: B/804
        %diary: B/828
        %dm-hook: B/660
        %docket: MB/11.182.280
        %dojo: KB/1.284
        %eth-watcher: B/804
        %graph-pull-hook: B/996
        %graph-push-hook: KB/1.368
        %graph-store:
          %live:
            %dm-inbox: B/48
          --B/48
          %logs:
            %dm-inbox: B/192
          --B/192
          %dot: KB/1.860
        --KB/2.100
        %group-pull-hook: B/996
        %group-push-hook: KB/1.164
        %group-store: KB/2.400
        %group-view: B/828
        %groups: KB/1.428
        %hark: KB/1.332
        %hark-chat-hook: B/564
        %hark-graph-hook: B/804
        %hark-group-hook: B/996
        %hark-invite-hook: B/996
        %hark-store: KB/2.472
        %hark-system-hook: B/828
        %heap: B/828
        %herm: B/828
        %hood: KB/2.520
        %invite-hook: B/660
        %invite-store: KB/1.652
        %invite-view: B/564
        %launch: KB/1.092
        %lens: B/660
        %metadata-hook: B/828
        %metadata-pull-hook: KB/1.404
        %metadata-push-hook: KB/1.308
        %metadata-store: KB/2.012
        %notify: B/948
        %observe-hook: KB/2.796
        %ping: B/732
        %s3-store: B/828
        %sane: B/828
        %settings-store: B/828
        %spider: KB/29.428
        %treaty: KB/1.188
        %weather: B/828
      --MB/11.269.612
      %dot: MB/1.748.544
      %typ: B/840
      %sac: KB/34.848
    --MB/13.054.300
    %iris:
      %nex: B/0
      %outbound: B/160
      %by-id: B/288
      %by-duct: B/48
      %axle: B/132
      %dot: KB/102.300
      %typ: KB/71.796
      %sac: KB/6.528
    --KB/181.252
    %jael:
      %pki: B/664
      %etn: B/0
      %dot: KB/898.208
      %typ: KB/2.520
      %sac: KB/15.720
    --KB/917.112
    %khan:
      %dot: KB/83.932
      %typ: KB/61.664
      %sac: KB/12.624
    --KB/158.220
  --MB/70.807.900
--MB/79.072.320
total userspace: MB/79.072.320
  kernel: KB/11.664
total arvo stuff: KB/11.664
  warm jet state: KB/67.092
  cold jet state: KB/55.188
  hank cache: B/456
  battery hash cache: B/288
  call site cache: B/200
  hot jet state: KB/191.788
total jet stuff: KB/315.012
  bytecode programs: KB/531.332
  bytecode cache: KB/45.860
total nock stuff: KB/577.192
  trace buffer: B/88
  memoization cache: B/288
total road stuff: B/376
space profile: KB/12.000
total marked: MB/79.988.564
free lists: KB/112.836
sweep: MB/79.988.564

pier: cram complete, shutting down
$ cp zod/.urb/bhk/*.bin zod/.urb/chk/
$ ./urbit play --batch-size 7 zod
~
urbit 1.20-c98600a5ed
boot: home is zod
loom: mapped 2048MB
lite: arvo formula 2a2274c9
lite: core 4bb376f0
lite: final state 4bb376f0
loom: mapped 2048MB
boot: protected loom
live: loaded: MB/96.092.160
boot: installed 652 jets
pier: replay in 7-event batches
---------------- playback starting ----------------
pier: replaying events 16-136
pier: (22): play: done
pier: (29): play: done
pier: (36): play: done
pier: (43): play: done
docket: fetching %http glob for %garden desk
pier: (50): play: done
pier: (57): play: done
pier: (64): play: done
ames: metamorphosis
pier: (71): play: done
pier: (78): play: done
pier: (85): play: done
pier: (92): play: done
pier: (99): play: done
pier: (106): play: done
/«play»113): play: done
pier: (120): play: done
pier: (127): play: done
pier: (134): play: done
pier: (136): play: done
---------------- playback complete ----------------

@mcevoypeter mcevoypeter requested a review from a team as a code owner February 1, 2023 17:59
@mcevoypeter
Copy link
Author

cc @zalberico

zalberico
zalberico approved these changes Feb 1, 2023
View reviewed changes
Copy link
Collaborator

@zalberico zalberico left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a comment about the default batch size, but otherwise seems reasonable to me (and thanks for fixing the off by one bug (that drove me crazy).

@philipcmonk can you +2?

philipcmonk
philipcmonk approved these changes Feb 6, 2023
View reviewed changes
Copy link
Contributor

@philipcmonk philipcmonk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

500 for every snapshot seems quite low (i expect a lot of snapshot time during replay), but I don't know how many you can safely put over the IPC in a batch anyway. Worth doing this for now, and we'll tune as necessary

@mcevoypeter mcevoypeter merged commit 47ba262 into develop Feb 6, 2023
@mcevoypeter mcevoypeter deleted the i/187/snapshotting-replay branch February 6, 2023 21:00
@zalberico zalberico mentioned this pull request Feb 10, 2023
Closed
@ryjm ryjm mentioned this pull request May 30, 2023
Closed
joemfb added a commit that referenced this pull request May 30, 2023
@joemfb
Revert "Implement simple replay command (#192)"
243aec5 
This reverts commit 47ba262.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@philipcmonk philipcmonk philipcmonk approved these changes

@zalberico zalberico zalberico approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Restore event log replay (urbit play) functionality
3 participants
@mcevoypeter @zalberico @philipcmonk