Skip to content
This repository has been archived by the owner on Dec 5, 2019. It is now read-only.

remove kvstore #108

Merged
merged 10 commits into from
Aug 6, 2019
Merged

remove kvstore #108

merged 10 commits into from
Aug 6, 2019

Conversation

warner
Copy link
Member

@warner warner commented Aug 6, 2019

This removes the 'kvstore' persistence mechanism, replacing it with a simple JS object that is mutated by the various Keepers. At the end of the turn, you can call controller.getState() to retrieve a stringified form of this structure (note: not deterministically serialized).

In #94 we found that updating the transcript gets really slow, because 1: it grows forever, 2: it goes through a deserialize-append-reserialize cycle for each message delivery, and 3: the reserialization uses a deterministic JSON stringify that spends too much of its time in JS instead of native code. This fix removes 2 and 3, giving us just one (native, non-deterministic) serialization call per "big turn" (i.e. one block of messages, i.e. one checkpoint).

The persistent-kernel apps that use SwingSet will have to change when this fix is released. Instead of creating an externalStorage object and passing it into buildVatController, they should pass a string as config.initialState, and then call controller.getState() after each big turn (which returns a string).

@warner
Copy link
Member Author

warner commented Aug 6, 2019

In the old code, after adding 20 batches of 100 trivial calls, the execution time of c.run() had grown to 1100ms. In the new code, c.run() takes less than 1ms even after 100 such batches.

In the new code, c.getState() takes 2ms after 20 batches, and is up to 12ms after 100 batches. It looks roughly linear in the size of the transcript. I think that'll give us some more breathing room, but we'll still need the engine-level checkpoints/snapshots before this will be ready for production.

@katelynsills take a look at what I changed in the keepers and see what you think:

  • the core state object is a plain JS object
  • the Maps were replaced with more plain JS objects, so the JSON.stringify() call at the end of the big turn can be as fast as possible (doing everything in native engine code, rather than needing to translate Map keys/values into object properties first)
  • I changed the way Promise records are manipulated by e.g. kernel-side promise-resolution code. I don't know if this is keeper-ish for your tastes.. the fact that the backing store is just a plain object might leak through the API in places, and maybe we should do it differently

Copy link
Contributor

@katelynsills katelynsills left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me!

@warner warner merged commit 1801d45 into Agoric:master Aug 6, 2019
@warner warner deleted the 94-remove-kvstore branch August 6, 2019 17:44
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants