Shard large types #12
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ts in fewer total elements
…ng deltas to shards
…which had different sharding configurations for one or more types.
…tomatically calculated based on a target excess memory pool size, which defaults to 25MB
… to indicate automatic calculation
…VALUE to disable type sharding by default. This is necessary to ensure backwards compatibility with pre v2.1.0 consumers.
ghost
approved these changes
Jan 27, 2017
Sunjeet
pushed a commit
that referenced
this pull request
Apr 10, 2019
…ster * commit '0fd2b894db703885817168c756f1f3101be1fc6c': Add missing PrimaryKeyEntity File
Sunjeet
pushed a commit
that referenced
this pull request
Jul 22, 2019
… to master
Squashed commit of the following:
commit 638f64b1cb295e0889d46bab69b2d1a8a46a88df
Author: Paul Sandoz <psandoz@netflix.com>
Date: Tue Jul 9 14:42:26 2019 -0700
Refine log level values.
commit ab0cdacfa331d204e1ea6bafee1d22dc4e122491
Author: Paul Sandoz <psandoz@netflix.com>
Date: Mon Jul 8 09:25:30 2019 -0700
Refactor command options.
- Lower case environment options and remove "local_" prefix.
- Replace general verbose option with logging and levels.
- Move port from general options to explore/history/diff.
- Change graph output to format.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What was the problem?
Currently, during a delta transition each individual record type builds the entire next state of the type in memory while the current state is retained to answer requests. This means that the memory pool Hollow reserves needs to be large enough to compose an extra copy of the largest type. This is a miniaturized version of the double snapshot problem.
What is the solution?
With this change, large types can be transparently sharded into smaller individual chunks, each of which can update independently. The effect of this change is that Hollow only needs to draw on the memory pool enough to update the largest chunk across all types.