Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cosmos: Provide ability to separate storage of snapshots #61

Open
bartelink opened this issue Dec 8, 2018 · 1 comment
Open

Cosmos: Provide ability to separate storage of snapshots #61

bartelink opened this issue Dec 8, 2018 · 1 comment
Labels
enhancement up-for-grabs https://up-for-grabs.net

Comments

@bartelink
Copy link
Collaborator

The unfolds mechanism, as documented in #50 provides the ability to efficiently stash relevant data facilitating reading of state based on a point read.

However, this is can be suboptimal in the cases of larger snapshot states:-

  • the data gets compressed and travels each time
  • writing/updating occasions read and write costs within the sync stored procedure each time
  • competing writers pay all but the update case in the case of a re-run of the same transaction
  • readers not interested in/unable to process a specific snapshot schema still have to pay the price to transport it
  • each update goes to the changefeed triggering more read costs
  • the size and cost of the snapshots is harder to separate out
  • snapshots contribute to partition size limits

Thus it makes sense to do some, or all of the following in the case where there is a projector in play and there is an aux collection (in the case of multiple writers one might even consider making an aux collection even if you don't have a projector):

  • maintain the snapshots in the aux collection, where one is not paying for changefeed re-reads of the same data
  • reduce the write frequency for snapshots (when the write happens should be deterministic for competing writers)
  • implement ability for stored proc to maintain events since snapshot in tip when not updating snapshot
  • write to a nominated collection (typically aux), with a deterministic guid based on the case / event type tag of the union as a point write
  • refer to the snapshot in the unfolds; automate loading of the referenced snapshot if picked by isOrigin
  • expose the ability to customize the snapshot loading process (but batteries should be included)
    • (perhaps do a sample showing parallel reads of the Snapshots and the Tip?)
@bartelink
Copy link
Collaborator Author

The designs for #145 and #157 should consider the potential overlap with this requirement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement up-for-grabs https://up-for-grabs.net
Projects
None yet
Development

No branches or pull requests

1 participant