Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move PieceStore To Map Encodings #415

Merged
merged 3 commits into from Sep 29, 2020

Conversation

hannahhoward
Copy link
Collaborator

@hannahhoward hannahhoward commented Sep 25, 2020

Goals

Switch to map encoding for piecestore, safely without breaking existing data

Implementation

This is the first of several introductions of data store migrations.

It's important to review https://github.com/filecoin-project/go-ds-versioning (unfortunately, not yet well documented)
go-ds-versioning provides various tools:

  • You can define datastore migrations via an individual record transformation function -- i.e. func(oldRecord oldStruct) (newStruct, error)
  • You can setup migrations that have an "old version" and a "new version"
  • You can setup versioned datastores, statestores, and fsm groups -- these have their normal parameters, plus a list of migrations, and a target version -- go-ds-versioning figure out how best to migrate to the target version based on the current version (versions are namespaces wrapped under a key, and there is code to migrate an unversioned store to a versioned store). When you setup datastores this way, you have to run a migrate function -- until you do, all of the functions provided by the interface return errors (this will allow us to let Lotus startup while leaving the markets in a disabled state until migrations complete)

The changes here to the piece store are as follows:

  • Switch to map-encoding for the current piecestore types
  • Add a migrations package that holds all the old types (actually, they're identical, just tuple encoded), and define functions to transform CIDInfo and PieceInfo (the top level ones that are the records of the database)
  • Define migration lists for both stores
  • Move the piecestore implementation to a seperate impl package to resolve import cycles
  • Modify the piecestore constructor to setup versioned stores
  • Add a Start function to piecestore to run migrations on stores
  • Add an OnReady function to listen for when the piecestore is done migrating (used only in tests so far -- probably this will just be a log in Lotus)
  • Update tests infrastructure elsewhere

@codecov-commenter
Copy link

codecov-commenter commented Sep 25, 2020

Codecov Report

Merging #415 into master will decrease coverage by 0.04%.
The diff coverage is 60.92%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #415      +/-   ##
==========================================
- Coverage   61.54%   61.51%   -0.03%     
==========================================
  Files          44       44              
  Lines        2722     2751      +29     
==========================================
+ Hits         1675     1692      +17     
- Misses        882      888       +6     
- Partials      165      171       +6     
Impacted Files Coverage Δ
piecestore/impl/piecestore.go 60.92% <60.92%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b1a8f19...6ceb959. Read the comment docs.

@hannahhoward hannahhoward force-pushed the feat/switch-to-cbor-map-piecestore branch from 352dc7b to 6ceb959 Compare September 29, 2020 10:00
Copy link
Contributor

@ingar ingar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tuple to map encoding and migration. 👍

@hannahhoward hannahhoward merged commit cd677ed into master Sep 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants