Add in-place store migrations #8485

amaury1093 · 2021-02-01T15:01:25Z

Description

This PR adds infrastructure to perform module-specific in-place store migrations. It also adds migration logic for the following modules:

x/bank
(the other modules are migrated in x{stake,slash,gov,distrib} In-place Store Migrations #8504 as to make this base PR easier to review)

In-place migrations follow the design outlined in #8429, namely:

each module now has its own ConsensusVersion. All consensus-breaking changes in module should increment this field.
modules, when introducing breaking changes, must register their migration functions from version N to N+1 inside the Configurator.
ModuleManager has a RunMigrations method, which loops through all modules and calls their registered migrations incrementally. RunMigrations is meant to be called from within an UpgradeHandler.

ref: #8345

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

Targeted PR against correct branch (see CONTRIBUTING.md)
Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
Code follows the module structure standards.
Wrote unit and integration tests
Updated relevant documentation (docs/) or specification (x/<module>/spec/)
Added relevant godoc comments.
Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
Re-reviewed Files changed in the Github PR explorer
Review Codecov Report in the comment section below once CI passes

simapp/app.go

types/module/module.go

…345-migration

codecov · 2021-02-02T17:42:47Z

Codecov Report

Merging #8485 (5add13a) into master (816306b) will decrease coverage by 0.09%.
The diff coverage is 21.17%.

@@            Coverage Diff             @@
##           master    #8485      +/-   ##
==========================================
- Coverage   61.50%   61.40%   -0.10%     
==========================================
  Files         651      656       +5     
  Lines       37253    37336      +83     
==========================================
+ Hits        22911    22927      +16     
- Misses      11940    12007      +67     
  Partials     2402     2402

Impacted Files	Coverage Δ
testutil/context.go	`0.00% <0.00%> (ø)`
types/module/module.go	`82.71% <0.00%> (-9.07%)`	⬇️
x/auth/module.go	`52.77% <0.00%> (-1.51%)`	⬇️
x/auth/vesting/module.go	`52.17% <0.00%> (-2.38%)`	⬇️
x/authz/module.go	`48.83% <0.00%> (-1.17%)`	⬇️
x/bank/keeper/migrations.go	`0.00% <0.00%> (ø)`
x/bank/legacy/v040/keys.go	`0.00% <0.00%> (ø)`
x/bank/module.go	`51.16% <0.00%> (-3.84%)`	⬇️
x/crisis/module.go	`32.43% <0.00%> (-0.91%)`	⬇️
x/distribution/module.go	`54.76% <0.00%> (-1.34%)`	⬇️
... and 19 more

types/module/configurator.go

amaury1093 · 2021-02-03T14:26:15Z

I'm marking this as R4R. Only x/bank is migrated here, serving as PoC to the ConsensusVersion/RunMigrations change. The other modules introduce a big line diff (+2k), so they are done in #8504.

amaury1093 · 2021-02-08T14:28:04Z

R4R again.

Let's create an ADR based on this work. I created a task for it: #8532

I'm happy to take care of that, but I'm expecting 2weeks+ before it gets merged. Given that we want to get 0.42 out of the door quite fast, are we okay to merge this PR first?

aaronc · 2021-02-08T14:54:37Z

R4R again.

Let's create an ADR based on this work. I created a task for it: #8532

I'm happy to take care of that, but I'm expecting 2weeks+ before it gets merged. Given that we want to get 0.42 out of the door quite fast, are we okay to merge this PR first?

I am yes.

robert-zaremba · 2021-02-08T15:38:39Z

R4R again.

Let's create an ADR based on this work. I created a task for it: #8532

I'm happy to take care of that, but I'm expecting 2weeks+ before it gets merged. Given that we want to get 0.42 out of the door quite fast, are we okay to merge this PR first?

👍 sure, it's just for having it on track.

robert-zaremba

Pre approving: let's get the #8485 (comment) (migrations starting from 1) done.

amaury1093 · 2021-02-08T15:56:54Z

let's get the #8485 (comment) (migrations starting from 1) done.

It's actually done already!

robert-zaremba · 2021-02-08T20:26:03Z

@alessio , @marbar3778 - We were discussing internally (Regen) about this issue.
We decided to not touch ed25519 in the scope of this task, and propose to leave SDK ed25519 only for handling Tendermint validators. I will open a discussion to talk more about this issue.

aaronc

I'd like to keep this API clean. Let's discuss if any of this approach isn't clear.

aaronc · 2021-02-09T00:03:38Z

types/module/module.go

@@ -328,6 +341,30 @@ func (m *Manager) ExportGenesis(ctx sdk.Context, cdc codec.JSONMarshaler) map[st
 	return genesisData
 }

+// MigrationHandler is the migration function that each module registers.
+type MigrationHandler func(store sdk.Context, storeKey sdk.StoreKey, cdc codec.Marshaler) error


Suggested change

type MigrationHandler func(store sdk.Context, storeKey sdk.StoreKey, cdc codec.Marshaler) error

type MigrationHandler func(store sdk.Context) error

All this stuff is not needed. Let's keep the AP simple

aaronc · 2021-02-09T00:04:10Z

types/module/configurator.go

+	// RegisterMigration registers an in-place store migration for a module. The
+	// handler is a migration script to perform in-place migrations from version
+	// `forVersion` to version `forVersion+1`.
+	RegisterMigration(moduleName string, forVersion uint64, handler MigrationHandler) error
 }


Suggested change

}

Marshaler() codec.Marshaler

}

aaronc · 2021-02-09T00:06:12Z

x/bank/module.go

@@ -100,6 +101,7 @@ type AppModule struct {
 func (am AppModule) RegisterServices(cfg module.Configurator) {
 	types.RegisterMsgServer(cfg.MsgServer(), keeper.NewMsgServerImpl(am.keeper))
 	types.RegisterQueryServer(cfg.QueryServer(), am.keeper)
+	cfg.RegisterMigration(types.ModuleName, 0, v042.MigrateStore)


The store key and codec are in the keeper already so they can be passed from the keeper to the migration handler closure not from the module manager into the handler.

And even if the keeper didn't have this stuff already we can add it to Configurator so that all services have it rather than separately to each migration handler. Does that make sense? I think it will be much cleaner this way. Otherwise every time we need some new bit of configuration stuff then we keep adding more params to migration handler. Instead we can keep the migration handler params minimal and pass around other static configuration higher up.

The store key and codec are in the keeper already so they can be passed from the keeper to the migration handler closure not from the module manager into the handler.

Re store key: it is in the keeper yes, but not accessible by the AppModule at this line (cannot do am.keeper.storeKey).

Happy to discuss it today (or feel free to push to this PR if it's faster).

Yeah, so I think the right way is to make the migration a keeper method.

Basically:

cfg.RegisterMigration(types.ModuleName, 0, func(ctx sdk.Context) error { return am.keeper.Migrate0(ctx) })

Ah, ok, yeah that works, but it also pollutes the keeper. In bank, the Keeper is an interface, so we'd need to put it as a public method there, and it'd also be exposed to other modules.

I also like the fact that migration scripts are in each module's legacy folder.

I prefer the current approach, but if you think the keeper's way is cleaner, I'm okay to change it. (cc @robert-zaremba, maybe you have an opinion, since you reviewed this PR?)

In ADR 033, we include both StoreKey and Marshaler on Configurator and we could explore that step. An alternative to polluting the keeper is just to put the storeKey and codec in AppModule when it gets created. Or we can just make a MigrationKeeper interface and cast Keeper to that and we still keep the scripts in legacy but import them from the keeper... But I really am pretty strongly opposed to making these parameters on the migration handler, it's just not the right place.

Or we can just make a MigrationKeeper interface and cast Keeper to that and we still keep the scripts in legacy but import them from the keeper

I went with this in 41e0339. I feel strongly about having those scripts in legacy, because I am copy/pasting legacy code into those folders.

Yeah, I agree they should be in legacy.

What do you think about having StoreKey and Marshaler on Configurator?

One big thing I'm trying to pay attention to with APIs is not trying to pass too many things with function params. What ends up happening is that people say, oh we need just one more parameter and then the function signature changes. It's much less breaking to add fields to structs and methods to interfaces.

This is worth keeping in mind: https://blog.golang.org/module-compatibility

What do you think about having StoreKey and Marshaler on Configurator?

Having Marshaler on Configurator is useless imo (for migrations), because most (all?) AppModuleBasics already have a cdc.

Would StoreKey on Configurator look like func StoreKey(moduleName string) sdk.StoreKey? How do you avoid moduleA doing in its AppModule: b:=configurator.StoreKey("moduleB")?

aaronc

Mostly LGTM. Left a bunch of comments mostly about improving docs.

simapp/app.go

types/module/module.go

types/module/configurator.go

x/auth/module.go

aaronc

ACK 🎉

types/module/configurator.go

odeke-em · 2021-02-10T23:23:03Z

x/bank/legacy/v040/keys.go

+// store. The key must not contain the perfix BalancesPrefix as the prefix store
+// iterator discards the actual prefix.
+func AddressFromBalancesStore(key []byte) sdk.AccAddress {
+	addr := key[:v040auth.AddrLen]


These 2 checks are unnecessary and the first one is bound to panic if I pass in a key less than v040auth.AddrLen. Please remove this one, or move if after ensure that there are at least v040auth.AddrLen values.

All this code (in the legacy/ folder) is copy-pasted from a previous commit of the SDK. To avoid breaking the migration scripts inadvertently (they are hard to debug), I prefer to keep them as is.

Accessing key slices before checking key length also makes me feel uncomfortable, I created #8451 for the current code.

Thank you @amaurymartiny!

* Add 1st version of migrate * Put migration logic into Configurator * add test to bank store migration * add test for configurator * Error if no migration found * Remove RunMigrations from Configurator interface * Update spec * Rename folders * copy-paste from keys.go * Fix nil map * rename function * Update simapp/app.go Co-authored-by: Robert Zaremba <robert@zaremba.ch> * Update simapp/app_test.go Co-authored-by: Robert Zaremba <robert@zaremba.ch> * Adderss reviews * Fix tests * Update testutil/context.go Co-authored-by: Robert Zaremba <robert@zaremba.ch> * Update docs for ConsensusVersion * Rename to forVersion * Fix tests * Check error early * Return 1 for intiial version * Use MigrationKeeper * Fix test * Revert adding marshaler to Configurator * Godoc updates * Update docs Co-authored-by: Robert Zaremba <robert@zaremba.ch> Co-authored-by: Aaron Craelius <aaron@regen.network>

amaury1093 added the C:Store label Feb 1, 2021

Add 1st version of migrate

10d72d7

amaury1093 force-pushed the am-8345-migration branch from 028c098 to 10d72d7 Compare February 1, 2021 15:18

amaury1093 commented Feb 1, 2021

View reviewed changes

simapp/app.go Outdated Show resolved Hide resolved

aaronc reviewed Feb 1, 2021

View reviewed changes

types/module/module.go Outdated Show resolved Hide resolved

amaury1093 added 5 commits February 2, 2021 12:24

Merge branch 'master' of ssh://github.com/cosmos/cosmos-sdk into am-8…

8036fca

…345-migration

Put migration logic into Configurator

a9a406e

add test to bank store migration

9ed3987

add test for configurator

a095d3a

Merge branch 'master' into am-8345-migration

d550bff

aaronc reviewed Feb 2, 2021

View reviewed changes

types/module/configurator.go Outdated Show resolved Hide resolved

aaronc reviewed Feb 2, 2021

View reviewed changes

types/module/configurator.go Outdated Show resolved Hide resolved

types/module/configurator.go Outdated Show resolved Hide resolved

amaury1093 added 4 commits February 3, 2021 11:31

Error if no migration found

b7141f9

Remove RunMigrations from Configurator interface

2d554ce

Update spec

3df3f44

Rename folders

ba6bd44

amaury1093 added the C:x/bank label Feb 3, 2021

amaury1093 commented Feb 3, 2021

View reviewed changes

types/module/configurator.go Outdated Show resolved Hide resolved

This was referenced Feb 3, 2021

x{stake,slash,gov,distrib} In-place Store Migrations #8504

Merged

Refactor store keys to allow variable-length addresses #8345

Closed

Merge branch 'master' into am-8345-migration

197b7ce

amaury1093 marked this pull request as ready for review February 3, 2021 14:26

amaury1093 requested review from alessio and alexanderbez as code owners February 3, 2021 14:26

amaury1093 requested review from aleem1314, anilcse, atheeshp and blushi February 3, 2021 14:26

amaury1093 added 2 commits February 8, 2021 15:19

Check error early

8f3b9d4

Return 1 for intiial version

29b85b7

robert-zaremba approved these changes Feb 8, 2021

View reviewed changes

amaury1093 requested review from fdymylja, jgimeno and sahith-narahari February 8, 2021 19:56

robert-zaremba mentioned this pull request Feb 8, 2021

Draft a brief for auditors describing focus areas of base-app #8078

Closed

4 tasks

aaronc requested changes Feb 9, 2021

View reviewed changes

amaury1093 added 3 commits February 10, 2021 16:21

Use MigrationKeeper

41e0339

Fix test

d922003

Revert adding marshaler to Configurator

f422b93

aaronc reviewed Feb 10, 2021

View reviewed changes

amaury1093 added 2 commits February 10, 2021 16:58

Godoc updates

8cf88fe

Update docs

5add13a

amaury1093 requested a review from aaronc February 10, 2021 16:59

aaronc approved these changes Feb 10, 2021

View reviewed changes

Merge branch 'master' into am-8345-migration

2cef291

amaury1093 added the A:automerge Automatically merge PR once all prerequisites pass. label Feb 10, 2021

mergify bot merged commit dfc8dd8 into master Feb 10, 2021

mergify bot deleted the am-8345-migration branch February 10, 2021 17:49

odeke-em reviewed Feb 10, 2021

View reviewed changes

types/module/configurator.go Show resolved Hide resolved

odeke-em reviewed Feb 10, 2021

View reviewed changes

ethanfrey mentioned this pull request Mar 16, 2021

Can't perform software upgrade #8885

Closed

4 tasks

clevinson mentioned this pull request Mar 16, 2021

Add in-place store migration for TrackDelegation bug #8896

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add in-place store migrations #8485

Add in-place store migrations #8485

amaury1093 commented Feb 1, 2021 •

edited

codecov bot commented Feb 2, 2021 •

edited

amaury1093 commented Feb 3, 2021 •

edited

amaury1093 commented Feb 8, 2021

aaronc commented Feb 8, 2021

robert-zaremba commented Feb 8, 2021

robert-zaremba left a comment •

edited

amaury1093 commented Feb 8, 2021

robert-zaremba commented Feb 8, 2021

aaronc left a comment

aaronc Feb 9, 2021

aaronc Feb 9, 2021

aaronc Feb 9, 2021

aaronc Feb 9, 2021

amaury1093 Feb 9, 2021

aaronc Feb 9, 2021

amaury1093 Feb 9, 2021

aaronc Feb 9, 2021 •

edited

amaury1093 Feb 10, 2021

aaronc Feb 10, 2021

aaronc Feb 10, 2021

amaury1093 Feb 10, 2021

aaronc left a comment

aaronc left a comment

odeke-em Feb 10, 2021

amaury1093 Feb 11, 2021 •

edited

odeke-em Feb 11, 2021

	type MigrationHandler func(store sdk.Context, storeKey sdk.StoreKey, cdc codec.Marshaler) error
	type MigrationHandler func(store sdk.Context) error

Add in-place store migrations #8485

Add in-place store migrations #8485

Conversation

amaury1093 commented Feb 1, 2021 • edited

Description

codecov bot commented Feb 2, 2021 • edited

Codecov Report

amaury1093 commented Feb 3, 2021 • edited

amaury1093 commented Feb 8, 2021

aaronc commented Feb 8, 2021

robert-zaremba commented Feb 8, 2021

robert-zaremba left a comment • edited

Choose a reason for hiding this comment

amaury1093 commented Feb 8, 2021

robert-zaremba commented Feb 8, 2021

aaronc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronc Feb 9, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronc left a comment

Choose a reason for hiding this comment

aaronc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amaury1093 Feb 11, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amaury1093 commented Feb 1, 2021 •

edited

codecov bot commented Feb 2, 2021 •

edited

amaury1093 commented Feb 3, 2021 •

edited

robert-zaremba left a comment •

edited

aaronc Feb 9, 2021 •

edited

amaury1093 Feb 11, 2021 •

edited