perf: Faster state export #14332

Reecepbcups · 2022-12-15T22:57:58Z

Description

This PR uses better logic to export the genesis state export. This has a real world performance increase of ~43% for a given export (CPU: AMD Ryzen 5 3600 6-Core & Disk: SAMSUNG MZVL2512HCJQ-00B00)

# Command
time junod export --height 6086660 > state_export.json

# With this PR
real    4m45.816s
user    10m34.517s
sys     2m1.134s

# Standard
real    10m52.126s
user    7m7.416s
sys     1m14.892s

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed ! in the type prefix if API or client breaking change
confirmed all author checklist items have been addressed
reviewed state machine logic
reviewed API design and naming
reviewed documentation is accurate
reviewed tests and test coverage
manually tested (if applicable)

types/module/module.go

julienrbrt

lgtm!

types/module/module.go

Co-authored-by: Julien Robert <julien@rbrt.fr>

alexanderbez

LGTM! Thanks :)

jackzampolin · 2022-12-16T15:48:10Z

types/module/module.go

 	}

 	// verify modules exists in app, so that we don't panic in the middle of an export
 	if err := m.checkModulesExists(modulesToExport); err != nil {
 		panic(err)
 	}

-	for _, moduleName := range modulesToExport {
+	channels := make([]chan json.RawMessage, len(modulesToExport))


One note here: this will create as many goroutines as modules. This may cause CPU contention. Limiting concurrent exports to GOMAXPROCS may improve perf slightly. This is a great PR tho and a huge improvement. Should be able to do something similar for migrations.

good point. but unless I'm mistaken, or things changed, Go is built to execute and handle hundreds of thousands of goroutines, whereas a typical app will have a dozen or so modules at most.

Alex is correct here, go 1.18+ can handle ~500k routines per 1GB of memory. So we shouldn't have any issues here. I will mess with this more in the future to see if I can squeeze out more runtime performance.

I will also take a look at doing it for migrations too, thanks for the tip

Not all modules may implement HasGenesis. The setup can take this into account (and use append later)

Suggested change

channels := make([]chan json.RawMessage, len(modulesToExport))

channels := make([]chan json.RawMessage, 0, len(modulesToExport))

julienrbrt · 2022-12-16T16:00:42Z

We need to check that data race in e2e tests before merging.
cc @Reecepbcups

types/module/module.go

+			go func(module HasGenesis, ch chan json.RawMessage) {
+				ch <- module.ExportGenesis(ctx, cdc)
+			}(module, channels[i])


tests/Makefile

julienrbrt · 2022-12-19T15:42:30Z

Note to self: add a changelog in the backported PR.

tests/Makefile

(cherry picked from commit de6ef1e)

Co-authored-by: Reece Williams <31943163+Reecepbcups@users.noreply.github.com> Co-authored-by: Julien Robert <julien@rbrt.fr>

alpe · 2022-12-22T11:25:12Z

types/module/module.go

 		}
 	}

+	for i, moduleName := range modulesWithGenesis {
+		genesisData[moduleName] = <-channels[i]


You are iterating over modulesWithGenesis now which may be a subset of modulesToExport. The i value therefore is not the same.
For example if we have [foo, bar] as modules in modulesToExport with foo not implementing HasGenesis then modulesWithGenesis has only the bar element and you would still wait for channel[0] to return.
Some tests on this feature would be very useful

alpe · 2022-12-22T11:27:48Z

types/module/module.go

+			channels[i] = make(chan json.RawMessage)
+			modulesWithGenesis = append(modulesWithGenesis, moduleName)
+
+			go func(module HasGenesis, ch chan json.RawMessage) {


This introduces a race condition on github.com/cosmos/cosmos-sdk/store/types.(*infiniteGasMeter).ConsumeGas(). CI on wasmd found this.

thanks for identifying this, agree we need better tests. @Reecepbcups can you help here?

alpe · 2022-12-22T11:38:38Z

tests/e2e/server/export_test.go

-//go:build e2e
-// +build e2e
+//go:build !race
+// +build !race


Why deactivate?

alpe · 2022-12-22T11:42:50Z

types/module/module.go

+			modulesWithGenesis = append(modulesWithGenesis, moduleName)
+
+			go func(module HasGenesis, ch chan json.RawMessage) {
+				ch <- module.ExportGenesis(ctx, cdc)


To fix the gas meter race, you could set a new one per Go-routine, like ctx.WithGasMeter(sdk.NewInfiniteGasMeter()) .

This reverts commit de6ef1e.

This reverts commit 58896f1.

(cherry picked from commit 6dfa0c9)

go routine for exporting genesis modules

a29e3c5

Reecepbcups requested a review from a team as a code owner December 15, 2022 22:57

github-advanced-security bot found potential problems Dec 15, 2022

View reviewed changes

types/module/module.go Fixed Show fixed Hide fixed

julienrbrt reviewed Dec 16, 2022

View reviewed changes

types/module/module.go Outdated Show resolved Hide resolved

julienrbrt reviewed Dec 16, 2022

View reviewed changes

types/module/module.go Outdated Show resolved Hide resolved

julienrbrt added the backport/v0.47.x PR scheduled for inclusion in the v0.47's next stable release label Dec 16, 2022

julienrbrt approved these changes Dec 16, 2022

View reviewed changes

julienrbrt reviewed Dec 16, 2022

View reviewed changes

types/module/module.go Outdated Show resolved Hide resolved

Reecepbcups and others added 5 commits December 16, 2022 09:14

Update types/module/module.go

b69c231

Co-authored-by: Julien Robert <julien@rbrt.fr>

Update types/module/module.go

0ede0ad

Co-authored-by: Julien Robert <julien@rbrt.fr>

init slice with set length

fb382a6

Co-authored-by: Julien Robert <julien@rbrt.fr>

add index to for loop

dd9ba1b

Merge branch 'main' into reece/faster-export

a76614d

alexanderbez approved these changes Dec 16, 2022

View reviewed changes

julienrbrt enabled auto-merge (squash) December 16, 2022 15:45

jackzampolin reviewed Dec 16, 2022

View reviewed changes

julienrbrt disabled auto-merge December 16, 2022 15:48

julienrbrt added the A:automerge Automatically merge PR once all prerequisites pass. label Dec 16, 2022

julienrbrt enabled auto-merge (squash) December 16, 2022 15:49

julienrbrt disabled auto-merge December 16, 2022 15:58

julienrbrt removed the A:automerge Automatically merge PR once all prerequisites pass. label Dec 16, 2022

github-advanced-security bot found potential problems Dec 16, 2022

View reviewed changes

types/module/module.go Fixed Show fixed Hide fixed

Reecepbcups added 2 commits December 17, 2022 16:48

only receive if module has genesis

b908cd2

ignore race for e2e export test only

e352bf3

github-advanced-security bot found potential problems Dec 17, 2022

View reviewed changes

types/module/module.go

Comment on lines +404 to +406

go func(module HasGenesis, ch chan json.RawMessage) {

ch <- module.ExportGenesis(ctx, cdc)

}(module, channels[i])

Check notice

Code scanning / CodeQL

Spawning a Go routine

Spawning a Go routine may be a possible source of non-determinism

Reecepbcups and others added 2 commits December 17, 2022 20:23

Merge branch 'main' into reece/faster-export

5f16585

Merge branch 'main' into reece/faster-export

ffcf5c4

alexanderbez enabled auto-merge (squash) December 19, 2022 15:37

julienrbrt reviewed Dec 19, 2022

View reviewed changes

tests/Makefile Outdated Show resolved Hide resolved

add test server incov tests

37ba2e5

julienrbrt reviewed Dec 19, 2022

View reviewed changes

tests/Makefile Outdated Show resolved Hide resolved

add coverage for sonarcloud

1abdfde

julienrbrt assigned alexanderbez and julienrbrt Dec 19, 2022

alexanderbez merged commit de6ef1e into cosmos:main Dec 19, 2022

mergify bot pushed a commit that referenced this pull request Dec 19, 2022

perf: Faster state export (#14332)

ae62e33

(cherry picked from commit de6ef1e)

mergify bot mentioned this pull request Dec 19, 2022

perf: Faster state export (backport #14332) #14355

Merged

julienrbrt added a commit that referenced this pull request Dec 19, 2022

perf: Faster state export (backport #14332) (#14355)

58896f1

Co-authored-by: Reece Williams <31943163+Reecepbcups@users.noreply.github.com> Co-authored-by: Julien Robert <julien@rbrt.fr>

alpe reviewed Dec 22, 2022

View reviewed changes

tests/e2e/server/export_test.go

//go:build e2e

// +build e2e

//go:build !race

// +build !race

Copy link

Contributor

alpe Dec 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why deactivate?

alpe reviewed Dec 22, 2022

View reviewed changes

tac0turtle added a commit that referenced this pull request Dec 22, 2022

Revert "perf: Faster state export (#14332)"

7541e16

This reverts commit de6ef1e.

This was referenced Dec 22, 2022

Revert "perf: Faster state export" #14387

Closed

genesis: add back faster exports #14389

Closed

tac0turtle added a commit that referenced this pull request Dec 22, 2022

Revert "perf: Faster state export (backport #14332) (#14355)"

b6e4853

This reverts commit 58896f1.

julienrbrt added a commit that referenced this pull request Dec 23, 2022

fix: fix race conditions from #14332

6c2bc3f

julienrbrt mentioned this pull request Dec 23, 2022

fix: fix race conditions from #14332 #14402

Merged

19 tasks

julienrbrt added a commit that referenced this pull request Dec 26, 2022

fix: fix race conditions from #14332 (#14402)

6dfa0c9

mergify bot pushed a commit that referenced this pull request Dec 26, 2022

fix: fix race conditions from #14332 (#14402)

a64ff51

(cherry picked from commit 6dfa0c9)

tac0turtle pushed a commit that referenced this pull request Dec 26, 2022

fix: fix race conditions from #14332 (backport #14402) (#14421)

c30a33a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: Faster state export #14332

perf: Faster state export #14332

Reecepbcups commented Dec 15, 2022 •

edited

julienrbrt left a comment

alexanderbez left a comment

jackzampolin Dec 16, 2022

alexanderbez Dec 16, 2022

Reecepbcups Dec 17, 2022

alpe Dec 22, 2022

julienrbrt commented Dec 16, 2022 •

edited

julienrbrt commented Dec 19, 2022

alpe Dec 22, 2022

alpe Dec 22, 2022

tac0turtle Dec 22, 2022

alpe Dec 22, 2022

alpe Dec 22, 2022

	channels := make([]chan json.RawMessage, len(modulesToExport))
	channels := make([]chan json.RawMessage, 0, len(modulesToExport))

perf: Faster state export #14332

perf: Faster state export #14332

Conversation

Reecepbcups commented Dec 15, 2022 • edited

Description

Author Checklist

Reviewers Checklist

julienrbrt left a comment

Choose a reason for hiding this comment

alexanderbez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

julienrbrt commented Dec 16, 2022 • edited

julienrbrt commented Dec 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Reecepbcups commented Dec 15, 2022 •

edited

julienrbrt commented Dec 16, 2022 •

edited