Cadence 1.0 Multi-Staging by jribbink · Pull Request #1582 · onflow/flow-cli

jribbink · 2024-05-07T16:27:59Z

Closes #1559

Description

Adds support for staging multiple contracts at once.

Approach:

Encapsulate staging functionality in "Staging Service"
Staging service can stage multiple contracts at once
Staging validator adapted to allow multiple contracts, returns separate errors per contract
Local dependency tree of all local/remote checkers is built, everything is cached (contracts, checkers, etc). Checking remote contracts is very expensive, so caching is important. Additionally, it's important that all contracts are validated together, because this way staging order is irrelevant (e.g. validation is done using local copies).

For contributor use:

Targeted PR against master branch
Linked to Github issue with discussion and accepted design OR link to spec that describes this work
Code follows the standards mentioned here
Updated relevant documentation
Re-reviewed Files changed in the Github PR explorer
Added appropriate labels

codecov-commenter · 2024-05-08T01:31:57Z

Codecov Report

❌ Patch coverage is 59.18762% with 211 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/migrate/stage.go	21.52%	103 Missing and 10 partials ⚠️
internal/migrate/staging_validator.go	75.71%	42 Missing and 9 partials ⚠️
internal/migrate/staging_service.go	79.06%	15 Missing and 12 partials ⚠️
internal/cadence/lint.go	0.00%	10 Missing ⚠️
internal/migrate/get_staged_code.go	54.54%	6 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

bjartek · 2024-05-14T16:24:40Z

Just tried to run this on find repo since I have 45 things to stage there and i got

flow migrate stage-contract --all -n testnet
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                           ⚠ Upgrade to Cadence 1.0
     The Crescendo network upgrade, including Cadence 1.0, is coming soon.
     You may need to update your existing contracts to support this change.
                     Please visit our migration guide here:
             https://cadence-lang.org/docs/cadence_migration_guide
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
panic: interface conversion: common.Location is common.IdentifierLocation, not common.AddressLocation

goroutine 1 [running]:
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).parseAndCheckAllStaged.func1({0x104c13e20, 0x140025977a0})
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:266 +0x128
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).forEachDependency.func1({0x104c13de0, 0x140024d97e0})
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:701 +0xfc
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).forEachDependency.func1({0x104c13de0, 0x14002ac2720})
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:702 +0x114
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).forEachDependency(0x14000d477c0, {{{0x140014d0400, 0xb}, {0x3e, 0x5b, 0x4c, 0x62, 0x70, 0x64, 0x62, ...}}, ...}, ...)
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:706 +0xd0
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).parseAndCheckAllStaged(0x14000d477c0)
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:264 +0x1e0
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).Validate(0x14000d477c0, {0x14000238608, 0x2e, 0x1?})
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:178 +0x1a8
github.com/onflow/flow-cli/internal/migrate.(*stagingServiceImpl).validateAndStageContracts(0x14000d47800, {0x104c0eb60, 0x1061543e0}, {0x14000238608, 0x2e, 0x4b})
	github.com/onflow/flow-cli/internal/migrate/staging_service.go:101 +0x11c
github.com/onflow/flow-cli/internal/migrate.(*stagingServiceImpl).StageContracts(0x14000d47800, {0x104c0eb60, 0x1061543e0}, {0x14000c31a00?, 0x2e, 0x0?})
	github.com/onflow/flow-cli/internal/migrate/staging_service.go:92 +0x80
github.com/onflow/flow-cli/internal/migrate.stageAll({0x104c000f0, 0x14000d47800}, 0x14000fbcf30, {0x104c359d8?, 0x14000d171a0?})
	github.com/onflow/flow-cli/internal/migrate/stage_contract.go:135 +0xbc
github.com/onflow/flow-cli/internal/migrate.stageContract({0x14000d4b410, 0x0, 0x3}, {{0x0, 0x0}, {0x103ebd395, 0x4}, {0x0, 0x0}, {0x0, ...}, ...}, ...)
	github.com/onflow/flow-cli/internal/migrate/stage_contract.go:92 +0x3cc
github.com/onflow/flow-cli/internal/command.Command.AddToParent.func1(0x14000d51100?, {0x14000d4b410, 0x0, 0x3})
	github.com/onflow/flow-cli/internal/command/command.go:142 +0x4cc
github.com/spf13/cobra.(*Command).execute(0x105fbd4a0, {0x140001b8150, 0x3, 0x3})
	github.com/spf13/cobra@v1.8.0/command.go:987 +0x828
github.com/spf13/cobra.(*Command).ExecuteC(0x1400055d808)
	github.com/spf13/cobra@v1.8.0/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.8.0/command.go:1039
main.main()
	github.com/onflow/flow-cli/cmd/flow/main.go:168 +0xd9c

jribbink · 2024-05-14T17:00:43Z

Just tried to run this on find repo since I have 45 things to stage there and i got

flow migrate stage-contract --all -n testnet
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                           ⚠ Upgrade to Cadence 1.0
     The Crescendo network upgrade, including Cadence 1.0, is coming soon.
     You may need to update your existing contracts to support this change.
                     Please visit our migration guide here:
             https://cadence-lang.org/docs/cadence_migration_guide
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
panic: interface conversion: common.Location is common.IdentifierLocation, not common.AddressLocation

goroutine 1 [running]:
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).parseAndCheckAllStaged.func1({0x104c13e20, 0x140025977a0})
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:266 +0x128
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).forEachDependency.func1({0x104c13de0, 0x140024d97e0})
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:701 +0xfc
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).forEachDependency.func1({0x104c13de0, 0x14002ac2720})
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:702 +0x114
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).forEachDependency(0x14000d477c0, {{{0x140014d0400, 0xb}, {0x3e, 0x5b, 0x4c, 0x62, 0x70, 0x64, 0x62, ...}}, ...}, ...)
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:706 +0xd0
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).parseAndCheckAllStaged(0x14000d477c0)
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:264 +0x1e0
github.com/onflow/flow-cli/internal/migrate.(*stagingValidatorImpl).Validate(0x14000d477c0, {0x14000238608, 0x2e, 0x1?})
	github.com/onflow/flow-cli/internal/migrate/staging_validator.go:178 +0x1a8
github.com/onflow/flow-cli/internal/migrate.(*stagingServiceImpl).validateAndStageContracts(0x14000d47800, {0x104c0eb60, 0x1061543e0}, {0x14000238608, 0x2e, 0x4b})
	github.com/onflow/flow-cli/internal/migrate/staging_service.go:101 +0x11c
github.com/onflow/flow-cli/internal/migrate.(*stagingServiceImpl).StageContracts(0x14000d47800, {0x104c0eb60, 0x1061543e0}, {0x14000c31a00?, 0x2e, 0x0?})
	github.com/onflow/flow-cli/internal/migrate/staging_service.go:92 +0x80
github.com/onflow/flow-cli/internal/migrate.stageAll({0x104c000f0, 0x14000d47800}, 0x14000fbcf30, {0x104c359d8?, 0x14000d171a0?})
	github.com/onflow/flow-cli/internal/migrate/stage_contract.go:135 +0xbc
github.com/onflow/flow-cli/internal/migrate.stageContract({0x14000d4b410, 0x0, 0x3}, {{0x0, 0x0}, {0x103ebd395, 0x4}, {0x0, 0x0}, {0x0, ...}, ...}, ...)
	github.com/onflow/flow-cli/internal/migrate/stage_contract.go:92 +0x3cc
github.com/onflow/flow-cli/internal/command.Command.AddToParent.func1(0x14000d51100?, {0x14000d4b410, 0x0, 0x3})
	github.com/onflow/flow-cli/internal/command/command.go:142 +0x4cc
github.com/spf13/cobra.(*Command).execute(0x105fbd4a0, {0x140001b8150, 0x3, 0x3})
	github.com/spf13/cobra@v1.8.0/command.go:987 +0x828
github.com/spf13/cobra.(*Command).ExecuteC(0x1400055d808)
	github.com/spf13/cobra@v1.8.0/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.8.0/command.go:1039
main.main()
	github.com/onflow/flow-cli/cmd/flow/main.go:168 +0xd9c

Awesome, thank you so much for doing this. Looks like an easy fix, glad we could catch it before it became an issue 🙏

bjartek · 2024-05-14T17:19:11Z

this is the output now

 flow-ms migrate stage-contract --all -n testnet
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                           ⚠ Upgrade to Cadence 1.0
     The Crescendo network upgrade, including Cadence 1.0, is coming soon.
     You may need to update your existing contracts to support this change.
                     Please visit our migration guide here:
             https://cadence-lang.org/docs/cadence_migration_guide
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
❌ Preliminary validation could not be performed on the following contracts:
  - 35717efbbce11c74.FindMarketDirectOfferEscrow
  - 3e5b4c627064625d.PartyFavorz
  - 35717efbbce11c74.FindLeaseMarketAuctionSoft
  - 35717efbbce11c74.FindMarketAuctionSoft
  - 35717efbbce11c74.FindLeaseMarketSale
  - 35717efbbce11c74.Dandy
  - 35717efbbce11c74.FindForgeOrder
  - 35717efbbce11c74.FindFurnace
  - 35717efbbce11c74.FindMarketDirectOfferSoft
  - 35717efbbce11c74.FindForge
  - 35717efbbce11c74.FindMarketSale
  - 35717efbbce11c74.FindLeaseMarketDirectOfferSoft
  - 35717efbbce11c74.FindThoughts
  - 3e5b4c627064625d.NFGv3
  - 35717efbbce11c74.FindLeaseMarket
  - 35717efbbce11c74.FindVerifier
  - 35717efbbce11c74.FindMarketAuctionEscrow
  - 35717efbbce11c74.FindLostAndFoundWrapper
  - 35717efbbce11c74.FIND
  - 35717efbbce11c74.FindMarketAdmin
  - 35717efbbce11c74.Admin
  - 35717efbbce11c74.FindAirdropper
  - 35717efbbce11c74.NameVoucher
  - 3e5b4c627064625d.Flomies
  - 35717efbbce11c74.FindPack
  - 3e5b4c627064625d.GeneratedExperiences

These contracts depend on the following contracts which have not been staged yet:
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT

You may still stage your contract, however it will be unable to be migrated until the missing contracts are staged by their respective owners.  It is important to monitor the status of your contract using the `flow migrate is-validated` command

should we sort and unique the dependent contracts maybe?

jribbink · 2024-05-14T17:20:48Z

this is the output now

 flow-ms migrate stage-contract --all -n testnet
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
                           ⚠ Upgrade to Cadence 1.0
     The Crescendo network upgrade, including Cadence 1.0, is coming soon.
     You may need to update your existing contracts to support this change.
                     Please visit our migration guide here:
             https://cadence-lang.org/docs/cadence_migration_guide
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
❌ Preliminary validation could not be performed on the following contracts:
  - 35717efbbce11c74.FindMarketDirectOfferEscrow
  - 3e5b4c627064625d.PartyFavorz
  - 35717efbbce11c74.FindLeaseMarketAuctionSoft
  - 35717efbbce11c74.FindMarketAuctionSoft
  - 35717efbbce11c74.FindLeaseMarketSale
  - 35717efbbce11c74.Dandy
  - 35717efbbce11c74.FindForgeOrder
  - 35717efbbce11c74.FindFurnace
  - 35717efbbce11c74.FindMarketDirectOfferSoft
  - 35717efbbce11c74.FindForge
  - 35717efbbce11c74.FindMarketSale
  - 35717efbbce11c74.FindLeaseMarketDirectOfferSoft
  - 35717efbbce11c74.FindThoughts
  - 3e5b4c627064625d.NFGv3
  - 35717efbbce11c74.FindLeaseMarket
  - 35717efbbce11c74.FindVerifier
  - 35717efbbce11c74.FindMarketAuctionEscrow
  - 35717efbbce11c74.FindLostAndFoundWrapper
  - 35717efbbce11c74.FIND
  - 35717efbbce11c74.FindMarketAdmin
  - 35717efbbce11c74.Admin
  - 35717efbbce11c74.FindAirdropper
  - 35717efbbce11c74.NameVoucher
  - 3e5b4c627064625d.Flomies
  - 35717efbbce11c74.FindPack
  - 3e5b4c627064625d.GeneratedExperiences

These contracts depend on the following contracts which have not been staged yet:
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - a983fecbed621163.FiatToken
  - 0afe396ebc8eee65.FLOAT

You may still stage your contract, however it will be unable to be migrated until the missing contracts are staged by their respective owners.  It is important to monitor the status of your contract using the `flow migrate is-validated` command

should we sort and unique the dependent contracts maybe?

Yeah 💯 (I was also planning not to stage contracts if the staged code hasn't changed, just have not added yet)

bjartek · 2024-05-14T17:27:02Z

It did work though, but it stages each contract in a single transaction. Each tx takes about 300 gas so you should be able to do loads of them in one go if you want to.

The command took 5 minutes to run for me :p

jribbink · 2024-05-14T17:40:49Z

It did work though, but it stages each contract in a single transaction. Each tx takes about 300 gas so you should be able to do loads of them in one go if you want to.

The command took 5 minutes to run for me :p

Awesome!! Super appreciative that you could run this, it really helps because my manual + unit tests can only go so far... lots of edge cases in this one.

Yeah, doing this in separate transactions is somewhat intentional. I didn't want to run into any headaches related to computation limits, so it seemed more reliable to just stick to individual TX than try to estimate computation & potentially introduce bugs (and since this is what flow project deploy does as well anyways).

I know waiting 5 minutes for the command to run isn't a great experience, but hopefully I can really shave that down by not sending unnecessary TX.

bjartek · 2024-05-14T19:24:46Z

Fair enough. I dont expect to run it that many times

[WIP] Cadence 1.0 Multi-Staging

abdb4a4

jribbink changed the base branch from master to feature/stable-cadence May 7, 2024 16:28

jribbink added 7 commits May 7, 2024 14:07

cleanup

b57aa40

fixup

3c43bc7

report missing dependencies error

91cecfc

cleanup small issues

68f473d

stash

dbae112

fix tests

3bcfdb0

fix header

c48c080

jribbink added 20 commits May 7, 2024 21:42

better filtering

fef9bcd

cleanup tests

2f015e8

enhance validator tests

6d66ea9

testing overhaul & bug fixes

cc86157

enhance tests

1eb11fd

update tests

bad8568

Refactor & improve result printing

2bcad5f

UI improvements

0c76d38

fix nil deref bug

31d1f16

cleanup checker cache

aea7cdc

add txid

e5975cf

fix test regressions

38171a4

stash

8ed698b

refactor stagedContractUpdate

6052d40

Fix goimports

46eb618

fix import

6a07a84

fix goimport-ed

6ea9a59

fix windows tests

4e794c4

fix tests

5b7d8d7

update formatting

fd9bd6c

chasefleming mentioned this pull request May 11, 2024

[C1.0] failed to stage contract (flow-cli crash) onflow/cadence#3336

Closed

fix account access

5812e1b

jribbink commented May 13, 2024

View reviewed changes

Comment thread internal/migrate/staging_service.go Outdated

jribbink commented May 13, 2024

View reviewed changes

Comment thread internal/migrate/staging_service.go

jribbink commented May 13, 2024

View reviewed changes

Comment thread internal/migrate/staging_validator.go Outdated

Fix crypto checker import

f9f03ea

cleanup missing dependencies messaging

30c96d2

jribbink added 8 commits May 14, 2024 12:27

export fields for json

922530b

skip staging unchanged contracts

0acecef

add more assertions to tests

9f21a3e

fix deps

734f8cc

fix getstagedcode test

12ee3b4

address feedback

114c4b7

rename to checkAllStaged

3929d24

rename test file & add limit

e11a9d1

chasefleming reviewed May 14, 2024

View reviewed changes

Comment thread internal/migrate/staging_service.go

jribbink commented May 14, 2024

View reviewed changes

Comment thread internal/migrate/get_staged_code.go

chasefleming approved these changes May 15, 2024

View reviewed changes

jribbink added 3 commits May 15, 2024 10:44

change wording

7c0c2ae

Merge branch 'feature/stable-cadence' into jribbink/stage-all

f2d3513

fix flowkit args

718d898

jribbink merged commit 2f95230 into feature/stable-cadence May 15, 2024

jribbink deleted the jribbink/stage-all branch May 15, 2024 18:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Cadence 1.0 Multi-Staging#1582

Cadence 1.0 Multi-Staging#1582
jribbink merged 50 commits into
feature/stable-cadencefrom
jribbink/stage-all

jribbink commented May 7, 2024 •

edited

Loading

Uh oh!

codecov-commenter commented May 8, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bjartek commented May 14, 2024 •

edited

Loading

Uh oh!

jribbink commented May 14, 2024

Uh oh!

bjartek commented May 14, 2024 •

edited

Loading

Uh oh!

jribbink commented May 14, 2024 •

edited

Loading

Uh oh!

bjartek commented May 14, 2024 •

edited

Loading

Uh oh!

jribbink commented May 14, 2024

Uh oh!

bjartek commented May 14, 2024

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

jribbink commented May 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

codecov-commenter commented May 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bjartek commented May 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jribbink commented May 14, 2024

Uh oh!

bjartek commented May 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jribbink commented May 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bjartek commented May 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jribbink commented May 14, 2024

Uh oh!

bjartek commented May 14, 2024

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jribbink commented May 7, 2024 •

edited

Loading

codecov-commenter commented May 8, 2024 •

edited

Loading

bjartek commented May 14, 2024 •

edited

Loading

bjartek commented May 14, 2024 •

edited

Loading

jribbink commented May 14, 2024 •

edited

Loading

bjartek commented May 14, 2024 •

edited

Loading