updating to new datastore/blockstore code with contexts #7646

whyrusleeping · 2021-11-19T01:51:31Z

Status

The following deps need to be tagged, as we have go mod versions pointing at unstable things:

github.com/drand/drand

The following deps also exist in the lotus-soup go.mod:

github.com/drand/drand

go.mod

vyzo · 2021-11-30T14:02:23Z

i'll work on finishing this as we need it to upgrade go-libp2p.

vyzo · 2021-11-30T14:07:04Z

the go-data-transfer dep is broken:

go: github.com/filecoin-project/go-data-transfer@v1.11.7-0.20211119001436-c0dbfa5fae4d requires
	github.com/ipfs/go-graphsync@v0.10.6-0.20211119000532-c416dad3bd56: invalid version: unknown revision c416dad3bd56

vyzo · 2021-11-30T14:07:39Z

@aarshkshah1992 @dirkmc is the go-data-transfer stuff done?

jennijuju · 2021-12-01T17:57:56Z

this is getting blocked by
filecoin-project/go-data-transfer#283

which is blocked by
filecoin-project/go-ds-versioning#3

which is blocked by a release tag of
https://github.com/filecoin-project/go-statestore

arajasek · 2021-12-07T22:28:51Z

@vyzo go-data-transfer thing should be fixed now

vyzo · 2021-12-09T12:15:42Z

broken go-ipfs-blockstore:

go: github.com/ipfs/go-ipfs-blockstore@v1.1.0 requires
	github.com/ipfs/go-datastore@v0.4.7-0.20211013204805-28a3721c2e66: invalid version: unknown revision 28a3721c2e66

vyzo · 2021-12-09T12:18:41Z

v1.1.1 also broken:

go get github.com/ipfs/go-ipfs-blockstore@v1.1.1
go: github.com/ipfs/go-ipfs-blockstore@v1.1.0 requires
	github.com/ipfs/go-datastore@v0.4.7-0.20211013204805-28a3721c2e66: invalid version: unknown revision 28a3721c2e66

vyzo · 2021-12-09T12:28:51Z

This occurs when trying to upgrade go-data-transfer:

$ go get github.com/filecoin-project/go-data-transfer@master

go: github.com/filecoin-project/go-data-transfer@v1.11.7-0.20211207053937-e06a599f202a requires
	github.com/ipfs/go-ipfs-blockstore@v1.1.0 requires
	github.com/ipfs/go-datastore@v0.4.7-0.20211013204805-28a3721c2e66: invalid version: unknown revision 28a3721c2e66

vyzo · 2021-12-09T13:11:23Z

seems it was a transient go mod cache error of some sorts.

vyzo · 2021-12-09T13:15:21Z

rebased and fixed deps; now the 1M places to add contexts...

vyzo · 2021-12-09T15:25:42Z

Blocker in go-fil-markets:

# github.com/filecoin-project/go-fil-markets/shared
../../../../pkg/mod/github.com/filecoin-project/go-fil-markets@v1.13.4/shared/movekey.go:9:20: not enough arguments in call to ds.Has
	have (datastore.Key)
	want (context.Context, datastore.Key)

vyzo · 2021-12-09T15:26:59Z

Blocker in go-car:

# github.com/ipld/go-car/v2/blockstore
../../../../pkg/mod/github.com/ipld/go-car/v2@v2.1.0/blockstore/readonly.go:24:5: cannot use (*ReadOnly)(nil) (type *ReadOnly) as type blockstore.Blockstore in assignment:
	*ReadOnly does not implement blockstore.Blockstore (wrong type for DeleteBlock method)
		have DeleteBlock(cid.Cid) error
		want DeleteBlock(context.Context, cid.Cid) error

vyzo · 2021-12-09T15:28:36Z

Blocker in go-ds-versioning

# github.com/filecoin-project/go-ds-versioning/internal/migrate
../../../../pkg/mod/github.com/filecoin-project/go-ds-versioning@v0.1.0/internal/migrate/migrate.go:24:26: not enough arguments in call to oldDs.Query
	have (query.Query)
	want (context.Context, query.Query)

vyzo · 2021-12-09T15:29:14Z

blocker in go-storedcounter:

# github.com/filecoin-project/go-storedcounter
../../../../pkg/mod/github.com/filecoin-project/go-storedcounter@v0.0.0-20200421200003-1c99c62e8a5b/storedcounter.go:28:23: not enough arguments in call to sc.ds.Has
	have (datastore.Key)
	want (context.Context, datastore.Key)

vyzo · 2021-12-09T15:30:46Z

blocker in drand:

# github.com/drand/drand/lp2p
../../../../pkg/mod/github.com/drand/drand@v1.2.1/lp2p/ctor.go:92:22: too many arguments in call to libp2p.New
	have (context.Context, ...config.Option)
	want (...config.Option)

arajasek · 2021-12-10T03:19:21Z

go-storedcounter should be unblocked

arajasek · 2021-12-15T23:41:08Z

This is now ready to be reviewed and merged.

magik6k

First pass, shouldn't be hard to land.

Was the ffi update intentional?

blockstore/badger/blockstore_test.go

blockstore/badger/blockstore_test_suite.go

chain/store/messages.go

chain/store/snapshot.go

cmd/lotus-bench/import.go

node/modules/genesis.go

node/modules/services.go

node/modules/storageminer.go

magik6k · 2021-12-16T13:36:54Z

node/repo/imports/manager.go

@@ -107,6 +108,7 @@ type Meta struct {
 // CreateImport initializes a new import, returning its ID and optionally a
 // CAR path where to place the data, if requested.
 func (m *Manager) CreateImport() (id ID, err error) {
+	ctx := context.TODO()


Probably open a tracking issue about this (in markets?)

Not sure what it should say and where it should be; what's the problem here?

IIRC This Manager struct implements some interface in markets, it would be nice to have contexts plumbed, but I don't think it needs to happen right now - so we could just open an issue saying 'add contexts to this stuff'

opened #7813.

vyzo · 2021-12-16T16:19:54Z

probably accident, that was not me.

magik6k

Just one nitpick.

Also skimming through the dependency diff quickly (it's.. 30k lines)

blockstore/badger/blockstore_test_suite.go

magik6k · 2021-12-17T11:20:16Z

(also need to merge master into here to sort out conflicts)

… needs more testing before landing remove ctx datastore from autobatch testplans go mod tidy fix ctx

… needs more testing before landing

whyrusleeping requested a review from a team as a code owner November 19, 2021 01:51

jennijuju marked this pull request as draft November 19, 2021 03:31

jennijuju added the blocked label Nov 19, 2021

aschmahmann reviewed Nov 29, 2021

View reviewed changes

go.mod Outdated Show resolved Hide resolved

arajasek force-pushed the deps/update-ctx-dsbs branch from 3498145 to c0fa25c Compare November 29, 2021 23:24

vyzo self-assigned this Nov 30, 2021

jennijuju removed the blocked label Dec 8, 2021

jennijuju added this to the v1.13.3 milestone Dec 8, 2021

whyrusleeping and others added 2 commits December 9, 2021 14:12

WIP: updating to new datastore/blockstore code with contexts

072297e

Update go-data-transfer

b9b4a26

update deps

a9e22df

vyzo force-pushed the deps/update-ctx-dsbs branch from 0c4f9fd to a9e22df Compare December 9, 2021 13:15

arajasek force-pushed the deps/update-ctx-dsbs branch from a9e22df to 6ba7481 Compare December 10, 2021 01:01

Update go-storedcounter and go-ds-versioning

276eabd

Merge branch 'master' into deps/update-ctx-dsbs

0e2278c

arajasek marked this pull request as ready for review December 15, 2021 23:33

Deps: Update drand to 1.3.0

62de84d

frrist mentioned this pull request Dec 16, 2021

improve tracing data and include lotus tracing filecoin-project/lily#794

Merged

magik6k reviewed Dec 16, 2021

View reviewed changes

vyzo and others added 4 commits December 17, 2021 11:42

plumb more contexts

dd327f0

fix lotus-shed

6ba2533

fix tests

cd266a0

don't update ffi

122b319

magik6k reviewed Dec 17, 2021

View reviewed changes

blockstore/badger/blockstore_test_suite.go Outdated Show resolved Hide resolved

fix blockstore test contexts

976a3a5

vyzo mentioned this pull request Dec 17, 2021

Plumb context into market manager interface #7813

Closed

Merge remote-tracking branch 'origin/master' into deps/update-ctx-dsbs

bc384c0

magik6k force-pushed the deps/update-ctx-dsbs branch from 63be845 to bc384c0 Compare December 17, 2021 12:01

fix shed

66f8f8a

magik6k approved these changes Dec 17, 2021

View reviewed changes

magik6k enabled auto-merge December 17, 2021 12:18

dirkmc mentioned this pull request Dec 17, 2021

fix: #7577 markets: When retrying Add Piece, first seek to start of reader #7812

Merged

2 tasks

magik6k disabled auto-merge December 17, 2021 12:50

magik6k merged commit 6806bff into master Dec 17, 2021

magik6k deleted the deps/update-ctx-dsbs branch December 17, 2021 12:50

BigLep mentioned this pull request Dec 19, 2021

Bubble up changes to ipfs/go-datastore #7536

Closed

BigLep linked an issue Dec 19, 2021 that may be closed by this pull request

Bubble up changes to ipfs/go-datastore #7536

Closed

jennijuju modified the milestones: v1.13.3, v1.15.0 Jan 2, 2022

jennijuju added a commit that referenced this pull request Jan 14, 2022

revert #7646 from v1.14.0 - this is not concensus breaking change and…

3f6c235

… needs more testing before landing remove ctx datastore from autobatch testplans go mod tidy fix ctx

jennijuju added a commit that referenced this pull request Jan 15, 2022

revert #7646 from v1.14.0 - this is not concensus breaking change and…

ca444bb

… needs more testing before landing

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

updating to new datastore/blockstore code with contexts #7646

updating to new datastore/blockstore code with contexts #7646

whyrusleeping commented Nov 19, 2021 •

edited by arajasek

vyzo commented Nov 30, 2021

vyzo commented Nov 30, 2021

vyzo commented Nov 30, 2021

jennijuju commented Dec 1, 2021 •

edited

arajasek commented Dec 7, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

arajasek commented Dec 10, 2021

arajasek commented Dec 15, 2021

magik6k left a comment

magik6k Dec 16, 2021

vyzo Dec 17, 2021

magik6k Dec 17, 2021

vyzo Dec 17, 2021

vyzo commented Dec 16, 2021

magik6k left a comment

magik6k commented Dec 17, 2021

updating to new datastore/blockstore code with contexts #7646

updating to new datastore/blockstore code with contexts #7646

Conversation

whyrusleeping commented Nov 19, 2021 • edited by arajasek

Status

vyzo commented Nov 30, 2021

vyzo commented Nov 30, 2021

vyzo commented Nov 30, 2021

jennijuju commented Dec 1, 2021 • edited

arajasek commented Dec 7, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

vyzo commented Dec 9, 2021

arajasek commented Dec 10, 2021

arajasek commented Dec 15, 2021

magik6k left a comment

Choose a reason for hiding this comment

magik6k Dec 16, 2021

Choose a reason for hiding this comment

vyzo Dec 17, 2021

Choose a reason for hiding this comment

magik6k Dec 17, 2021

Choose a reason for hiding this comment

vyzo Dec 17, 2021

Choose a reason for hiding this comment

vyzo commented Dec 16, 2021

magik6k left a comment

Choose a reason for hiding this comment

magik6k commented Dec 17, 2021

whyrusleeping commented Nov 19, 2021 •

edited by arajasek

jennijuju commented Dec 1, 2021 •

edited