feat: postage stamp indexing #1625

zelig · 2021-04-29T22:31:41Z

this PR extends postage stamps with indexes

indexes enable replacing stored chunks at the discretion of the owner: storer nodes store only the latest chunk stamped with the same stamp (same batch same index)
offer an easy way to catch overissuance as part of stamp validity check
makes it possible to limit spoofing density proofs by mining chunks to a batch
makes it possible to transfer ownership of arbitrary indexes

the index is a 64-bit serialisation of

bucket depth (uint8, 1st byte, <24)
bucket index (neighbourhood index, uint32 <2^depth, bytes 2-4)
and the within-bucket index (uint32 <2^(batchdepth-bucketdepth), bytes 5-8)

This change is

pkg/localstore/mode_put.go

pkg/postage/stampissuer.go

IxaBrjnko · 2021-06-01T20:34:30Z

Yay!! Thank you <3

pkg/localstore/mode_put.go

pkg/localstore/export.go

pkg/api/postage_test.go

pkg/localstore/mode_put.go

ralph-pichler · 2021-06-03T06:42:42Z

rebase on master, then change https://github.com/ethersphere/bee/blob/master/.github/workflows/beekeeper.yml#L16 to 0.2.0 to run integration tests with the new contract.

istae · 2021-06-03T12:05:18Z

rebase on master, then change https://github.com/ethersphere/bee/blob/master/.github/workflows/beekeeper.yml#L16 to 0.2.0 to run integration tests with the new contract.

Ok! Thanks.

acud

Reviewed 17 of 33 files at r1, 1 of 2 files at r2, 1 of 10 files at r4, 15 of 19 files at r7, 12 of 12 files at r8, 1 of 1 files at r9.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @esadakar and @zelig)

pkg/localstore/gc.go, line 184 at r9 (raw file):

			return 0, false, err
		}
		err = db.postageIndexIndex.DeleteInBatch(batch, item)

need to add a test for the index, to see that the entry gets removed when GCing

pkg/localstore/localstore.go, line 511 at r9 (raw file):

		},
		DecodeValue: func(keyItem shed.Item, value []byte) (e shed.Item, err error) {
			e = keyItem

this should not be here, there are other occurrences of this usage in this file and they should also be removed

pkg/localstore/mode_put_test.go, line 494 at r9 (raw file):

	ts := time.Now().Unix()

	modes := []storage.ModePut{storage.ModePutRequest, storage.ModePutRequestPin, storage.ModePutSync, storage.ModePutUpload, storage.ModePutUploadPin}

nit, this can be pulled out to a package-level variable

pkg/localstore/mode_set_test.go, line 113 at r9 (raw file):

				for _, ch := range chs {
					wantErr := leveldb.ErrNotFound
					_, err := db.retrievalDataIndex.Get(addressToItem(ch.Address()))

this is not needed since index counts are tested below

pkg/localstore/mode_set_test.go, line 119 at r9 (raw file):

					// access index should not be set
					_, err = db.retrievalAccessIndex.Get(addressToItem(ch.Address()))

this is not needed since index counts are tested below

pkg/localstore/mode_set_test.go, line 131 at r9 (raw file):

			for _, ch := range chs {
				newPullIndexTest(db, ch, 0, leveldb.ErrNotFound)(t)

this is not needed since we test this anyway below

pkg/postage/stamp.go, line 71 at r9 (raw file):

// MarshalBinary gives the byte slice serialisation of a stamp:
// batchID[32]|index[32]|SignatureUser[65]|SignatureOwner[65].

comment incorrect

pkg/postage/stamper.go, line 48 at r9 (raw file):

	toSign, err := toSignDigest(addr.Bytes(), st.issuer.batchID, index, ts)
	if err != nil {
		return nil, err

if there was an error here, we need to think about decrementing the issuer count

pkg/postage/listener/listener_test.go, line 41 at r9 (raw file):

			amount:           big.NewInt(42),
			normalisedAmount: big.NewInt(43),
			// bucketDepth:      10,

commented code can be removed

istae

Reviewed 1 of 33 files at r1, 3 of 10 files at r3.
Reviewable status: 41 of 47 files reviewed, 9 unresolved discussions (waiting on @acud and @zelig)

istae

Reviewed 6 of 6 files at r10.
Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @acud and @zelig)

pkg/localstore/localstore.go, line 511 at r9 (raw file):