tvx: a test vector extraction and execution tool #4064

raulk · 2020-09-27T19:30:43Z

This PR contributes tvx, a test vector extraction and execution tool.

The tvx binary supports two subcommands.

`tvx extract --cid <message id>`

Extracts a message-class test vector from a live chain/network, using the JSON-RPC API pointed at by the API info env variable (FULLNODE_API_INFO=token:multiaddr), or the --api flag.

It does the following:

Locates the message on chain (if the message is old, Lotus takes ages, so if you're testing this, use a recent message; Lotus needs better indexing).
Retrieves the inclusion and execution tipsets.
Finds any precursor messages in the same block (messages sent by the same actor with a lower nonce). If there are such messages, we can't take the previous tipset's state as the base state; instead we need to cherry-pick apply precursor messages first.
Applies the message on top of the base state, turning on blockstore tracing. This records the CIDs that were effectively accessed during the execution of the transaction.
Prunes the state tree encoded in the test vector's CAR by only retaining the effectively accessed CIDs and ignoring any others.
- This deliberately causes dangling links, but it is non harmful.
- If the actor code changed in a way that it started accessing more CIDs, the gas costs would change, which would invalidate the test vector anyway.
- So trimming down the state tree this way is a pretty safe bet.
- If it does break, we can always regenerate from chain history.
Writes out the test vector to a file (if -o <file> was specified), or to stdout.

tvx uses a special "proxying" blockstore, which fetches objects from a remote node via JSON-RPC ChainReadObj.

Try:

$ env FULLNODE_API_INFO="token:multiaddr" tvx extract --cid bafy2bzaced3gxdknd5zuxcglnyec3hwe4tom3nmunt5neaglybxurclgkkr6e -o vector.json

(^^ depending on when you run this, it can be extremely slow to locate the message, due to Lotus' lack of indices; substitute the CID with a fresher one).

`tvx exec --file <path>`

Executes a single vector against Lotus. It's super fast, so if you don't believe it, mangle the vector (e.g. change the postcondition state root, the gas cost in the receipt, the exit code, etc.) and notice how it fails.

Try:

$ tvx exec --file vector.json

These tools were incubated in the Oni repo (https://github.com/filecoin-project/oni/), and heavily refactored before submitting here. Thanks @willscott and @nonsense for past contributions!

cmd/tvx/main.go

willscott · 2020-09-27T19:42:35Z

cmd/tvx/state.go

+}
+
+// loadInitActor loads the init actor state from a given tipset.
+func (sg *StateSurgeon) loadInitActor(st *state.StateTree) (*types.Actor, init_.State, error) {


will this need to be interface'd to support versioning?

Maybe, but honestly I don't think we'll end up using the accessed-actors ever. I just don't want to throw it away too quickly in case it comes in handy, but it's very inefficient and ultimately useless.

cmd/tvx/stores.go

magik6k · 2020-09-28T07:33:04Z

cmd/tvx/main.go

+	}
+}
+
+func makeAPIClient() (api.FullNode, jsonrpc.ClientCloser, error) {


This should just use the methods from cli/:

Add a global repo flag to cli.App above:

&cli.StringFlag{ Name: "repo", EnvVars: []string{"LOTUS_PATH"}, Hidden: true, Value: "~/.lotus", },

In any commands which need the API:

lcli "github.com/filecoin-project/lotus/cli" ... api, closer, err := lcli.GetFullNodeAPI(cctx) if err != nil { return err } defer closer() ctx := lcli.ReqContext(cctx)

I can add this, which can be useful if you're running the binary from the same machine as the client, that is not my case, for example ;-)

Caught up with @magik6k 1:1. I've dropped the --api flag (now only an env var), and added the --repo flag and env variable binding. We now use the lotus CLI lib, and the usage docs on the command document how the API endpoint is resolved.

raulk · 2020-09-28T13:06:11Z

@magik6k this is ready to merge. The test failure looks like an unrelated, flaky test!

Kubuxu · 2020-09-28T13:53:18Z

Any concrete reason for placing it in separate command instead of lotus-shed?

raulk · 2020-09-28T13:58:41Z

@Kubuxu no concrete reason other than lotus-shed appearing to be a miscellaneous/catch-all/utilities location. This binary doesn't seem to fall in that category. It will become a pretty important tool in our testing workflow going forward, and will also be used by other implementers. Also, I want to be the code owner of this, and no other lotus-shed gadget ;-)

Kubuxu · 2020-09-28T14:18:35Z

cmd/tvx/stores.go

+// ChainReadObj RPC.
+func NewProxyingStores(ctx context.Context, api api.FullNode) *Stores {
+	ds := dssync.MutexWrap(ds.NewMapDatastore())
+	ds = dssync.MutexWrap(ds)


It is twice as race safe.

I want to be super extra sure :D Thanks for catching this!

magik6k · 2020-09-29T12:38:50Z

cmd/tvx/actor_mapping.go

+	builtin.InitActorCodeID:             builtin.MethodsInit,
+	builtin.CronActorCodeID:             builtin.MethodsCron,
+	builtin.AccountActorCodeID:          builtin.MethodsAccount,
+	builtin.StoragePowerActorCodeID:     builtin.MethodsPower,
+	builtin.StorageMinerActorCodeID:     builtin.MethodsMiner,
+	builtin.StorageMarketActorCodeID:    builtin.MethodsMarket,
+	builtin.PaymentChannelActorCodeID:   builtin.MethodsPaych,
+	builtin.MultisigActorCodeID:         builtin.MethodsMultisig,
+	builtin.RewardActorCodeID:           builtin.MethodsReward,
+	builtin.VerifiedRegistryActorCodeID: builtin.MethodsVerifiedRegistry,


With #3936 there are multiple code CIDs depending on actor version.

Ideally we would deduplicate this with the registry in chain/vm/invoker.go

I don't think there's anything I can do right now. When that PR lands, we can enhance the invoker so that it can be introspected from the outside and return metadata about the actors/methods it knows about. For that to be usable here, we'd need to record the method name too (probably just the name of the actor function).

magik6k · 2020-09-29T12:51:49Z

cmd/tvx/stores.go

+	"github.com/filecoin-project/lotus/api"
+	"github.com/filecoin-project/lotus/lib/blockstore"
+
+	"github.com/filecoin-project/specs-actors/actors/util/adt"


Should use chain/actors/adt to handle network upgrades correctly

raulk added 2 commits September 27, 2020 20:10

move conformance tvx tool to lotus.

e5c56da

tvx/extract: print confirmation.

a712c10

raulk requested review from Kubuxu, magik6k and whyrusleeping as code owners September 27, 2020 19:30

raulk changed the title ~~move conformance tvx tool to lotus.~~ tvx: a test vector extraction and execution tool Sep 27, 2020

raulk marked this pull request as draft September 27, 2020 19:32

willscott reviewed Sep 27, 2020

View reviewed changes

raulk added 3 commits September 27, 2020 20:55

driver: option for VM flushing.

f5f23f7

address review comments; lint.

fe869c9

fix lint errors.

9a355c4

raulk marked this pull request as ready for review September 27, 2020 20:12

magik6k reviewed Sep 28, 2020

View reviewed changes

raulk added 6 commits September 28, 2020 12:27

use lotus CLI package; document API endpoint setting in usage.

f05a40f

add --repo flag.

dfdcbd1

tvx/extract: allow passing in block to speed things up.

8f3be78

tvx/extract: perform sanity check on receipt.

a0dffb4

tvx/extract: small refactor.

4c97171

fix lint.

9f6862a

Kubuxu reviewed Sep 28, 2020

View reviewed changes

add extract-many command.

96f8828

raulk mentioned this pull request Sep 28, 2020

combine #260 and #246; remove driver (now in Lotus) filecoin-project/oni#261

Merged

raulk added 4 commits September 28, 2020 23:04

conformance: record and feed circulating supply.

4b3b35c

fix lint.

9b403e2

Merge pull request #4078 from filecoin-project/tvx-circsupply

0070d27

fix double mutex.

0446744

magik6k reviewed Sep 29, 2020

View reviewed changes

raulk added 4 commits September 30, 2020 11:02

tvx: precursor selection modes; canonical message fetching; basefee.

eb6191d

minor fixes.

f58881e

update test-vectors submodule.

ff8663f

tvx/extract-many: add batch id, change generated filename.

1c4f8e8

magik6k approved these changes Sep 30, 2020

View reviewed changes

magik6k merged commit 5bffea6 into master Sep 30, 2020

magik6k deleted the tvx branch September 30, 2020 15:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tvx: a test vector extraction and execution tool #4064

tvx: a test vector extraction and execution tool #4064

raulk commented Sep 27, 2020 •

edited

Loading

willscott Sep 27, 2020

raulk Sep 27, 2020

magik6k Sep 28, 2020 •

edited

Loading

raulk Sep 28, 2020

raulk Sep 28, 2020

raulk commented Sep 28, 2020

Kubuxu commented Sep 28, 2020

raulk commented Sep 28, 2020

Kubuxu Sep 28, 2020

raulk Sep 28, 2020

magik6k Sep 29, 2020

raulk Sep 30, 2020

magik6k Sep 29, 2020

raulk Sep 30, 2020

tvx: a test vector extraction and execution tool #4064

tvx: a test vector extraction and execution tool #4064

Conversation

raulk commented Sep 27, 2020 • edited Loading

tvx extract --cid <message id>

tvx exec --file <path>

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magik6k Sep 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raulk commented Sep 28, 2020

Kubuxu commented Sep 28, 2020

raulk commented Sep 28, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raulk commented Sep 27, 2020 •

edited

Loading

`tvx extract --cid <message id>`

`tvx exec --file <path>`

magik6k Sep 28, 2020 •

edited

Loading