refactor: analyzer/runtime: Store a more-parsed tx in the db #368

mitjat · 2023-03-29T03:53:51Z

This PR changes the stage at which we process runtime txs: The processing used to be spread between the analyzer and a "renderer" at http api call time; it now happens entirely in the analyzer, and the parsed tx is stored in the db.

This is a no-op externally, but allows us

one-off queries/analyses in the db
implementation of filters on inner fields, such as method = accounts.Transfer

The first two commits are just pre-work and could well be separate small PRs. I recommend reviewing separately.

Testing: reindex_and_run.sh for 5000 rounds.
Diffs: The body of consensus.Withdraw txs is now serialized in a more verbose/interpretable way: Before:

      "body": {
        "amount": [
          "HLs6P/CNCAAA",
          ""
        ],
        "to": "AJifyI0wEVN16b6H61PxYQ/Hkvaw"
      },

After:

      "body": {
        "amount": {
          "Amount": "530000000000000000000",
          "Denomination": ""
        },
        "to": "oasis1qzvfljydxqg4xa0fh6r7k5l3vy8u0yhkkqak5uaq"
      },

Andrew7234

Looks good overall; would appreciate another pair of eyes on extract.go to double-check that it's not missing anything from logic.go.

storage/client/types.go

Andrew7234 · 2023-03-30T00:59:31Z

analyzer/runtime/runtime.go

+			transactionData.Method,
+			transactionData.Body,
+			transactionData.To,
+			transactionData.Amount,


Since it's also a common.BigInt, should this be

Suggested change

transactionData.Amount,

transactionData.Amount.String(),

This conversion finally works "automatically" (we do provide a special serialization method on BigInt) as it should since we've migrated to pgx v5. (Edit: Not really, read on)

If you're alluding to transactionData.Fee.String() above, that did require the String() for some reason ... The difference is that Fee is BigInt whereas Amount is *BigInt. I change Fee.String() to &Fee and now everything (still) works, with no explicit String(). 😬

Andrew7234 · 2023-03-30T01:01:25Z

storage/client/queries/queries.go

+		LEFT JOIN chain.runtime_transaction_signers AS signer0 USING (runtime, round, tx_index)
+		LEFT JOIN chain.address_preimages AS signer_eth ON
+			(signer0.signer_address = signer_eth.address) AND
+			(signer_eth.context_identifier = 'oasis-runtime-sdk/address: secp256k1eth') AND (signer_eth.context_version = 0)
+		LEFT JOIN chain.address_preimages AS to_eth ON
+			(txs.to = to_eth.address) AND
+			(to_eth.context_identifier = 'oasis-runtime-sdk/address: secp256k1eth') AND (to_eth.context_version = 0)
+		LEFT JOIN chain.runtime_related_transactions AS rel ON 
+			(txs.round = rel.tx_round) AND
+			(txs.tx_index = rel.tx_index) AND
+			(txs.runtime = rel.runtime) AND


this query is quite impressive; do we expect any performance issues that might arise?

I actually expect it to be faster. (Famous last words ...)
They're very lightweight JOINs, they each look up a single row per tx, and using an index. The nested SELECT we had before is where the engine is more likely to have a hard time optimizing it into a reasonable plan. This just replaces the nested SELECT (with a JOIN of its own, no less!) with two new JOINs.

Andrew7234 · 2023-03-30T01:04:04Z

storage/client/queries/queries.go

+			($4::text IS NOT NULL)
+		WHERE
+			(txs.runtime = $1) AND
+			(signer0.signer_index = 0) AND


might be a dumb q, but is it worth moving this line to the join above? eg, LEFT JOIN chain.runtime_transaction_signers AS signer0 USING (runtime, round, tx_index) ON (signer0.signer_index = 0)

Not dumb at all, might be a little more readable. I debated it myself. A gotcha is that you cannot mix ON and USING, so we'd have to expand the USING into a much longer ON block, which I think hurts readability overall. Happy to change it if you have stronger feelings here; I don't.

question about this: does factoring out this WHERE signer0.signer_index = 0 change the meaning of the join outer-sidedness? if we tried doing the same for a hypothetical signer1 field, would the dbms conceptually first come up with a record:

method=accounts.Transfer, signer0=oasis1xxxx, signer1=NULL

and then determine NULL != 0 and discard it?

we're lucky here that everything actually has a signer0

I thought they were equivalent. I now see there's a subtle difference:

a restriction placed in the ON clause is processed before the join, while a restriction placed in the WHERE clause is processed after the join. That does not matter with inner joins, but it matters a lot with outer joins.

See "Notice that placing the restriction in the WHERE clause produces a different result:" here.

Thanks. For our case ... Perhaps we should keep the WHERE even though it's less "pure"? It makes the join less "outer": In the hypothetical case of a tx without a signer, WHERE would discard that tx, whereas ON would return the tx but with a null signer. But our data types don't support null signers. Hrm it's actually good that we'd then crash and return a 500 for a result involving such a tx.

OK let's change the query. It's all pretty philosophical because we always have a signer, but still.

changing in #367

Andrew7234 · 2023-03-30T01:12:39Z

storage/migrations/02_runtimes.up.sql

+
+  fee         UINT_NUMERIC NOT NULL,
+  gas_limit   UINT63 NOT NULL,
+  gas_used    UINT63 NOT NULL,


if a tx fails, will gas_used still be not null? or do we assume it's either 0 or gas_limit?

We do assume a number. Having NULL here is an interesting option ... one not discussed in #362 (comment).
I'd like to keep it NOT NULL here so we're not introducing another feature/change in this PR. This PR just puts into the DB what we already had in Go variables, with unchanged semantics.

…ration on just the API

…TTP API fetch

mitjat · 2023-03-30T20:49:57Z

would appreciate another pair of eyes on extract.go to double-check that it's not missing anything from logic.go.

That is my biggest concern too; e2e regressions at least cover the most common events though. (They might well miss rarer events if they don't happen to appear in that single API query that run.sh does.)

pro-wh · 2023-03-30T21:44:14Z

storage/client/queries/queries.go

@@ -355,7 +355,7 @@ const (
 					($5::text IS NULL OR type = $5::text) AND
 					($6::text IS NULL OR evm_log_signature = $6::text) AND
 					($7::text IS NULL OR related_accounts @> ARRAY[$7::text])
-			ORDER BY round DESC, tx_index
+			ORDER BY round DESC, tx_index, type, body::text


bleh, we ought to file a low priority issue about preserving event order from the node

pro-wh · 2023-03-30T21:51:38Z

storage/migrations/02_runtimes.up.sql

+
+  -- Transaction contents.
+  method      TEXT NOT NULL,  -- accounts.Transter, consensus.Deposit, consensus.Withdraw, evm.Create, evm.Call
+  body        JSON NOT NULL,  -- For EVM txs, the EVM method and args are encoded in here.


is this EVM only? or is the comment about this having a different meaning in some cases?

pro-wh · 2023-03-30T22:10:43Z

storage/client/queries/queries.go

-			) AS eth_addr_lookup
+			txs.method,
+			txs.body,
+			-- .to, .to_eth, and .amout have to be inferred from the body post-hoc


amout -> amount

~~what's our strategy for to_eth now? second query?~~ next commit, we decide one during analysis

pro-wh · 2023-03-30T22:15:29Z

storage/client/client.go

+		}
+		// Fancy-format eth addresses: Apply checksum capitalization, prepend 0x.
+		if t.Sender0Eth != nil {
+			*t.Sender0Eth = ethCommon.HexToAddress(*t.Sender0Eth).Hex()


view logic in storage layer 🙃

pro-wh · 2023-03-30T22:23:02Z

storage/client/queries/queries.go

-			AND txs.tx_index = rel.tx_index
-			AND txs.runtime = rel.runtime
+		LEFT JOIN chain.runtime_transaction_signers AS signer0 USING (runtime, round, tx_index)
+		LEFT JOIN chain.address_preimages AS signer_eth ON


AS signer0_eth

pro-wh · 2023-03-31T00:02:51Z

storage/client/queries/queries.go

@@ -325,7 +325,9 @@ const (
 			txs.size,
 			txs.method,
 			txs.body,
-			-- .to, .to_eth, and .amout have to be inferred from the body post-hoc
+			txs.to,


pro-wh · 2023-03-31T00:04:09Z

analyzer/uncategorized/quantities.go

@@ -21,6 +21,6 @@ func StringifyBytes(value []byte) string {

 func QuantityFromBytes(value []byte) quantity.Quantity {
 	q := *quantity.NewQuantity()
-	q.FromBigInt(new(big.Int).SetBytes(value))
+	_ = q.FromBigInt(new(big.Int).SetBytes(value))


here's this to put the errors into 🕳️

pro-wh · 2023-03-31T00:04:41Z

analyzer/runtime/runtime.go

@@ -304,7 +304,7 @@ func (m *Main) queueDbUpdates(batch *storage.QueryBatch, data *BlockData) {
 			transactionData.Index,
 			transactionData.Hash,
 			transactionData.EthHash,
-			transactionData.Fee.String(),
+			&transactionData.Fee, // pgx bug? Needs a *BigInt (not BigInt) to know how to serialize.


maybe we defined a method only on *BigInt?

v tangential but encoding/gob complains if you use pointer receivers like func (t *MyType) MarshalBytes... go pointers are funky i guess

pro-wh

ah that was less hectic than I was afraid of

mitjat requested review from aefhm, Andrew7234 and pro-wh as code owners March 29, 2023 03:53

mitjat mentioned this pull request Mar 29, 2023

storage client: get entire address preimage #367

Merged

Andrew7234 approved these changes Mar 30, 2023

View reviewed changes

mitjat force-pushed the mitjat/cobalt-runtime branch 2 times, most recently from 7a221cb to 72ad55f Compare March 30, 2023 20:11

Base automatically changed from mitjat/cobalt-runtime to main March 30, 2023 20:37

mitjat added 7 commits March 30, 2023 13:43

tests/e2e_regression: Run the API server in run.sh; allows easier ite…

3eff2e9

…ration on just the API

api: Return events in deterministic order

9327bb9

analyzer/runtime: Store a more-parsed tx in the db

cd21b56

storage/client: Pull analyzed tx data from db, do not re-analyze on H…

20a13af

…TTP API fetch

Stop storing raw tx in db

25e2bc0

populate and fetch .to, .to_eth, .amount

c8dbca9

lint

13298d3

mitjat force-pushed the mitjat/store-parsed-runtime-tx-in-db branch from 86e046a to 13298d3 Compare March 30, 2023 20:43

Serialize RuntimeTransaction.Fee using default serializer

02ef2d0

mitjat enabled auto-merge March 30, 2023 21:05

mitjat merged commit 307f879 into main Mar 30, 2023

mitjat deleted the mitjat/store-parsed-runtime-tx-in-db branch March 30, 2023 21:07

pro-wh reviewed Mar 30, 2023

View reviewed changes

pro-wh reviewed Mar 31, 2023

View reviewed changes

pro-wh mentioned this pull request Mar 31, 2023

sender_0_eth doesn't check AddressPreimage context before converting oasis1 address #365

Closed

mitjat mentioned this pull request Nov 14, 2023

runtime/evm: ignore unsupported token types #563

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: analyzer/runtime: Store a more-parsed tx in the db #368

refactor: analyzer/runtime: Store a more-parsed tx in the db #368

mitjat commented Mar 29, 2023

Andrew7234 left a comment

Andrew7234 Mar 30, 2023

mitjat Mar 30, 2023

Andrew7234 Mar 30, 2023

mitjat Mar 30, 2023

Andrew7234 Mar 30, 2023

mitjat Mar 30, 2023

pro-wh Mar 30, 2023

mitjat Apr 3, 2023 •

edited

Loading

pro-wh Apr 3, 2023

Andrew7234 Mar 30, 2023

mitjat Mar 30, 2023

mitjat commented Mar 30, 2023

pro-wh Mar 30, 2023

pro-wh Mar 30, 2023

pro-wh Mar 30, 2023

pro-wh Mar 30, 2023 •

edited

Loading

pro-wh Mar 30, 2023

pro-wh Mar 30, 2023

pro-wh Mar 31, 2023

pro-wh Mar 31, 2023

pro-wh Mar 31, 2023

Andrew7234 Mar 31, 2023

pro-wh left a comment

refactor: analyzer/runtime: Store a more-parsed tx in the db #368

refactor: analyzer/runtime: Store a more-parsed tx in the db #368

Conversation

mitjat commented Mar 29, 2023

Andrew7234 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mitjat Apr 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mitjat commented Mar 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pro-wh Mar 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pro-wh left a comment

Choose a reason for hiding this comment

mitjat Apr 3, 2023 •

edited

Loading

pro-wh Mar 30, 2023 •

edited

Loading