Transaction Validation #736

SethDusek · 2023-12-28T23:58:40Z

Initial stateless validation work (things we can validate without needing blockchain context). Also includes a fix where we were allowing as many as u16::MAX inputs/outputs (65535) when ergo only allows i16::MAX (Short.MaxValue in scala) which is (32767). https://github.com/ergoplatform/ergo/blob/48239ef98ced06617dc21a0eee5670235e362933/ergo-core/src/main/scala/org/ergoplatform/modifiers/mempool/ErgoTransaction.scala#L93

…saction Scala's Short.MaxValue is equivalent to the maximum a 16 bit signed integer can hold

coveralls · 2023-12-29T00:28:41Z

Pull Request Test Coverage Report for Build 8306068744

Details

165 of 228 (72.37%) changed or added relevant lines in 15 files are covered.
30 unchanged lines in 17 files lost coverage.
Overall coverage increased (+0.05%) to 80.801%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
ergo-lib/src/chain/transaction/ergo_transaction.rs	28	29	96.55%
ergotree-interpreter/src/eval/error.rs	0	2	0.0%
ergotree-interpreter/src/eval/scontext.rs	3	5	60.0%
ergotree-interpreter/src/eval/scoll.rs	8	11	72.73%
ergotree-ir/src/mir/value.rs	3	6	50.0%
ergo-lib/src/chain/transaction.rs	7	11	63.64%
ergo-lib/src/chain/transaction/storage_rent.rs	7	21	33.33%
ergo-lib/src/chain/parameters.rs	12	28	42.86%
ergo-lib/src/wallet/tx_context.rs	73	91	80.22%

Files with Coverage Reduction	New Missed Lines	%
ergotree-interpreter/src/sigma_protocol/prover.rs	1	78.43%
ergotree-ir/src/mir/avl_tree_data.rs	1	91.3%
ergo-chain-generation/src/chain_generation.rs	1	87.33%
ergotree-interpreter/src/eval/exponentiate.rs	1	81.82%
ergotree-ir/src/serialization/sigmaboolean.rs	1	81.63%
ergo-lib/src/wallet/tx_builder.rs	1	73.83%
ergo-lib/src/wallet/derivation_path.rs	1	79.17%
ergo-rest/src/api/peer_discovery_internals/non_chrome.rs	1	73.28%
ergo-lib/src/wallet/box_selector/simple.rs	1	78.1%
ergotree-interpreter/src/eval/deserialize_register.rs	1	79.17%

Totals
Change from base Build 7307508480:	0.05%
Covered Lines:	10719
Relevant Lines:	13266

💛 - Coveralls

set_hook can only be called once, this means displaying an error multiple times causes a panic in sigma-rust. ignoring seems like the best solution

When converting a signed integer i8 like -1 (0xff) to a bigger signed integer like i64, the 2's complement representation will be padded with ff (0xffffffff...). Thus if input[7] for example is -1, when it is |'ed, all high bits will be set to 1. Converting u8 0xff to i64 0xff leads to correct conversion (0x000000...ff) This fixes validation for transactions like https://ergexplorer.com/transactions#7f838c5060185125cb554323401d87c0fc135a0c5fc545f9e315ffc38b97b06e

The current indexOf implementation doesn't match the behavior of the one in sigmastate-interpreter. The previous version wrongly treated the second argument as a fallback index whereas it is actually the position in the array to start searching from

Some mainnet transactions like 9f4d40607a561c14a504c96919362f150fca3337ab43219d9a683f1617b74b0a have proofs that are smaller than usual but still valid. Relevant sigmastate-interpreter commit ScorexFoundation/sigmastate-interpreter@ef6ff08

In v5.0 onwards flatMap can have any arbitrary lambda in body as long as it returns a Coll[T]

This brings down TransactionContext creation for transactions with 32767 inputs down from ~1.2 seconds to ~160 milliseconds

This reduces validation time for some transactions like 34fecabe033ca5b516ddaf99a251baf1a580db5be1ccd7ca41bfcd953bb1923f from 11 seconds down to ~1.2

Dependencies and rust-analyzer are broken on older rust version. Also bump hashbrown due to ahash's nightly features breaking on older versions

SethDusek · 2024-03-18T11:20:58Z

Should be good to go now! Some notes:

This PR despite the usual unit tests was tested using ergovalidation which downloads transactions from node (requires extra indexing to be enabled) and validates them. After fixing some script bugs we're now down to 15 transactions on mainnet that sigma-rust rejects, all of which rely on buggy behavior from v4.0 that no longer holds in v5.0, including CONTEXT.selfBoxIndex being -1 and sigmaProp(sigmaProp(true)) being allowed. I believe fixing these (and adding version-specific actions to ergotree-interpreter) is out of scope for this PR. Gist of failing transactions
I had to bump rust version since sigma-rust no longer builds as some dependencies don't support rust 1.64 anymore. Clippy rejection seems to be related and should be ignored for now
There are some limitations still in interpreter. The biggest one is that atLeast(0, ...) isn't supported (I plan on addressing this separately in another PR, but I fixed atLeast(1, ...) not being supported in this PR).
There are some breaking changes in this PR, namely ErgoStateContext now also requiring Parameters. I believe this is an acceptable trade-off even if parameters aren't needed for signing. Later when sigma-rust adds support for costing we could use Parameters to calculate costs during signing
Also fixed quadratic scaling in TransactionContext::new
old:

new:

SethDusek · 2024-03-18T11:31:50Z

ergotree-interpreter/src/eval/scoll.rs

@@ -24,15 +24,20 @@ pub(crate) static INDEX_OF_EVAL_FN: EvalFn = |_env, _ctx, obj, args| {
            .get(0)
            .cloned()
            .ok_or_else(|| EvalError::NotFound("indexOf: missing first arg".to_string()))?;
-        let fallback_index = args


indexOf now has the same semantics as scala version, https://github.com/ScorexFoundation/sigmastate-interpreter/blob/efa68e7079393d70d25d43dd63dfba9a18d03415/data/shared/src/main/scala/sigma/ast/methods.scala#L880

kushti · 2024-04-26T11:09:13Z

ergo-lib/src/chain/transaction/ergo_transaction.rs

+    /// When using sigma-rust where heights are always unsigned, this error may be because creation height was set to be >= 2147483648
+    NegativeHeight,
+    #[error("Output box size {0} > maximum {}", ErgoBox::MAX_BOX_SIZE)]
+    /// Box size is > [ErgoBox::MAX_SCRIPT_SIZE]


MAX_BOX_SIZE

kushti · 2024-04-26T11:09:56Z

Please fix linter issues

SethDusek added 2 commits December 29, 2023 04:53

Use i16::MAX instead of u16::MAX for maximum inputs/outputs in a tran…

8eaa31a

…saction Scala's Short.MaxValue is equivalent to the maximum a 16 bit signed integer can hold

Add stateless validation for transactions

7d3cb1f

SethDusek added 3 commits January 17, 2024 13:27

Add ArbBoxParameters for ErgoBox/ErgoBoxCandidate proptests

b65378c

Add Parameters to ErgoStateContext

2ee2516

Stateful Transaction Validation

d5206fc

SethDusek force-pushed the txvalidation branch from 2183f20 to d5206fc Compare January 30, 2024 18:44

SethDusek added 3 commits February 3, 2024 05:10

Add documentation for Parameters/TxValidationError

6ee9e82

Add monotonic height validation

b6384d4

Add Parameters to WASM/C bindings

a4e5a92

SethDusek force-pushed the txvalidation branch from 678472f to a4e5a92 Compare February 5, 2024 01:06

Ignore miette set_hook errors

6a1547c

set_hook can only be called once, this means displaying an error multiple times causes a panic in sigma-rust. ignoring seems like the best solution

kushti added the bounty - 500 SigUSD label Feb 14, 2024

SethDusek force-pushed the txvalidation branch from 9f1d83e to 8614bab Compare February 23, 2024 11:29

SethDusek added 9 commits March 14, 2024 11:55

Fix indexOf implementation

a5e93cb

The current indexOf implementation doesn't match the behavior of the one in sigmastate-interpreter. The previous version wrongly treated the second argument as a fallback index whereas it is actually the position in the array to start searching from

Add additional byteArrayToLong test

fcbf1a4

Add CONTEXT.minerPubkey method

37dbd7d

Relax flatMap restrictions + fix inference bugs

130acbe

In v5.0 onwards flatMap can have any arbitrary lambda in body as long as it returns a Coll[T]

Handle negative from/replaced in Coll.patch

2555e99

perf: Fix quadratic TransactionContext::new creation

89b54f9

This brings down TransactionContext creation for transactions with 32767 inputs down from ~1.2 seconds to ~160 milliseconds

Add Storage rent support in transaction validation

ea9e67c

perf: re-use bytes_to_sign() in verify_tx_input_proof

90c8d36

This reduces validation time for some transactions like 34fecabe033ca5b516ddaf99a251baf1a580db5be1ccd7ca41bfcd953bb1923f from 11 seconds down to ~1.2

SethDusek force-pushed the txvalidation branch 4 times, most recently from 6a556e1 to 7529e70 Compare March 15, 2024 04:41

SethDusek added 2 commits March 15, 2024 12:15

Add TX validation bindings

da8adb2

support atLeast(1, ...)

8a91c6f

Bump Rust version to 1.76

3751422

Dependencies and rust-analyzer are broken on older rust version. Also bump hashbrown due to ahash's nightly features breaking on older versions

SethDusek force-pushed the txvalidation branch from 7529e70 to 3751422 Compare March 15, 2024 08:29

Fix doc comments + merge TxValidationError/TxVerifyError

fdedb60

SethDusek force-pushed the txvalidation branch from 8818810 to fdedb60 Compare March 16, 2024 04:46

Fix flakey transaction validation tests

df822c6

SethDusek marked this pull request as ready for review March 18, 2024 11:14

SethDusek requested a review from ross-weir March 18, 2024 11:29

SethDusek commented Mar 18, 2024

View reviewed changes

SethDusek mentioned this pull request Apr 16, 2024

Criterion Benchmarks #739

Open

SethDusek assigned SethDusek and unassigned SethDusek Apr 25, 2024

kushti approved these changes Apr 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transaction Validation #736

Transaction Validation #736

SethDusek commented Dec 28, 2023 •

edited

coveralls commented Dec 29, 2023 •

edited

SethDusek commented Mar 18, 2024 •

edited

SethDusek Mar 18, 2024

kushti Apr 26, 2024

kushti commented Apr 26, 2024

Transaction Validation #736

Are you sure you want to change the base?

Transaction Validation #736

Conversation

SethDusek commented Dec 28, 2023 • edited

coveralls commented Dec 29, 2023 • edited

Pull Request Test Coverage Report for Build 8306068744

Details

💛 - Coveralls

SethDusek commented Mar 18, 2024 • edited

SethDusek Mar 18, 2024

Choose a reason for hiding this comment

kushti Apr 26, 2024

Choose a reason for hiding this comment

kushti commented Apr 26, 2024

SethDusek commented Dec 28, 2023 •

edited

coveralls commented Dec 29, 2023 •

edited

SethDusek commented Mar 18, 2024 •

edited