trace compatibility testing with Trueblocks #3721

wakamex · 2023-07-11T15:30:47Z

Describe the bug

trying to create a local trueblocks unchained index, I'm getting some warnings:

Unknown trace type selfdestruct
Unknown trace type selfdestruct
Unknown trace type selfdestructng 391  of 2000 at block 11182389 of 17659147 (6476758 blocks from head)
Unknown trace type selfdestruct
Unknown trace type selfdestruct

I'm not clear how impactful this particular warning is to Trueblocks operation, but it points to something non-standard in Reth's trace implementation.

Digging into this further, I couldn't find specifically where in the trueblocks or erigon codebases they deal with selfdestruct in a way that could explain this discrepancy.

However, Trueblocks has a long list of tests for expected tracing functionality in /src/other/trace_tests/curl_tests. Running these on my reth node, I get a difference from expected for 26 out of 35 tests. You can find the diffs of those tests in this PR: https://github.com/TrueBlocks/trueblocks-core/pull/3037/files

@tjayrush could explain these further.

Steps to reproduce

reproduce warnings above:

sync erigon
install trueblocks
run chifra scrape

get comparison vs. expected Trueblocks trace tests

run_all in /src/other/trace_tests/curl_tests of trueblocks repo (modify line 3 as needed: export RPC_ENDPOINT="http://localhost:23456")
compare_expected_produced.sh

Node logs

No response

Platform(s)

Linux (x86)

What version/commit are you on?

reth Version: 0.1.0-alpha.2 Commit
SHA: 1330fc1
Build Timestamp: 2023-07-08T22:15:01.820025074Z
Build Features:
Build Profile: release

What database version are you on?

Current database version: 1
Local database is uninitialized

If you've built Reth from source, provide the full command you used

cargo build --release

Code of Conduct

I agree to follow the Code of Conduct

The text was updated successfully, but these errors were encountered:

mattsse · 2023-07-11T17:08:48Z

this is very helpful, thanks a lot

looking into this

tjayrush · 2023-07-11T19:38:09Z

Thanks @wakamex. @mattsse, please makes sure to ping me either in discord or here if you have any questions about any of these tests. We wrote these tests when we were porting from OpenEthereum to Erigon and as far as I know they worked about two and a half years ago with Erigon.

Things may have changed under the covers with Erigon (although I hope not). So, if a test does not seem right, don't assume there's something wrong with Reth. Ping me and I can help you figure it out.

I'm going to install Reth this weekend, so should have a locally running copy soon to help test this.

mattsse · 2023-07-12T14:46:27Z

looked at the unknown trace type error, and turns out the OE traces never migrated from suicide to selfdestruct

https://github.com/TrueBlocks/trueblocks-core/blob/7c65387d5d5aa97af7ccf6269281f200711a74ca/src/apps/chifra/internal/scrape/handle_blaze_blaze.go#L294-L294

so in order to be compatible I think we should rename this to suicide as well

mattsse · 2023-07-12T14:50:39Z

@tjayrush @wakamex

is the insufficient funds error erigon, and the success response reth?

https://github.com/TrueBlocks/trueblocks-core/pull/3037/files#diff-b294536ca537aa07d86d1b87ba6bf94552732b947fb6ecf1fdf06c5d8279557aR7-R17

wakamex · 2023-07-12T15:15:36Z

is the insufficient funds error erigon, and the success response reth?

yup, you can see the reth output for each test in /src/other/trace_tests/curl_tests/reth

for that test in src/other/trace_tests/curl_tests/reth/trace_callMany_all.json we have:
https://github.com/TrueBlocks/trueblocks-core/pull/3037/files?file-filters%5B%5D=.json&show-viewed-files=true#diff-43fbbd2af40737486871fc9b12f73ae842426b16a37678e8b5ece40be1be9f78R1-R8

Perhaps more useful would be running these tests again today against an erigon node, as that seems to be a better source of truth than OE?

tjayrush · 2023-07-12T15:38:49Z

We should definitely run the tests again against Erigon as these 'expected' results are about two and a half years old. While it would be surprising if they changed things, it's very possible that they did.

tjayrush · 2023-07-12T15:40:08Z

Just a quick note. These tests are purposefully using curl (as opposed to say, TrueBlocks) so as to eliminate any weird processing in TrueBlocks. The curl responses from both nodes should be identical (at least that's what I think).

mattsse · 2023-07-12T16:05:33Z

I think the insufficient fund error can be explained by:

https://github.com/ledgerwatch/erigon/blob/af504f675a84868b941fbad840a3d2819fb525bf/turbo/jsonrpc/trace_adhoc.go#L967-L967

which is different from eth_call for example:

https://github.com/ledgerwatch/erigon/blob/af504f675a84868b941fbad840a3d2819fb525bf/turbo/transactions/call.go#L86-L86

note the disabled basefee, reth traceCall currently behaves as eth_call: basefee disabled, gas price 0

which differs from erigon

mattsse · 2023-07-12T16:06:54Z

it's still suicide

https://github.com/ledgerwatch/erigon/blob/af504f675a84868b941fbad840a3d2819fb525bf/turbo/jsonrpc/trace_adhoc.go#L40-L40

so I changed selfdestruct to suicide here for compact #3736

tjayrush · 2023-07-12T16:08:53Z

We should probably check against a current version of Erigon. These 'expected' results are two and a half years old. @wakamex Do you have an Erigon node?

gakonst · 2023-07-12T18:22:43Z

@tjayrush reaching out in DMs with an RPC.

wakamex · 2023-07-12T19:40:52Z

I got my hands on an erigon node. Here are the reth vs. erigon results:

Comparing reth_2023_07_11/ to erigon_2023_07_12/, placing results into reth_erigon/
Command executed on Wed Jul 12 19:36:46 UTC 2023
=== Not found ===

=== Identical ===
erigon_2023_07_12/trace_get_0.json
erigon_2023_07_12/trace_get_3.json
erigon_2023_07_12/trace_replayBlockTransactions_trace.json
erigon_2023_07_12/trace_replayTransaction_all.json
erigon_2023_07_12/trace_replayTransaction_trace.json
=== Difference found ===
erigon_2023_07_12/trace_block.json
erigon_2023_07_12/trace_callMany_all.json
erigon_2023_07_12/trace_callMany_none.json
erigon_2023_07_12/trace_callMany_stateDiff.json
erigon_2023_07_12/trace_callMany_trace.json
erigon_2023_07_12/trace_callMany_vmTrace.json
erigon_2023_07_12/trace_call_all.json
erigon_2023_07_12/trace_call_error.json
erigon_2023_07_12/trace_call_none.json
erigon_2023_07_12/trace_call_stateDiff.json
erigon_2023_07_12/trace_call_trace.json
erigon_2023_07_12/trace_call_view.json
erigon_2023_07_12/trace_call_view_none.json
erigon_2023_07_12/trace_call_vmTrace.json
erigon_2023_07_12/trace_filter.json
erigon_2023_07_12/trace_get_10.json
erigon_2023_07_12/trace_get_5.json
erigon_2023_07_12/trace_rawTransaction_none.json
erigon_2023_07_12/trace_rawTransaction_trace.json
erigon_2023_07_12/trace_rawTransaction_vmTrace.json
erigon_2023_07_12/trace_replayBlockTransactions_all.json
erigon_2023_07_12/trace_replayBlockTransactions_none.json
erigon_2023_07_12/trace_replayBlockTransactions_stateDiff.json
erigon_2023_07_12/trace_replayBlockTransactions_vmTrace.json
erigon_2023_07_12/trace_replayTransaction_none.json
erigon_2023_07_12/trace_replayTransaction_stateDiff.json
erigon_2023_07_12/trace_replayTransaction_vmTrace.json
erigon_2023_07_12/trace_transaction.json

You can find the diffs here: https://github.com/wakamex/trueblocks-core/tree/compare_script/src/other/trace_tests/curl_tests/reth_erigon

See this PR for the scripts I used, or run it yourselves: TrueBlocks/trueblocks-core#3037

tjayrush · 2023-07-13T18:36:15Z

I thought I'd add some color to some of these test cases. This is from memory, so take it with a grain of salt, but...

These failed tests return null instead of [] for traceAddress. This should be simple to fix.

We thought that someone had hand-coded assembly language for this transaction and had made a mistake, so reporting an error is correct. The error, I think, is correct if you look at the actual data. Not sure why it's saying something about the nonce.

Be a little bit careful with this one. I'm not sure this is the exact transaction, but I remember that OpenEthereum may have had a bug in certain places where the first trace in some transactions was incorrectly reported. Erigon found may have hit on this same issue (which is why this test is included). If I remember right, Erigon ended up adding a "compatibility" mode for some cases where they would act correctly by default but allow the user to cause old OpenEthereum bugs to persist. I don't know if this is such a case, but be careful here.

https://github.com/wakamex/trueblocks-core/blob/compare_script/src/other/trace_tests/curl_tests/reth_erigon/trace_get_5.json.diff

This may have been one of those backward compatibility things. The spec may say it requires a u64, but most of the other RPC endpoints require hex. So, as a compromise, even thought it's not in the spec, this was allowed. The spec is not really a spec after all.

https://github.com/wakamex/trueblocks-core/blob/compare_script/src/other/trace_tests/curl_tests/reth_erigon/trace_filter.json.diff

Personally, I think better error messages are better, but this may break existing code. It won't break TrueBlocks, but it may break others.

https://github.com/wakamex/trueblocks-core/blob/compare_script/src/other/trace_tests/curl_tests/reth_erigon/trace_call_error.json.diff

mattsse · 2023-07-14T09:02:10Z

tysm for these

for trace_rawTransaction, I think the nonce error occurs because the transaction's nonce is lower that what's in latest state.
so this tx would be invalid at this state

gakonst · 2023-07-14T13:14:16Z

We'll get back to these - thank you TJ, we're resyncing that node for now so it won't be available if you're trying to do more things.

wakamex · 2023-07-14T17:39:21Z

updated the baseline expectation in favor of modern Erigon in these benign cases: changed gas value, remove of null transactionHash and transactionPosition, improved error message (for more info see wakamex/trace_tests#2)

this leaves only 2 categories of differences between this baseline and modern Erigon, which require further investigation:

error where there was result before
- can't tell if the previous results were bogus or just more verbose (5 trace_rawTransaction_* tests)
different behavior of trace_get returning chained traces (2 trace_get_* tests)
- modern Erigon returns full-depth trace which matches Etherscan 0x88162c6bcbb040fd8f6676fe336f83d7a27a5765b18bfce524beea2a0e107159

result of test_trace_get.sh (Array is the second parameter as described here, table displays result.traceAddress field):

Array	reth	erigon
0	[0]	[0]
1	[1]	[1]
2	[2]	[2]
3	[3]	[3]
4	[4]	[4]
5	[5]	[4,0]
6	[6]	[4,1]
7	null	[4,2]
8	null	[5]
9	null	[5,0]
10	null	[5,1]
11	null	[5,2]
12	null	[6]
13	null	[6,0]
14	null	[6,1]
15	null	[6,2]
16	null	null

tjayrush · 2023-07-14T20:12:49Z

Just one quick comment. The traceAddress example above is super interesting and important for getting the correct calling structure of the transaction. Basically, the array is a call tree. It's almost certainly zero-based, but it may be one-based. I can't remember. [6,2] is the sixth call from the second call in the top level (or seventh and third if zero-based, but you get the idea.)

I'm 99% sure the very top level (trace) is null. The top level trace basically is the transaction itself with slight (very confusing) differences in the gas calc.

mattsse · 2023-07-19T15:21:20Z

thanks for the example I was able to track this down,

previously we matched the indices to the trace address which explains the null values.

should be fixed in #3852

gakonst · 2023-07-20T20:23:41Z

What else do we need to close this? I spoke with Peter and he said 50M gas for tracing was a heuristic, so maybe we should just set it to that, and call it a day?

tjayrush · 2023-07-21T17:24:42Z

Are the tests passing? The tests represent previous behaviour.

yabirgb · 2023-07-28T16:18:08Z

@gakonst any update on this? Was the limit increased?

mattsse · 2023-07-28T17:24:15Z

yes, we also have some open tracing fixes that will land shortly, I will check compatibility again

tjayrush · 2023-08-07T17:06:51Z

I'm in the process of syncing my node. Once it's synced, the first thing I'll do is double-check these issues and try to make them more consumable (by creating a separate, much simpler, example for each outstanding issue).

mattsse · 2023-08-07T17:23:52Z

tysm for this,

fyi we're aiming for a new alpha release within the next few days

onbjerg · 2024-04-02T18:30:54Z

@wakamex @tjayrush Hi, I realize this issue is pretty old, but I am wondering if this is still an issue for you? We've had quite a few tracing fixes since the versions mentioned in here

erwin-wee · 2024-06-28T09:23:57Z

tysm for these

for trace_rawTransaction, I think the nonce error occurs because the transaction's nonce is lower that what's in latest state. so this tx would be invalid at this state

Hi @mattsse, is the node supposed to validate nonce against latest state for a trace of a transaction that's already happened in the past? As an example, for the following trace_transaction call on Ethereum Mainnet, Reth responds with "nonce too low" error, while the request goes through on other clients (e.g. Erigon).

{
    "jsonrpc": "2.0",
    "method": "trace_transaction",
    "params": [
        "0xbfbc675c06af4ed5ebdf35935fb506a41b0c4e1d382db39bbe1fa2ba528bdbb4"
    ],
    "id": 58938095
}

cakevm · 2024-09-26T10:50:09Z

@erwin-wee I just like to mention that with version 1.0.7 (75b7172c) the response from reth and erigon is the same for your request.

mattsse · 2024-09-26T12:03:04Z

I'm optimistically closing this,
in case there are still inconsistencies, we'll track separately once flagged

tjayrush · 2024-09-26T14:32:03Z

Sounds good. We're finally syncing Reth on our machines. Any discrepancies will appear naturally once we start testing our code against Reth. I'll post issues if we find any.

…

On Thu, Sep 26, 2024 at 8:03 AM Matthias Seitz ***@***.***> wrote: Closed #3721 <#3721> as completed. — Reply to this email directly, view it on GitHub <#3721 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJKXPR22EVQELBJW2TIUXTZYPZY3AVCNFSM6AAAAAA2GGAFS2VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGQYTKMRQGUYDANI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Thomas Jay Rush http://t <http://quickblocks.io>rueblocks.io ***@***.***

wakamex added C-bug An unexpected or incorrect behavior S-needs-triage This issue needs to be labelled labels Jul 11, 2023

github-project-automation bot added this to Reth Tracker Jul 11, 2023

github-project-automation bot moved this to Todo in Reth Tracker Jul 11, 2023

mattsse added S-needs-investigation This issue requires detective work to figure out what's going wrong A-rpc Related to the RPC implementation and removed S-needs-triage This issue needs to be labelled labels Jul 11, 2023

mattsse self-assigned this Jul 11, 2023

mattsse mentioned this issue Jul 12, 2023

Rename parity selfdestruct action to suicide #3734

Closed

1 task

wakamex mentioned this issue Jul 12, 2023

add diff script and results TrueBlocks/trueblocks-core#3037

Merged

mattsse mentioned this issue Jul 13, 2023

fix(rpc): disable balance check if call gas price is 0 #3760

Closed

This was referenced Jul 14, 2023

fix(rpc): serialize traces always as vec #3770

Merged

fix(rpc): make trace filter req field hex or decimal #3772

Merged

sslivkoff mentioned this issue Jul 14, 2023

continuation of previous discrepancy tests #3782

Open

1 task

mattsse mentioned this issue Jul 19, 2023

fix(rpc): trace get is index not trace address #3852

Merged

onbjerg self-assigned this Apr 2, 2024

mattsse closed this as completed Sep 26, 2024

github-project-automation bot moved this from Todo to Done in Reth Tracker Sep 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

trace compatibility testing with Trueblocks #3721

trace compatibility testing with Trueblocks #3721

wakamex commented Jul 11, 2023 •

edited

Loading

mattsse commented Jul 11, 2023

tjayrush commented Jul 11, 2023

mattsse commented Jul 12, 2023 •

edited

Loading

mattsse commented Jul 12, 2023 •

edited

Loading

wakamex commented Jul 12, 2023 •

edited

Loading

tjayrush commented Jul 12, 2023

tjayrush commented Jul 12, 2023

mattsse commented Jul 12, 2023

mattsse commented Jul 12, 2023

tjayrush commented Jul 12, 2023

gakonst commented Jul 12, 2023

wakamex commented Jul 12, 2023

tjayrush commented Jul 13, 2023

mattsse commented Jul 14, 2023

gakonst commented Jul 14, 2023

wakamex commented Jul 14, 2023 •

edited

Loading

tjayrush commented Jul 14, 2023 •

edited

Loading

mattsse commented Jul 19, 2023

gakonst commented Jul 20, 2023

tjayrush commented Jul 21, 2023

yabirgb commented Jul 28, 2023 •

edited

Loading

mattsse commented Jul 28, 2023

tjayrush commented Aug 7, 2023

mattsse commented Aug 7, 2023

onbjerg commented Apr 2, 2024

erwin-wee commented Jun 28, 2024 •

edited

Loading

cakevm commented Sep 26, 2024

mattsse commented Sep 26, 2024

tjayrush commented Sep 26, 2024 via email

trace compatibility testing with Trueblocks #3721

trace compatibility testing with Trueblocks #3721

Comments

wakamex commented Jul 11, 2023 • edited Loading

Describe the bug

Steps to reproduce

Node logs

Platform(s)

What version/commit are you on?

What database version are you on?

If you've built Reth from source, provide the full command you used

Code of Conduct

mattsse commented Jul 11, 2023

tjayrush commented Jul 11, 2023

mattsse commented Jul 12, 2023 • edited Loading

mattsse commented Jul 12, 2023 • edited Loading

wakamex commented Jul 12, 2023 • edited Loading

tjayrush commented Jul 12, 2023

tjayrush commented Jul 12, 2023

mattsse commented Jul 12, 2023

mattsse commented Jul 12, 2023

tjayrush commented Jul 12, 2023

gakonst commented Jul 12, 2023

wakamex commented Jul 12, 2023

tjayrush commented Jul 13, 2023

mattsse commented Jul 14, 2023

gakonst commented Jul 14, 2023

wakamex commented Jul 14, 2023 • edited Loading

tjayrush commented Jul 14, 2023 • edited Loading

mattsse commented Jul 19, 2023

gakonst commented Jul 20, 2023

tjayrush commented Jul 21, 2023

yabirgb commented Jul 28, 2023 • edited Loading

mattsse commented Jul 28, 2023

tjayrush commented Aug 7, 2023

mattsse commented Aug 7, 2023

onbjerg commented Apr 2, 2024

erwin-wee commented Jun 28, 2024 • edited Loading

cakevm commented Sep 26, 2024

mattsse commented Sep 26, 2024

tjayrush commented Sep 26, 2024 via email

wakamex commented Jul 11, 2023 •

edited

Loading

mattsse commented Jul 12, 2023 •

edited

Loading

mattsse commented Jul 12, 2023 •

edited

Loading

wakamex commented Jul 12, 2023 •

edited

Loading

wakamex commented Jul 14, 2023 •

edited

Loading

tjayrush commented Jul 14, 2023 •

edited

Loading

yabirgb commented Jul 28, 2023 •

edited

Loading

erwin-wee commented Jun 28, 2024 •

edited

Loading