Receipt could not be found when calling trace_transaction

[Uxio0]

Description
Receipt could not be found when calling trace_transaction

Steps to Reproduce
Send this payload to the RPC {'id': 0, 'jsonrpc': '2.0', 'method': 'trace_transaction', 'params': ['0x78738a28e62b9ab4250b0fa3baaaa93961b2d657ae17fb9e4adb49cebde7b350']}

Actual behavior
We receive: {'jsonrpc': '2.0', 'error': {'code': -32001, 'message': '0x78738a28e62b9ab4250b0fa3baaaa93961b2d657ae17fb9e4adb49cebde7b350 receipt could not be found'}, 'id': 0}

Expected behavior
Get traces for transaction: https://gnosisscan.io/tx/0x78738a28e62b9ab4250b0fa3baaaa93961b2d657ae17fb9e4adb49cebde7b350

Additional context
Nethermind version: v1.19.3
Chain: Gnosis Chain

[emlautarom1]

On Nethermind/v1.19.3+e8ac1da4/linux-x64/dotnet7.0.8 provided by Ankr (https://rpc.ankr.com/gnosis) I get the correct trace for this particular transaction. Other RPC providers also using Nethermind fail with a TrieException, and locally I get a "receipt could not be found" on Nethermind/v1.21.0-unstable+a35f97da/linux-x64/dotnet7.0.7.

Do you have your client fully synced in? What CL are you using alongside Nethermind?

[luarx]

Yes @emlautarom1 , we have a Nethermind client fully synced and as CL we are using Lighthouse v4.3.0

Yesterday after opening the issue we upgraded from v1.19.3 to v1.20.0, but same issue around

[emlautarom1]

I'm able to reproduce the issue with the following flags: --Sync.NonValidatorNode=true --Sync.DownloadBodiesInFastSync=false --Sync.DownloadReceiptsInFastSync=false; I'm getting the exact same response as you.

Could you please list the flags that you're using with Nethermind?

[LukaszRozmej]

@Uxio0 @luarx looks like by some unforeseen circumstance your receipt DB got corrupted (it is also where we keep tx hash->blok number index). I think best way to proceed is to try running a process that will scan all receipts and download any missing from the network. You can run it by restarting the node with the flag --Sync.FixReceipts true as it is an option here:
https://docs.nethermind.io/nethermind/ethereum-client/configuration/sync

In some time it should scan through all receipts and download missing ones. It should output logs for every block with missing receipt it finds.

[diegopazosrego]

ok @LukaszRozmej ! We enable this flag --Sync.FixReceipts true and we are checking now the receipts. We wil inform you when the process finish if its working fine or not.

[Uxio0]

After running the node with the flag everything is working as expected. We are still worried about how multiple nodes got into that state, though

Thank you very much @LukaszRozmej