State root mismatches (both calibnet and mainnet)

[elmattic]

Describe the bug

Encountering some state mismatch while validating tipsets. This happens both on calibnet and mainnet.

To Reproduce

• Import snapshot from right before one of the affected epochs.
• Wait for Forest to attempt to validate the epoch and then fail.

The following command will trigger the bug:

forest --no-gc --chain calibnet\
   --encrypt-keystore false \
   --import-snapshot https://forest-snapshots.fra1.cdn.digitaloceanspaces.com/debug/filecoin_snapshot_calibnet_2023-06-13_height_643680.car.zst \
   --height=643585

Failure epochs

• Calibnet: 632281
• Calibnet: 643586
• Mainnet: 2942404
• Mainnet: 2949733

Log output

2023-06-09T08:03:58.250789Z  WARN forest_chain_sync::tipset_syncer: Validating block [CID = bafy2bzacecqyzrmtd26wgtyfmklryxvx7dvwlvnkrjpkffzft6iz7mhju74pm] in EPOCH = 632281 failed: Validation error: Validation error: Parent state root did not match computed state: bafy2bzaceddo3ryqvrb5tvznyufsqbe454c64vspiwb3ueohhsk3srnzgrdpq (header), bafy2bzaceaekjcuofhgotm5r7grstipv5mm44oycrvndodkeiuqwj6v2nn342 (computed)    
2023-06-09T08:03:58.250914Z ERROR forest_chain_sync::tipset_syncer: Sync messages check state failed for tipset range    
2023-06-09T08:03:58.250922Z ERROR forest_chain_sync::chain_muxer: Bootstrapping failed, re-evaluating the network head to retry the bootstrap. Error = TipsetRangeSyncer(Validation("Validation error: Parent state root did not match computed state: bafy2bzaceddo3ryqvrb5tvznyufsqbe454c64vspiwb3ueohhsk3srnzgrdpq (header), bafy2bzaceaekjcuofhgotm5r7grstipv5mm44oycrvndodkeiuqwj6v2nn342 (computed)"))    

Environment:

• Branch: main
• Snapshot: forest_snapshot_calibnet_2023-06-09_height_632266.car

[lemmih]

I can reproduce it.

[lemmih]

Uploaded a snapshot that contains the correct state tree. It makes it possible to find the diff: https://forest-snapshots.fra1.cdn.digitaloceanspaces.com/debug/632400_2023_06_09T08_13_00Z.car.zst

[lemmih]

Will upload temporary snapshots from Lotus to avoid breaking our CI setup.

[lemmih]

State diff:

Address f021018 changed: 
 ActorState(ActorState { code: Cid(bafk2bzaceabftmhejmvjvpzmbsv4cvaew6v5juj5sqtq7cfijugwsnahnsy5w)
- state: Cid(bafy2bzacea4vsbsolom2ykhdzcfoo2cl7pgngjcru72s43g6fakchtraeqmkc)
+ state: Cid(bafy2bzacedqrvubrbmmy3qtuxstgki5wvahckhdlncj54wooftcm5egdddxki)
  sequence: 0
  balance: TokenAmount(0.0)
  delegated_address: Some(snip)
V11(State { bytecode: Cid(bafk2bzacec5zpp6524sbdvmxsdw2b2n3am2ohxaxjmtpplyspehkj2ywn7pr4)
  bytecode_hash: BytecodeHash(4052AD80BC433CB6B163155FC7E49A838869EAFB5EC4E705450BEFAC2E2F2D88)
- contract_state: Cid(bafy2bzacea7t237lqfedxo5goabrgtvq5hhfhtqjelcjh7jghueeafkxk4z6e)
+ contract_state: Cid(bafy2bzacec763brjalwj3w562mq5kfgpygtnzoa3xkyz75gb6qkjbfzofy5so)
  nonce: 1
  tombstone: None })
Address f099 changed: 
 ActorState(ActorState { code: Cid(bafk2bzacebor5mnjnsav34cmm5pcd3dy4wubbv4wtcrvba7depy3sct7ie4sy)
  state: Cid(bafy2bzacedpuk5ggwoq3s2wixsyjjnexpsjstdlyntio76vs2lt2jvy3o6mau)
  sequence: 0
- balance: TokenAmount(8097143.09617497843333088)
+ balance: TokenAmount(8097143.09617497843190788)
  delegated_address: None })
V11(State { address: Address { payload: ID(99) } })
Address f019260 changed: 
 {
  "code": {
    "/": "bafk2bzacedrbpvjvyzif2cjxosm4pliyq2m6wzndvrg7r6hzdhixplzvgubbw"
  }
 
  "sequence": 4
-  "balance": "89.950857469415656869"
+  "balance": "89.950857469417079869"
 
  "state": []
}

Looks like a problem with the way we handle eth contracts.

[elmattic]

Nice finding. Maybe related to #2916?

[lemmih]

I hope it is related. If it's the same bug then we only have to fix it once. :)

[lemmih]

Another mismatch at epoch 643586.

[elmattic]

I will have a look.