PDP: addRoots intermittently fails - chain watcher stops processing transactions

[rjan90]

Description:

Curio daemon intermittently stops processing root additions to proof sets, causing StorageService addRoots operations to fail with timeout errors. The transactions are confirmed on-chain but the server fails to acknowledge them.

StorageService addRoots failed: Failed to add root to proof set
Caused by: Failed to verify root addition after 420 seconds: Server did not provide confirmation. The transaction was confirmed on-chain but the server failed to acknowledge it.

We have logs (stderr and stdout), but can’t find any explanation for the chain watcher stopping watching and processing these root adds.

SELECT
       add_message_ok,
       roots_added,
       COUNT(*)
   FROM curio.pdp_proofset_root_adds
   GROUP BY add_message_ok, roots_added;
+----------------+-------------+-------+
| add_message_ok | roots_added | count |
|----------------+-------------+-------|
| <null>         | False       | 14    |
| True           | True        | 12    |
+----------------+-------------+-------+

Temporary Workaround:

• Restarting the Curio daemon process resolves the issue temporarily
• Issue is likely to recur

Investigation Notes:

• Memory issues suspected but adding 32GB swap didn't resolve it
• Need to catch the exact moment when chain watcher stops to narrow down timing for log analysis

[rvagg]

We seem to have a similar problem with proofset creation, the same server now has a stuck transaction in pdp_proofset_creates, with ok=null and proofset_created=false. The transaction is visible on chain and was successful. Restarting the server doesn't appear to flush this one out, it's just stuck.

[rvagg]

Debugging datapoint: for the proofset create I can see this:

> select * from curio.message_waits_eth where signed_tx_hash = '0x3005f4a91b26648cf54d94933d147798885aa38cce2dc6cdebb66f53693cc75d';
+--------------------------------------------------------------------+-------------------+------------------------+-------------------+-------------------+-----------+------------+------->
| signed_tx_hash                                                     | waiter_machine_id | confirmed_block_number | confirmed_tx_hash | confirmed_tx_data | tx_status | tx_receipt | tx_suc>
|--------------------------------------------------------------------+-------------------+------------------------+-------------------+-------------------+-----------+------------+------->
| 0x3005f4a91b26648cf54d94933d147798885aa38cce2dc6cdebb66f53693cc75d | 1                 | <null>                 | <null>            | <null>            | pending   | <null>     | <null>>
+--------------------------------------------------------------------+-------------------+------------------------+-------------------+-------------------+-----------+------------+------->
SELECT 1

with no change in status, in fact there's a lot of pending transactions in there which is odd.

Running this to reset it to get it reassigned and re-watched:

update curio.message_waits_eth set waiter_machine_id = null where signed_tx_hash = '0x3005f4a91b26648cf54d94933d147798885aa38cce2dc6cdebb66f53693cc75d';

But it then quickly get's reassigned back to waiter_machine_id=1, presumably by this:

curio/tasks/message/watch_eth.go

Lines 75 to 86 in d359767

	// Assign pending transactions with null owner to ourselves
	{
	n, err := mw.db.Exec(ctx, `UPDATE message_waits_eth SET waiter_machine_id = $1 WHERE waiter_machine_id IS NULL AND tx_status = 'pending'`, machineID)
	if err != nil {
	log.Errorf("failed to assign pending transactions: %+v", err)
	return
	}
	if n > 0 {
	log.Debugw("assigned pending transactions to ourselves", "assigned", n)
	}
	}

And then nothing. Still stuck. Restarting Curio doesn't budge it. Something about this transaction in watch_eth.go stops it proceeding, or something about the message status lookup stops it being registered as successful.

[rjan90]

I think I have a relatively easy way to replicate the StorageService addRoots failed log on the client side at leat. I used a fresh wallet in MetaMask, added some USDFC and Calibnet FIL to it - and then tried to upload a file:

1. The create proofset transactions gets to the "Confirmed On-Chain" status, but then just waits there indefinitely. This is most likely because the datapoint Rod has above about that the transaction does not change its status.

The two client error you can encounter from here on, on the client side is: Failed to create storage service because eventually it times out:

If you refresh, and then try to upload another file - you will encounter the Upload failed: StorageService addRoots failed: Failed to add root to proof set failure:

[rvagg]

#549 🤞 ; this seems to have flushed out the problem transactions on polynomial.computer at least (for now).

[rjan90]

Uploading worked again now at least: