Chainlink node unable to start after backfill fails

[boxhock]

System Information

• Commit (INFO line when starting the node): ce1acc5

Additional Information

If the Chainlink node attempts to backfill logs, but fails, it will be unable to properly start again.

Steps to recreate:

1. Start a Chainlink node with a large backlog, which the ETH node will be unable to respond to. Optimally you would want the ETH node to be on cheap hardware too (larger chance of crashing), or straight up route the request through a reverse proxy that will imitate a failed request/error. Even just stop the ETH node while the CL node is getting the backlog?
2. Expect following error: [ERROR] Unable to backfill logs services/subscription.go:346
3. Restart CL node
4. CL node will get ETH and LINK balances, but not process anything further. (No new headers being shown in console)

Every time it's restarted, it stops at balances. It definitely is receiving response for getLogs and getting new headers, but nothing is shown in the console.

Also: not sure if this is expected behavior, but when restarting the CL node, it's not attempting to backfill with the same block number that failed previously, but using the latest block number instead. Should it retry to backfill from the last successful block number?

[se3000]

Thanks for reporting. Tracking this here: https://www.pivotaltracker.com/story/show/163037084

[se3000]

@boxhock we were having trouble reproing this before version-0.6. I imagine it may have disappeared with that update. Can you still reproduce this?

[boxhock]

@se3000 I did an effort to help John reproduce this issue, using the exact same steps as I followed when I initially encountered this issue, without any results. I'm going to assume the issue has been resolved with 0.6.