Excessive out of sync time at epoch boundary #889
Description
At the start of epoch 297:
[2021-10-17 21:43:32.18 UTC] insertShelleyBlock(Alonzo): epoch 296, slot 42940718, block 6384036, hash 66c251e4e66ce224f4184c9bfcce5870b19512651e7fbf0c611932fca2fddf07
[2021-10-17 21:44:18.95 UTC] insertShelleyBlock(Alonzo): epoch 296, slot 42940763, block 6384037, hash cd155f53de7d3985194c73540daf2e1dc79d5f2bcd09e9c6c5830c093fb34ba8
[2021-10-17 21:58:44.07 UTC] Took a ledger snapshot at ledger-state/mainnet/42940763-cd155f53de-296.lstate
[2021-10-17 21:58:44.78 UTC] Removing epoch boundary files ["ledger-state/mainnet/42076780-dbeeb69efb-294.lstate"]
[2021-10-17 21:59:17.14 UTC] adjustEpochRewards: epoch 295, 2398 orphaned rewards removed (5698.979449 ADA)
[2021-10-17 21:59:48.38 UTC] Starting epoch 297
[2021-10-17 21:59:49.48 UTC] Handling 896871 stakes for epoch 297 slot 42940975, hash a6576ce5bd4cf7ac7e0c198453703f2b0fe50db68be1787d1c22fae4f1b60109
[2021-10-17 22:01:23.48 UTC] validateEpochRewards: total rewards that become spendable in epoch 297 is 15096030.98095 ADA
[2021-10-17 22:19:15.95 UTC] insertShelleyBlock(Alonzo): epoch 297, slot 42940975, block 6384038, hash a6576ce5bd4cf7ac7e0c198453703f2b0fe50db68be1787d1c22fae4f1b60109
[2021-10-17 22:19:40.85 UTC] Rolling back to slot 42940763, hash cd155f53de7d3985194c73540daf2e1dc79d5f2bcd09e9c6c5830c093fb34ba8
[2021-10-17 22:19:40.85 UTC] Deleting 1 blocks up to slot 42940975
[2021-10-17 22:19:42.90 UTC] Blocks deleted
[2021-10-17 22:19:42.91 UTC] Found in memory ledger snapshot at slot 42940763, hash cd155f53de7d3985194c73540daf2e1dc79d5f2bcd09e9c6c5830c093fb34ba8
[2021-10-17 22:20:23.00 UTC] Flushing remaining 897 BulkOperations
[2021-10-17 23:01:42.24 UTC] insertEpochInterleaved: Epoch 297, 896871 stake addresses
[2021-10-17 23:01:42.40 UTC] File ledger-state/mainnet/42940763-cd155f53de-296.lstate exists
[2021-10-17 23:02:48.38 UTC] validateEpochRewards: total rewards that become spendable in epoch 297 is 15096030.98095 ADA
[2021-10-17 23:19:02.06 UTC] insertShelleyBlock(Alonzo): epoch 297, slot 42942773, block 6384114, hash 3bb4e0159c6c28f12e587caa682d92e7af1ba8ba60dfe8ce1530f2d0cb325ff9
and:
> select now (), now () at time zone 'utc' - max (time) as behind_by, max (epoch_no) as epoch_no from block ;
now | behind_by | epoch_no
-------------------------------+-----------------+----------
2021-10-18 10:18:03.778522+11 | 01:30:48.778522 | 297During most of this time the node was following the chain tip.
Obviously, db-sync being AWOL for over 1.5 hours is not acceptable.