Processor crashed with missing blocks despite correct DB entries

[poka-IT]

My subsquid node processor crashed with this logs:

duniter-squid-processor-1     | {"level":2,"time":1709683176077,"ns":"sqd:processor","msg":"435159 / 435159, rate: 0 blocks/sec, mapping: 85 blocks/sec, 254 items/sec, eta: 0s"}
duniter-squid-processor-1     | {"level":2,"time":1709683182174,"ns":"sqd:processor","msg":"435160 / 435160, rate: 0 blocks/sec, mapping: 86 blocks/sec, 259 items/sec, eta: 0s"}
duniter-squid-processor-1     | {"level":2,"time":1709683188188,"ns":"sqd:processor","msg":"435161 / 435161, rate: 0 blocks/sec, mapping: 91 blocks/sec, 274 items/sec, eta: 0s"}
duniter-squid-processor-1     | {"level":2,"time":1709683194091,"ns":"sqd:processor","msg":"435162 / 435162, rate: 0 blocks/sec, mapping: 90 blocks/sec, 270 items/sec, eta: 0s"}
duniter-squid-processor-1     | {"level":2,"time":1709683200099,"ns":"sqd:processor","msg":"435163 / 435163, rate: 0 blocks/sec, mapping: 90 blocks/sec, 269 items/sec, eta: 0s"}
duniter-squid-processor-1     | {"level":2,"time":1709683608909,"ns":"sqd:processor","msg":"435172 / 435172, rate: 0 blocks/sec, mapping: 138 blocks/sec, 415 items/sec, eta: 0s"}
duniter-squid-processor-1     | {"level":5,"time":1709683609149,"ns":"sqd:processor","err":{"generatedMessage":true,"code":"ERR_ASSERTION","actual":false,"expected":true,"operator":"==","stack":"AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:\n\n  (0, assert_1.default)(head.height >= this.chain[0].height)\n\n    at HotProcessor.moveToBlocks (/squid/node_modules/@subsquid/util-internal-ingest-tools/lib/hot.js:60:30)\n    at HotProcessor.goto (/squid/node_modules/@subsquid/util-internal-ingest-tools/lib/hot.js:38:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async RpcDataSource.handleNewHeads (/squid/node_modules/@subsquid/substrate-data-raw/lib/datasource.js:213:24)\n    at async RpcDataSource.subscription (/squid/node_modules/@subsquid/substrate-data-raw/lib/datasource.js:198:25)\n    at async RpcDataSource.processHotBlocks (/squid/node_modules/@subsquid/substrate-data-raw/lib/datasource.js:113:13)"}}

Error: [ERR_ASSERTION]: The expression evaluated to a falsy value:\n\n (0, assert_1.default)(head.height >= this.chain[0].height)

It look likes it missed 8 higher blocks, between 435163 and 435172, which is the cause of this error.

This indexer is listening a local duniter v2s archive node.

indexer enpoind: https://gdev-squid.axiom-team.fr/v1/graphql

duniter endpoint: wss://gdev.p2p.legal/ws

---

We observe that the problem occurs exactly at midnight, which would be a strange coincidence:

Bloc 435160 : 2024-03-05 23:59:42
Bloc 435161 : 2024-03-05 23:59:48
Bloc 435162 : 2024-03-05 23:59:54
Bloc 435163 : 2024-03-06 00:00:00
Bloc 435172 : 2024-03-06 00:06:48

We also notice a gap of 6 minutes and 48 seconds between block 435163 and 435172, whereas only 48 seconds should have passed.

---

Disturbingly, the missing blocks in the logs seem to be present in the DB, and correctly dated:

{
  "data": {
    "block": [
      {
        "height": 435172,
        "timestamp": "2024-03-06T00:01:06+00:00"
      },
      {
        "height": 435171,
        "timestamp": "2024-03-06T00:01:00+00:00"
      },
      {
        "height": 435170,
        "timestamp": "2024-03-06T00:00:54.001+00:00"
      },
      {
        "height": 435169,
        "timestamp": "2024-03-06T00:00:48.002+00:00"
      },
      {
        "height": 435168,
        "timestamp": "2024-03-06T00:00:42+00:00"
      },
      {
        "height": 435167,
        "timestamp": "2024-03-06T00:00:24+00:00"
      },
      {
        "height": 435166,
        "timestamp": "2024-03-06T00:00:18+00:00"
      },
      {
        "height": 435165,
        "timestamp": "2024-03-06T00:00:12.001+00:00"
      },
      {
        "height": 435164,
        "timestamp": "2024-03-06T00:00:06.001+00:00"
      },
      {
        "height": 435163,
        "timestamp": "2024-03-06T00:00:00.001+00:00"
      },
      {
        "height": 435162,
        "timestamp": "2024-03-05T23:59:54+00:00"
      },
      {
        "height": 435161,
        "timestamp": "2024-03-05T23:59:48+00:00"
      },
      {
        "height": 435160,
        "timestamp": "2024-03-05T23:59:42.001+00:00"
      },
      {
        "height": 435159,
        "timestamp": "2024-03-05T23:59:36.001+00:00"
      },
      {
        "height": 435158,
        "timestamp": "2024-03-05T23:59:30.001+00:00"
      }
    ]
  }
}

Here a simple processor restart solve the problem, but whatever the cause of the problem, processor should be able to recover using its checkpoints, without manual restart.

Version: "@subsquid/substrate-processor": "^8.1.1"

[belopash]

Not sure that caused the error, but blocks weren't missed. More looks like there was some issue on the rpc node side due to that it was unable to return new blocks for a certain period of time, and these 8 blocks weren't printed in the logs because only the last block of the batch is printed.

Regarding to the error itself, it seems that there was a fork/rollback, but processor didn't handle it correctly, will investigate.