Postgres serialization issue after multiple permanent live streams re-starting simultaneously

[infinitron]

Describe the current behavior

My peertube instance drops a few live streams when many live streams start simultaneously. I have a peertube instance with six other nodes, each writing to a permanent live stream using an ffmpeg command. For my use case, I restart the live streams once every 24hrs to post a daily replay. When the live streams start at the same time, I get the following error:

error[5/11/2024, 8:36:23 PM] Cannot handle sessions.
{
  "err": {
    "stack": "Error\n    at Query.run (/var/www/peertube/versions/peertube-v6.0.2/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)\n    at /var/www/peertube/versions/peertube-v6.0.2/node_modules/sequelize/lib/sequelize.js:315:28\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async PostgresQueryInterface.insert (/var/www/peertube/versions/peertube-v6.0.2/node_modules/sequelize/lib/dialects/abstract/query-interface.js:308:21)\n    at async VideoLiveSessionModel.save (/var/www/peertube/versions/peertube-v6.0.2/node_modules/sequelize/lib/model.js:2490:35)\n    at async VideoLiveSessionModel.create (/var/www/peertube/versions/peertube-v6.0.2/node_modules/sequelize/lib/model.js:1362:12)\n    at async /var/www/peertube/versions/peertube-v6.0.2/node_modules/sequelize/lib/sequelize.js:507:18\n    at async LiveManager.runMuxingSession (file:///var/www/peertube/versions/peertube-v6.0.2/dist/core/lib/live/live-manager.js:215:29)",
    "message": "could not serialize access due to read/write dependencies among transactions",
    "name": "SequelizeDatabaseError",
    "parent": {
      "stack": "error: could not serialize access due to read/write dependencies among transactions\n    at Parser.parseErrorMessage (/var/www/peertube/versions/peertube-v6.0.2/node_modules/pg-protocol/dist/parser.js:287:98)\n    at Parser.handlePacket (/var/www/peertube/versions/peertube-v6.0.2/node_modules/pg-protocol/dist/parser.js:126:29)\n    at Parser.parse (/var/www/peertube/versions/peertube-v6.0.2/node_modules/pg-protocol/dist/parser.js:39:38)\n    at Socket.<anonymous> (/var/www/peertube/versions/peertube-v6.0.2/node_modules/pg-protocol/dist/index.js:11:42)\n    at Socket.emit (node:events:514:28)\n    at addChunk (node:internal/streams/readable:324:12)\n    at readableAddChunk (node:internal/streams/readable:297:9)\n    at Readable.push (node:internal/streams/readable:234:10)\n    at TCP.onStreamRead (node:internal/stream_base_commons:190:23)",
      "message": "could not serialize access due to read/write dependencies among transactions",
      "length": 403,
      "name": "error",
      "severity": "ERROR",
      "code": "40001",
      "detail": "Reason code: Canceled on conflict out to pivot 15441, during read.",
      "hint": "The transaction might succeed if retried.",
      "where": "SQL statement \"SELECT 1 FROM ONLY \"public\".\"videoLiveReplaySetting\" x WHERE \"id\" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x\"",
      "file": "predicate.c",
      "line": "4610",
      "routine": "OnConflict_CheckForSerializationFailure",
      "sql": "INSERT INTO \"videoLiveSession\" (\"id\",\"startDate\",\"saveReplay\",\"endingProcessed\",\"liveVideoId\",\"replaySettingId\",\"createdAt\",\"updatedAt\") VALUES (DEFAULT,$1,$2,$3,$4,$5,$6,$7) RETURNING \"id\",\"startDate\",\"endDate\",\"error\",\"saveReplay\",\"endingProcessed\",\"replayVideoId\",\"liveVideoId\",\"replaySettingId\",\"createdAt\",\"updatedAt\";",
      "parameters": [
        "2024-05-12 03:36:23.553 +00:00",
        true,
        false,
        13,
        458,
        "2024-05-12 03:36:23.553 +00:00",
        "2024-05-12 03:36:23.553 +00:00"
      ]
    },
    "sql": "INSERT INTO \"videoLiveSession\" (\"id\",\"startDate\",\"saveReplay\",\"endingProcessed\",\"liveVideoId\",\"replaySettingId\",\"createdAt\",\"updatedAt\") VALUES (DEFAULT,$1,$2,$3,$4,$5,$6,$7) RETURNING \"id\",\"startDate\",\"endDate\",\"error\",\"saveReplay\",\"endingProcessed\",\"replayVideoId\",\"liveVideoId\",\"replaySettingId\",\"createdAt\",\"updatedAt\";"
  },
  "tags": [
    "live",
    "9OTATJD7"
  ]
}

The following warning shows up after this error:

warn[5/11/2024, 8:54:57 PM] Video <video uuid> has already a live session. Refusing stream <stream key>.
{
  "tags": [
    "live",
    "---",
    "<video uuid>"
  ]
}

Each live stream is written by a ffmpeg call, which throws an input/output error first, followed by connection reset by peer. I'm starting this issue after the discussion in #3607.

Steps to reproduce

1. Setup multiple permanent live streams
2. Start writing to all the live streams at the same time.
3. Restart writing to the live streams at the same time

Describe the expected behavior

The peertube instance should drop the old video session and start a new session for the live stream.

Additional information

• PeerTube instance:

  • URL: https://livetv.epic-astronomy.org
  • Version: 6.0.2
  • NodeJS version: 18.0
  • Ffmpeg version: 4.4.2

• Browser name, version, and platforms on which you could reproduce the bug: Edge, 124.0.2478.97, Ubuntu 22.04

• Link to browser console log if relevant:

• Link to server log if relevant (journalctl or /var/www/peertube/storage/logs/):

[Chocobozzz]

Thanks! I think it will be fixed by 3d10529

(don't hesitate to comment if the next release doesn't fix your issue ;) )