Long running schema deltas are killed by systemd and lead to bootlooping

[matrixbot]

This issue has been migrated from #14401.

---

When a schema delta takes too long, Synapse will get restarted by systemd, possibly repeatedly.

~~When using postgres, schema deltas are transactional. Synapse will run the schema delta and time out again. ~~
When using sqlite, schema deltas are not fully transactional (see docstring here). The schema delta will end up being half applied and the database will end up in an inconsistent state. Depending on the nature of the schema delta, the next run may fail. Tracked in #14909.

This manifested as:

• Dummy issue #13193
• [sqlite] Upgrade to 1.68 leads to bootlooping Synapse systemd service #14100
• Dummy issue #14333

Things we may consider doing:

• Write some docs?
• Increase the timeout?
• Pretend to be alive while running schema deltas?
• dmr: Make sqlite schema deltas transactional too? Tracked in Dummy issue #14909.