Attempt to avoid losing subscribers

[tfausak]

Haskell Weekly has sadly lost its subscriber list again. (Last time: https://taylor.fausak.me/2026/03/25/haskell-weekly/)

Fortunately this time I have backups. The backup from 2026-07-02 is fine. The one from 2026-07-04 is missing all the subscribers, which suggests the database was initialized from scratch. I find it unlikely that a managed PostgreSQL instance on Fly would randomly lose data. Perhaps this upgrade is to blame. However I don't have any concrete data to suggest anything one way or the other. This is just a hunch.

[tfausak]

Note that listmonk --upgrade is meant to migrate the schema from an old version to a new one. My intent with adding this in the first place was to make upgrades as hands off as possible. I would simply deploy the new version and the Dockerfile would automatically do everything that I would do manually.

It's possible that bad things can happen when upgrading with another instance of the app running. The docs explicitly call this out: https://listmonk.app/docs/upgrade/

However I haven't actually upgraded anything (see #20), so listmonk --upgrade should be idempotent.

[tfausak]

This is the only other bit that could be causing problems. (At least from the Listmonk side of things.) But it should be harmless:

      --idempotent          make --install run only if the database isn't already setup
      --install             setup database (first time)
      --yes                 assume 'yes' to prompts during --install/upgrade

[tfausak]

Unfortunately I'm not able to figure out what happened here. Subscribers were lost sometime between 2026-07-02 and 2026-07-04. Logs in Fly only go back 7 days, which currently lands in the middle of 2026-07-03 UTC. I don't see anything suspicious in those logs.

I have not made any deploys. I have not changed anything in Fly. In fact, I've been out of town for the past week and wasn't even thinking about Haskell Weekly at all during that time. So I am sure that this problem was not caused by some action that I took.

Obviously an issue of the newsletter was published (haskellweekly/haskellweekly#566), but that shouldn't be able to cause a problem like this. Plus the timelines don't match up: newsletters were published on the 2nd and the 9th, but data was lost sometime between the 2nd and the 4th.

[tfausak]

I don't think --upgrade caused this problem in the first place, and I don't think that removing it would prevent this from happening in the future.

No upgrade happened at all. No new version was deployed. So the upgrade command was a no-op.

The problem must have happened before the upgrade. I suppose it's possible that --install --idempotent somehow isn't always idempotent, and that caused the problem. Nothing else really makes sense to me. The only other moving part is the PostgreSQL database itself. I'm using Fly's managed Postgres, and it would be very unusual for it to simply drop data.

Another possible cause is something nefarious like an attack, but that doesn't make sense to me. Nobody has reached out with any demands, I haven't seen any other unusual behavior, and this has happened twice at seemingly random intervals.

My plan is to restore the latest backup with subscribers, then merge in any new subscribers that subscribed after that.

Longer term, I need a way to proactively notice that this happened so that I can look at the logs and figure out what went wrong.

[Copilot]

Pull request overview

This PR adjusts the Fly deployment approach so the app no longer auto-installs or auto-migrates the database schema on container boot, aiming to prevent a blank database from being silently initialized (which could mask data loss and lead to an “empty” subscriber list).

Changes:

• Move schema migrations to a Fly release_command that runs once per deploy.
• Make container boot “serve-only” (no install/upgrade on startup).
• Document manual, deliberate database installation steps and the rationale.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
README.md	Documents the new DB install/migration operational workflow and rationale.
fly.toml	Adds a deploy release_command to run listmonk --upgrade once per deploy.
Dockerfile	Removes install/upgrade-on-boot; runs only the listmonk server.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tfausak]

• I exported the subscribers from the current database. This was done through the Listmonk web UI.
• I restored the backup from 2026-07-02. This was just changing some environment variables to point to the restored instance rather than the original one.
• I imported the subscribers I just exported. This was to capture any new subscribers since the data loss.
• Since the backup was from before 2026-07-09, a new campaign was created to send issue 532 of the newsletter. It will go out in a few minutes.

With these changes, everything should be back to normal.

[tfausak]

For final cleanup, I took snapshots of both the old and new databases. Then I deleted the old one. (It'll hang around for another 30 days.) Everything has already been migrated to the new database. Since both install and upgrade commands have been moved out of the start command, it's unlikely that this will happen again. However, if it does, then I can restore from backup again just like I did this time.