Skip to content

migrate-db: warn loudly before writing irreversible tombstones#86

Open
0xfandom wants to merge 1 commit intolightninglabs:mainfrom
0xfandom:fix/81-tombstone-warning
Open

migrate-db: warn loudly before writing irreversible tombstones#86
0xfandom wants to merge 1 commit intolightninglabs:mainfrom
0xfandom:fix/81-tombstone-warning

Conversation

@0xfandom
Copy link
Copy Markdown

@0xfandom 0xfandom commented May 1, 2026

Summary

Implements option 4 from #81 — the "at minimum" mitigation. After a successful migration lndinit migrate-db writes a tombstone marker into every source bbolt file, after which lnd refuses to open them. When the lnd binary that runs against the destination backend was not compiled with the matching build tag (e.g. kvdb_sqlite), the operator is left with tombstoned bbolt files AND a destination backend lnd cannot read — only a pre-migration backup recovers the node.

This PR adds:

  • A multi-line WARN banner printed at the very start of migrate-db that names the destination backend and asks the operator to confirm both (1) they have a backup of every bbolt source file and (2) their lnd binary supports the destination backend.
  • A per-DB WARN line printed immediately before each tombstone write so the warning shows up next to the affected prefix in the log.

Logging-only change. No new flags, no behavioral change for existing callers. Pre-flight build-tag detection (option 1) and a deferred --finalize flow (option 3) are real fixes too, but each deserves its own design discussion — happy to follow up on either if you want this PR to land first as the floor.

Sample output (running with --dest.backend sqlite):

[WRN] LNDINIT ################################################################
[WRN] LNDINIT #                                                              #
[WRN] LNDINIT #   WARNING: lndinit migrate-db will write IRREVERSIBLE        #
[WRN] LNDINIT #   tombstone markers into every source bbolt database file    #
[WRN] LNDINIT #   after a successful migration. Once tombstoned, lnd will    #
[WRN] LNDINIT #   refuse to open the bbolt files with the error:             #
[WRN] LNDINIT #                                                              #
[WRN] LNDINIT #     refusing to use db, it was marked with a tombstone       #
[WRN] LNDINIT #     after successful data migration                          #
[WRN] LNDINIT #                                                              #
[WRN] LNDINIT #   Before continuing, please confirm BOTH of the following:   #
[WRN] LNDINIT #                                                              #
[WRN] LNDINIT #     1. You have a backup of every bbolt source DB file       #
[WRN] LNDINIT #        (channel.db, wallet.db, macaroons.db, etc.).          #
[WRN] LNDINIT #     2. The lnd binary that will run against the destination  #
[WRN] LNDINIT #        backend was compiled with the sqlite build tag.       #
[WRN] LNDINIT #                                                              #
[WRN] LNDINIT #   Without (2), lnd will fail on startup with `unknown        #
[WRN] LNDINIT #   database type` and the only recovery path is restoring     #
[WRN] LNDINIT #   the bbolt files from the backup taken in (1).              #
[WRN] LNDINIT #                                                              #
[WRN] LNDINIT ################################################################

Testing

  • go test -tags="kvdb_etcd kvdb_postgres kvdb_sqlite" ./... clean (lndinit 14.9s, migratekvdb cached).
  • golangci-lint run --timeout 10m --build-tags="kvdb_etcd kvdb_postgres kvdb_sqlite" ./... clean (v1.64.8 from tools/go.mod).
  • go build clean.
  • Manual smoke: invoked migrate-db against an empty dir to confirm the banner prints with the destination backend name interpolated correctly.

Related

Refs #81

After a successful migration lndinit writes a tombstone marker into
every source bbolt file. From that point lnd refuses to open those
files, which is fine when the destination backend works but turns into
unrecoverable data loss when the lnd binary that will run against the
new backend was not compiled with the matching build tag (e.g.
kvdb_sqlite). The user is left with tombstoned bbolt files and a
destination backend that lnd cannot read.

This is the "at minimum" mitigation requested in the issue: print a
loud, multi-line WARN banner at the start of migrate-db that names the
destination backend and tells the operator to confirm both that they
have a backup of the bbolt files and that the target lnd binary
supports the destination backend. A second per-DB WARN line is also
printed immediately before each tombstone write so it shows up next
to the affected prefix in the log.

This is intentionally a logging-only change: no new flags, no
behavioral or interface differences for existing callers. Pre-flight
build-tag detection (option 1) and a deferred --finalize step
(option 3) are real fixes worth their own design discussion and are
not part of this PR.

Refs lightninglabs#81
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant