Revisit ldk-node's persistence architecture

[joostjager]

During the July 23 developer call, we discussed that persistence in and around ldk-node has become difficult to reason about. State is managed across LDK, ldk-node, and BDK, while storage implementations and persistence abstractions are split across multiple layers.

Even when these components use the same underlying store, related updates are generally written independently rather than committed atomically. A crash or persistence error can therefore leave protocol state, payment records, wallet state, and user-facing events out of sync. Replay and startup reconciliation repair some cases, but can also produce missing records, stale state, or duplicate events.

Addressing these problems individually is adding more special-case recovery logic without giving us a clear overall persistence contract.

We would like to collect ideas for simplifying this architecture and making its consistency and recovery guarantees easier to understand. Please share possible approaches, tradeoffs, or relevant experience with local, remote, and custom storage backends.

#51 #171 #298 #381 #425 #472 #932 #966 #967 #968 #969 #970

[joostjager]

I think we need atomic transactions, as proposed in #970. Ideally LDK would let us commit its writes together with the related ldk-node state.

Otherwise, ldk-node could buffer the writes made through the store passed to LDK, then commit them atomically with its own writes before completing LDK's pending futures. This would only provide stronger guarantees for backends that support atomic transactions.