Proposed Timeline for Legacy Wallet and BDB removal

[achow101]

What to review next: #31378

---

Here is a proposed timeline for the move from legacy wallets to descriptor wallets, and at the same time, bdb to sqlite, culminating with the removal of both legacy wallets and bdb. Even though the wallet type and the wallet storage format are orthogonal to each other, as noted in #19077, I would like to tie them together for the sake of simplicity when it comes to how we make new wallets and the migration of one type to another.

The 4 combinations of type and format are: legacy-bdb, legacy-sqlite, descriptor-bdb, and descriptor-sqlite. While all 4 types will be supported, we will primarily deal with legacy-bdb and descriptor-sqlite. Setting descriptors=True in createwallet (or checking the Descriptors option in the GUI) will always make a descriptor-sqlite. Setting that to False will always make a legacy-bdb.

• 0.21 (late 2020)
  • Initial release of descriptor type and sqlite format. (Native Descriptor Wallets using DescriptorScriptPubKeyMan #16528, wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets #19077)
  • Descriptor type is experimental with a warning as such (Native Descriptor Wallets using DescriptorScriptPubKeyMan #16528)
  • Only legacy-bdb and descriptor-sqlite will be created (wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets #19077)
• 22.0 (mid 2021)
  • Implement descriptor exports (wallet, rpc: add listdescriptors command #20226)
  • Legacy type remains default (no pr needed)
  • Introduce dump and createfromdump wallet tool commands with an additional -format option to specify the wallet type (wallettool: Add dump and createfromdump commands #19137)
    • Users may use dump and then createfromdump in order to make legacy-sqlite from legacy-bdb
    • Users may use dump and then createfromdump in order to make descriptor-bdb from descriptor-sqlite
  • If a wallet specified in conf or command line doesn't exist, don't make a wallet and instead warn (Remove the automatic creation and loading of the default wallet #15454)
• 23.0 (early 2022)
  • Change descriptor type to non-experimental (Make descriptor wallets by default #23002)
  • Change createwallet descriptors default to True (Make descriptor wallets by default #23002)
  • Change wallet tool create -descriptors default to True (Make descriptor wallets by default #23002)
• 24.0 (late 2022)
  • Introduce migratewallet command to migrate a legacy-bdb to descriptor-sqlite (wallet: Migrate legacy wallets to descriptor wallets #19602)
  • For newly created legacy-bdb wallets, warn that they are deprecated and will soon be unsupported (wallet: Add a deprecation warning for newly created legacy wallets #24505)
• 26.0 (late 2023)
  • When loading legacy-bdb, descriptor-bdb, or legacy-sqlite wallets, warn that the legacy type and bdb format will soon be unsupported, as applicable. (wallet: Give deprecation warning when loading a legacy wallet #27869)
  • Deprecate creating new legacy-bdb wallets wallet: No BDB creation, unless -deprecatedrpc=create_bdb #28597
  • migratewallet bug fixes
    • wallet: Reload watchonly and solvables wallets after migration #28609
    • wallet: prevent bugs from invalid transaction heights with asserts, comments, and refactoring #28546
• 27.0 (early 2024)
  • address book performance improvement wallet: batch and simplify addressbook migration process #26836
  • bulk deletion of txs performance improvement wallet: simplify and batch zap wallet txes process #28987
  • bulk deletion of records performance improvement wallet: batch erase procedures and improve 'EraseRecords' performance #29403
  • Caching to improve performance of migrated non-HD descriptor wallets (also perf improvement in general) wallet: cache IsMine scriptPubKeys to improve performance of descriptor wallets #26008
  • Remove migratewallet experimental warning rpc: Drop migratewallet experimental warning #28037
• 28.0 (late 2024)
  • Improve performance of migratewallet wallet: optimize migration process, batch db transactions #28574
  • Reimplement legacy-bdb to descriptor-sqlite migration (maybe as a separate command or in the wallet tool) to be independent of LegacyScriptPubKeyMan and BDB (wallet: Migrate legacy wallets to descriptor wallets without requiring BDB #26596)
  • For dump, implement an independent BDB file loader (this should probably be shared with the previous) (wallet: Implement independent BDB parser #26606)
  • GUI: Migrate wallets that are not loaded (Migrate legacy wallets that are not loaded bitcoin-core/gui#824)
  • Fix default wallet migration naming (wallet: default wallet migration, modify inconvenient backup filename #29586)
  • Fix listwalletdir of migrated default wallets and generated backups (wallet: Fix listwalletdir listing of migrated default wallets and generated backup files #30265)
• 29.0 (early 2025)
  • Watch-only wallet migration edge case (wallet: fix crash during watch-only wallet migration #31374)
  • Avoid weird script corner cases (wallet: Utilize IsMine() and CanProvide() in migration to cover edge cases #31495)
  • Rework wallet_migration.py (test: Rework wallet_migration.py to use previous releases #31248)
  • Add combinerawtransaction test that doesn't need legacy wallet (test: Add combinerawtransaction test to rpc_createmultisig #31249)
  • Remove legacy wallet from migration benchmark (wallet: remove BDB dependency from wallet migration benchmark #31241)
• 30.0 (late 2025)
  • Remove optional sqlite when building with wallet (Require sqlite when building the wallet #31961)
  • Invalid multisigs migration edge case (wallet: fix crash during migration due to invalid multisig descriptors #31378)
  • Stop loading legacy-bdb, descriptor-bdb, legacy-sqlite wallet entirely. Inform users how to migrate (wallet: Disable creating and loading legacy wallets #31250)
  • Remove LegacyScriptPubKeyMan except for the independent migration stuff (Remove the legacy wallet and BDB dependency #28710)
  • Remove BDB except for independent loading for migration stuff (Remove the legacy wallet and BDB dependency #28710)

Additional improvements that have no target deadline:

• Backwards compatibility tests fixes for descriptor wallets test: Fixes and updates to wallet_backwards_compatibility.py for 25.0 and descriptor wallets #28027

The migration code will probably be around for a long time/forever.

Note that we expect users who go through the effort to make a legacy-sqlite or descriptor-bdb wallet to suck it up and deal with the consequences of doing something that isn't really supported. Such users should be able to figure out for themselves how to migrate their wallets. (Maybe migratewallet can migrate legacy-sqlite to descriptor-sqlite. bdb to sqlite is way easier than legacy to descriptor).

[jonasschnelli]

Look good except 0.27 (late 2023) (Stop loading legacy-bdb [...]).
AFAIK this would be the first time one couldn't load an historical wallet.dat file in Core (which is IMO a great feature).
Disabling the possibility of loading wallets that have been created just two years ago seems to be too strict.

Since the wallet files are precious data sources and could lead to stress when failing to load, I suggest to continue support for BDB wallets indefinitely. Support could also mean an automatic migration to sqlite with an internal dependency free converter.

[achow101]

I suggest to continue support for BDB wallets indefinitely. Support could also mean an automatic migration to sqlite with an internal dependency free converter.

This is almost what the last step is. There will be a dependency free migration tool that users can use to move legacy-bdb wallets to descriptor-sqlite. I would be hesitant to make this automatic though.

[ChristopherA]

I'm hoping not for indefinite support of BDB wallets. It is easy enough to have an external tool do the conversion post 0.27.

[maflcko]

0.23 Introduce migratewallet command to migrate a legacy-bdb to descriptor-sqlite

Any reason this can't be done in 22.0 already?

[achow101]

0.23 Introduce migratewallet command to migrate a legacy-bdb to descriptor-sqlite

Any reason this can't be done in 22.0 already?

No, I just don't think it will make it.

[S3RK]

23.0 (late 2021)

• Change createwallet descriptors default to True

I think it's good to add wallet tool here as well to not forget to change default wallet type there as well

[achow101]

Something that is not on this list, but is going to need to be done at some point prior to removing the legacy wallet is updating all of the unit tests and benchmarks to use descriptor wallets. Currently a bunch of them us a LegacyScriptPubKeyMan because it was convenient. Those will need to be changed.

[fanquake]

I'd suggest we move:

24.0 (late 2022)
    For newly created legacy-bdb wallets, warn that they are deprecated and will soon be unsupported (not yet started)

into the 0.23 milestone, unless there is a particular reason to delay warning? I don't think we need to wait an extra 6 months to start warning about this.