feat(core): multi-pubkey DB support #2093

onur-ozkan · 2024-04-01T09:30:57Z

This pull request enables support for managing multiple public keys in mm2 databases. Initially, a single database instance is generated using the startup public key rmd160. With this update, the mm_ctx will manage multiple database instances by leveraging the public key from MmCoin::account_db_id if defined, or defaulting to the mm2 startup public key.

Completed

Introduced sql_connection_pool for managing all instances of shared and sqlite connections from one place
Introduced async_sql_connection_pool for managing all instances of async sqlite connections from one place
Refactored indexeddb driver implementation to allow multi pubkey support
Refactored SwapContext, MySwapStorage, MakerSwapStorage, TakerSwapStorage NftCtx, UtxoBlockHeaderStorage, Zcoin storages(web only), lp_ordermatch and other storage impl to support multipubkey
Implement database migrations upon new database instance creation or account activation using a custom public key.

Complex Todos:

kick_start unfinished swap upon new database instance creation or account activation using a custom public key.

Todos

Further refactor some RPC methods to enable results for a single public key or all active public keys.

Signed-off-by: onur-ozkan <work@onurozkan.dev>

onur-ozkan

Thank you for the work @borngraced. We need to add test coverage on this massive change of core mm2 logic.

This PR needs to be tested very carefully.

cc @KomodoPlatform/qa

Some initial notes from my end:

onur-ozkan · 2024-05-14T04:02:49Z

mm2src/coins/hd_wallet/storage/mod.rs

@@ -219,25 +220,27 @@ impl Default for HDWalletCoinStorage {
 }

 impl HDWalletCoinStorage {
+    // TODO: Since hd_wallet_rmd160 is unique for a device, do we use it as db_id too? or we can just use mm2 shared_db_id and use hd_wallet_rmd160 for primary key as it's currently done


Can we combine hd wallet with external wallets? I don't think it is possible, is it? @shamardy

I've previously discussed this with @borngraced via DM. The db path which stores accounts and addresses previously utilized by the user (which is hd_wallet_rmd160) should be global for a seed. However, each address should use its own db path for orders, swaps, etc.

This approach ensures more flexibility. For instance, if the same seed is used in both Keplr and mm2, and a swap is initiated using Keplr, then the app was closed midway through the swap. Upon restarting, if the Tendermint coin is activated using mm2 seed and the address enabled for swaps is the same address that was used before in Keplr, the swap will complete successfully.

For instance, if the same seed is used in both Keplr and mm2, and a swap is initiated using Keplr, then the app was closed midway through the swap. Upon restarting, if the Tendermint coin is activated using mm2 seed and the address enabled for swaps is the same address that was used before in Keplr, the swap will complete successfully.

I guess this is one of the test cases that should be tested by QA/GUI team once this PR is ready for testing
c.c. @smk762 @naezith

mm2src/coins/hd_wallet/storage/mod.rs

mm2src/coins/lightning/ln_utils.rs

mm2src/coins/lp_coins.rs

mm2src/mm2_db/src/indexed_db/db_lock.rs

onur-ozkan · 2024-05-14T04:27:07Z

mm2src/mm2_gui_storage/src/rpc_commands.rs

 pub async fn enable_account(ctx: MmArc, req: EnableAccountRequest) -> MmResult<SuccessResponse, AccountRpcError> {
-    let account_ctx = AccountContext::from_ctx(&ctx).map_to_mm(AccountRpcError::Internal)?;
+    // TODO db_id
+    let account_ctx = AccountContext::from_ctx(&ctx, None).map_to_mm(AccountRpcError::Internal)?;


I guess there is no way for this function to figure out db_id, if I am not mistaken? If I am right, we should probably extend the function args with db_id arg.

Yeah there're some other rpc methods like this too.. cc @shamardy

This is part of the GUI storage methods which was done by Sergey a long time ago and we later decided to not utilize it for HD wallet. GUI storage methods can be safely removed from the codebase, if we ever need it in the future, we can get it from commits history but my implementation approach for HD wallets will not need it as far as I can see.

I guess just comment them out?

onur-ozkan · 2024-05-14T04:43:21Z

Complex Todos:

Implement database migrations upon new database instance creation or account activation using a custom public key.

kick_start unfinished swap upon new database instance creation or account activation using a custom public key.

Currently, without making a big refactoring, this can be achieved from the coin activation functions I believe.

borngraced · 2024-05-16T23:40:47Z

Complex Todos:

Implement database migrations upon new database instance creation or account activation using a custom public key.

kick_start unfinished swap upon new database instance creation or account activation using a custom public key.

Currently, without making a big refactoring, this can be achieved from the coin activation functions I believe.

yep I accomplished task 1 by implementing a db migration watcher which keeps track of new db_id and run migrations/fix shared_db_dir if needed b370bec

onur-ozkan

yep I accomplished task 1 by implementing a db migration watcher which keeps track of new db_id and run migrations/fix shared_db_dir if needed b370bec

Q: What we call first when we are about to create new db? Can we do this migrations in the on-demand mode without running async runtime?

mm2src/coins/eth.rs

mm2src/coins/lp_coins.rs

mm2src/coins/nft/nft_structs.rs

onur-ozkan

Can you add a TODO note somewhere on db id handling implementation which includes the address of #2122 issue ?

mm2src/coins_activation/src/tendermint_with_assets_activation.rs

mm2src/mm2_core/src/sql_connection_pool.rs

…iew notes

onur-ozkan

I think tendermint_tx_history_v2 also needs some magic in order to keep the history data separated between inner-external accounts 🫤 I hope I am wrong.. A confirmation from GUI team or QA team could be nice.

onur-ozkan · 2024-05-22T05:51:17Z

mm2src/coins/hd_wallet/storage/wasm_storage.rs

-        Ok(HDWalletIndexedDbStorage { db })
+        Ok(HDWalletIndexedDbStorage {
+            db,
+            db_id: db_id.map(|e| e.to_string()),


e may lead to confusions as it refers to error in most cases.

Suggested change

db_id: db_id.map(|e| e.to_string()),

db_id: db_id.map(|str| str.to_string()),

OR:

Suggested change

db_id: db_id.map(|e| e.to_string()),

db_id: db_id.map(String::from),

onur-ozkan · 2024-05-22T05:53:54Z

mm2src/coins/hd_wallet/storage/wasm_storage.rs

 /// This function is used in `hd_wallet_storage::tests`.
 pub(super) async fn get_all_storage_items(ctx: &MmArc) -> Vec<HDAccountStorageItem> {
    let coins_ctx = CoinsContext::from_ctx(ctx).unwrap();
-    let db = coins_ctx.hd_wallet_db.get_or_initialize().await.unwrap();
+    let db = coins_ctx.hd_wallet_db.get_or_initialize(None).await.unwrap();


Can we add conditional compilation attributes on fn get_all_storage_items to make it accessible only from tests?

mm2src/coins/lp_coins.rs

onur-ozkan · 2024-05-22T06:13:26Z

mm2src/coins/lp_coins.rs

+            if active_only && coin.is_available() {
+                account_ids.insert(account.clone());
+                continue;
+            };
+
+            if !active_only {
+                account_ids.insert(account);
+                continue;
+            }


nit: active_only && coin.is_available() || !active_only should handle both in single block.

onur-ozkan added 3 commits March 28, 2024 16:27

WIP: resolve db paths by account keys

aa0909f

Signed-off-by: onur-ozkan <work@onurozkan.dev>

WIP: handle db ids optionally

f051600

Signed-off-by: onur-ozkan <work@onurozkan.dev>

use dummy values to fix build errors for development

c8a434e

Signed-off-by: onur-ozkan <work@onurozkan.dev>

onur-ozkan added enhancement New feature or request in progress Changes will be made from the author labels Apr 1, 2024

borngraced added 2 commits April 9, 2024 14:49

fix conflicts after merge

3e0061f

more wip changes and start coin pubkey for indexeddb storage

c6146d0

borngraced self-assigned this Apr 9, 2024

borngraced added 5 commits April 9, 2024 23:50

Z coin or default pubkey for initializing blocksdb

eff374c

zcoin - allow db_id for walletdb

fef8a4e

wip — SwapsContext

6247766

changes to maker and taker swap db_id

fa0f5ab

complete zcoin dbs multi pubkey support for WASM target

947ab8d

borngraced linked an issue Apr 10, 2024 that may be closed by this pull request

Implementing Per-Coin Database Paths Based on PubkeyHash for Enhanced Modularization #2094

Open

borngraced added 3 commits April 10, 2024 17:13

fix clippy/fmt

ae95d4e

impl account_db_id for wasm target

c0d13e7

complete some todos in swap mod

71c4e9c

borngraced marked this pull request as ready for review April 10, 2024 20:01

borngraced added 12 commits April 10, 2024 21:03

minor changes to DbIdentifier constructor(new)

be3dc81

changes to swaps db

ffbc262

handle recv_swap_msg with db_id

07c1168

add todos for StateMachineStorage for TakerSwapStorage

1ec96bc

add todos for StateMachineStorage for TakerSwapStorage

3ccfa00

get_swap_repr

29763e2

mark account_db_id as todo for native target

cf749eb

impl find_unique_active_account_ids and use for swap_kick_starts

368441c

make swap_v2 multi_key

7095605

WIP - start nft multi key support

8823be2

WIP - my_swaps_storage

42a22b1

fix field assignment outside of initializer

23f0566

implement account db_id for tendermint coin

b036c62

onur-ozkan commented May 14, 2024

View reviewed changes

borngraced added 8 commits May 14, 2024 06:56

implement account db_id for tendermint coin

9dcb662

use coin db_id in init_db and a minor change to account_db_ids loop

0ac682d

cleanup test mod in hd_wallet/storage/mod

c409ecc

drop connections as early as possible

23bb803

refactor db_lock and substitute Mutex for RwLock

8281609

revert db_lock connection list lock handling to mutex

115a873

improve my_orders_storage

9062bee

implement DbMigrationWatcher

b370bec

handle shared_db_id and minor fixes

46f8712

onur-ozkan commented May 17, 2024

View reviewed changes

mm2src/coins/eth.rs Outdated Show resolved Hide resolved

mm2src/coins/lp_coins.rs Outdated Show resolved Hide resolved

mm2src/coins/nft/nft_structs.rs Outdated Show resolved Hide resolved

borngraced added 2 commits May 17, 2024 10:21

polish sql_connection_pool and fix review notes

5bde012

cargo fmt

e036383

onur-ozkan commented May 17, 2024

View reviewed changes

borngraced added 7 commits May 20, 2024 03:31

abstract db migration from tendermint activation fn and fix other rev…

fab32e2

…iew notes

minor changes to DbMigrationWatcher

789ce8d

update coins_needed_for_kick_start for new db activation

ac5cc96

minor fix

d8caebd

comment out gui_storage_dispatcher and related

76efdf3

merge with dev and fix conflicts

24774a4

comment out gui_storage_accounts_functionality unit tests

6a0d0b6

onur-ozkan commented May 22, 2024

View reviewed changes

borngraced added 5 commits May 22, 2024 08:05

fix review notes

fbfe747

merge with dev and fix conflicts

097a63e

fix error handling

8a6d0cd

fix hd_wallet todos

f685f72

remove unneeded comments

df3cf1e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): multi-pubkey DB support #2093

feat(core): multi-pubkey DB support #2093

onur-ozkan commented Apr 1, 2024 •

edited by borngraced

onur-ozkan left a comment

onur-ozkan May 14, 2024

shamardy May 20, 2024 •

edited

shamardy May 20, 2024 •

edited

onur-ozkan May 14, 2024

borngraced May 14, 2024

shamardy May 20, 2024 •

edited

onur-ozkan May 21, 2024

onur-ozkan commented May 14, 2024 •

edited by borngraced

Complex Todos:

borngraced commented May 16, 2024 •

edited

Complex Todos:

onur-ozkan left a comment

onur-ozkan left a comment

onur-ozkan left a comment

onur-ozkan May 22, 2024

borngraced May 22, 2024

onur-ozkan May 22, 2024

borngraced May 22, 2024

onur-ozkan May 22, 2024

	db_id: db_id.map(\|e\| e.to_string()),
	db_id: db_id.map(\|str\| str.to_string()),

	db_id: db_id.map(\|e\| e.to_string()),
	db_id: db_id.map(String::from),

feat(core): multi-pubkey DB support #2093

Are you sure you want to change the base?

feat(core): multi-pubkey DB support #2093

Conversation

onur-ozkan commented Apr 1, 2024 • edited by borngraced

Completed

Complex Todos:

Todos

onur-ozkan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shamardy May 20, 2024 • edited

Choose a reason for hiding this comment

shamardy May 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shamardy May 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

onur-ozkan commented May 14, 2024 • edited by borngraced

Complex Todos:

borngraced commented May 16, 2024 • edited

Complex Todos:

onur-ozkan left a comment

Choose a reason for hiding this comment

onur-ozkan left a comment

Choose a reason for hiding this comment

onur-ozkan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

onur-ozkan commented Apr 1, 2024 •

edited by borngraced

shamardy May 20, 2024 •

edited

shamardy May 20, 2024 •

edited

shamardy May 20, 2024 •

edited

onur-ozkan commented May 14, 2024 •

edited by borngraced

borngraced commented May 16, 2024 •

edited