[WASM] Fixing, refactoring, and implementing P2P, IndexedDB, SwapLock, SavedSwap, MySwaps, MyOrdersStorage #1007

sergeyboyko0791 · 2021-07-07T13:23:17Z

Refactor IndexedDB API to make it easier to add new databases and tables
- Move the low-level Indexed DB interface to the separate db_driver module
- Remove channel logic from tx_history_db (tx_history_db has to communicate with Indexed DB directly)
- Change type of index_value from &str to Json
- Remove TxHistoryOps and native TxHistoryDb
- Return tx_history_path to MmCoin
- Add DbNamespaceId to MmCtx to allow running multiple MmCtx at the same time
- Add the stringified rmd160 of the wallet public key to the names of IndexedDb databases.
Add and implement SwapLock in WASM
- Add SwapsContext::swap_db
- Add SwapLockTable
- Add FileLockError, FileLockResult to common::file_lock
- Integrate SwapLock to maker/taker swaps
Add P2P in-memory transport
- Add NodeType::LightInMemory and NodeType::RelayInMemory variants
- Refactor parsing relay addresses
- Use in-memory transport in atomicdex_tests
Add the 'Build MM2 WASM Debug' step to the Linux CI script
Remove defined-in-js.js
Design and implement IdbCursor in Rust
Finish integrating swaps in WASM
Add and implement MyOrdersStorage

* Move the low-level Indexed DB interface to the separate 'db_driver' module * Remove channel logic from tx_history_db (tx_history_db communicates with Indexed DB directly)

* Change type of index_value from &str to Json * Move the logic of initializing database from 'CoinsContext::tx_history_db()' to 'indexed_db::db_lock::get_or_initialize_db' * Remove 'TxHistoryOps' and native 'TxHistoryDb' * Replace 'TxHistoryError', 'TxHistoryResult' to lp_coins * Return 'tx_history_path' to MmCoin * Add 'FileLockError', 'FileLockResult' to 'common::file_lock'

* Add DbNamespaceId

* Add 'SwapsContext::swap_db' * Add 'SwapLockTable'

* Remove 'SwapLock::last_timestamp'

* Add 'NodeType::LightInMemory' and 'NodeType::RelayInMemory' variants * Refactor parsing relay addresses

* Do not add relays to 'PeersExchange' in WASM

artemii235

Looks great! I have a couple of questions 🙂
I will take one more look at this when we discuss them.

mm2src/mm2_tests.rs

mm2src/mm2_libp2p/src/relay_address.rs

* Refactor 'MarketMakerIt' * Add 'common::log_tag' macro * Configure the 'trade_test_rick_and_morty' and 'test_qrc20_history' tests to run in WASM

# Conflicts: # mm2src/mm2_tests.rs

* Move 'SavedSwap' to the separate 'lp_swap/saved_swap' module * Add 'get_item_by_unique_index', 'replace_item_by_unique_index', 'delete_item_by_unique_index' methods to 'DbTable' * Refactor 'for_tests::register_wasm_log'

* Add 'common::wasm_http' mod with 'FetchRequest'

* Add driver/cursor.rs, Idb<Single/Multi><Bound>KeyCursor * Add indexed_cursor.rs, Db<Single/Multi><Bound>KeyCursor * Add TableUpgrader::create_multi_index * Add IdbTableImpl::cursor_builder * Add DbTable::open_cursor

* Add 'spawn_highload_future' to dispatcher_legacy.rs

* Move common::file_lock to common::fs::file_lock * Move slurp, safe_slurp, file_lock, remove_file, write, read_dir_async, read_dir, json_dir_entries to common::fs

* Use MyOrdersStorage everywhere instead of working with fs/Db manually * Rename swap_db::MySwapsTable to swap_db::MySwapsFiltersTable * Move fs functions from lp_swap::saved_swap to common::fs * Add error_if, warn_if logging macros * Remove OrdermatchContext::my_cancelled_orders

* Add 'OrdermatchContext::ordermatch_db' * Add 'MyActive[Maker/Taker]OrdersTable', 'MyHistoryOrdersTable', 'MyFilteringHistoryOrdersTable' * Kick-start active maker/taker orders on restart

* Enable 'order_status' RPC call in WASM

* Rename 'my_swaps_db::MySwaps' to 'my_swaps_storage::MySwapsStorage' * Rename swap_db.rs to swap_wasm_db.rs

sergeyboyko0791 · 2021-08-11T11:27:49Z

@artemii235 @shamardy PR is ready for the review 🥳

artemii235

Thanks for the great work!
I've reviewed this PR partially, might have more comments when we solve the questions below.

mm2src/coins/lp_coins.rs

mm2src/lp_ordermatch/my_orders_storage.rs

mm2src/common/log.rs

artemii235 · 2021-08-30T12:15:33Z

@sergeyboyko0791 Could you please also solve git conflicts?

* Remove excessive trait bounds of the `load_history_from_file_impl` function * Refactor error messages

sergeyboyko0791 · 2021-08-30T23:51:02Z

This PR consists of critical changes in many different modules such as Swaps, Orders, Tx History.
Basically, the changes are concerned the process of saving, loading swap/order/tx_history files data to a database.

The test cases:

Transaction history (check in Native and WASM):

The history saves and loads normally;
If the tx history file is broken (for example, invalid JSON character), mm2 has to clear it and reload the transaction history again.

Swaps (in Native only):

The swap data saves and loads normally ⚠️
My Maker/Taker swaps file path: DB/<ID>/SWAPS/MY/<UUID>.json
Not finished Maker/Taker swaps yet have to be kick-starterted on restart ⚠️
Swap data is broadcast after the finish of a Maker/Taker swap;
all_swaps_uuids_by_filters, import_swaps, my_recent_swaps RPC calls work correctly;
my_swap_status returns correct info of already finished swaps.
If two instances of MarketMaker are run with the same passphrase and with the same DB directory, not finished swaps must be kick-started on only one instance ⚠️

Orders (in Native only):

My active orders save and load normally;
My Maker/Taker orders file path: DB/<ID>/ORDERS/MY/<Maker/Taker>/<UUID>.json;
Maker/Taker active orders have to be kick-started on mm2 restart;
My Maker/Taker orders has to be saved to DB/<ID>ORDERS/MY/HISTORY/<UUID>.json if save_in_history is true when the orders were matched/canceled;
update_maker_order RPC call works correctly;
my_orders RPC call work correctly for Maker and Taker orders;
orders_history_by_filter RPC call work correctly with many different parameter combinations. Especially for Maker and Taker orders;
order_status returns correct info of already matched/canceled Maker and Taker orders.

* Add the 'LogOnError' trait

artemii235

LGTM! I have left a couple of non-blocker possible refactoring ideas. We can discuss them later. Thanks!

artemii235 · 2021-09-01T10:14:50Z

mm2src/common/indexed_db/indexed_cursor.rs

+//!
+//! # Important
+//!
+//! Please make sure all keys of the index are specified


It seems that created indexes should be represented in the type system somehow to avoid invalid cursor creation. E.g

enum SwapIndexes { BaseRelStartedAt, ... }

It might be a part of the TableSignature trait to pin indexes to a specific table: associated type or another way, just thinking out loud.

Thank you for the advice! It makes sense and it's nice to have :)
We can consider integrating it later as you said or if we move it to a public crate

artemii235 · 2021-09-01T10:22:51Z

mm2src/lp_ordermatch/ordermatch_wasm_db.rs

+    use serde_json::Value as Json;
+
+    #[derive(Debug, Deserialize, Eq, PartialEq, Serialize)]
+    pub struct MyActiveMakerOrdersTable {


I have an intuitive feeling that it might be worth splitting a Table and it's item into separate structs. They can be pinned to each other using generics or associated types of the TableSignature trait. It's just an idea, I'm unsure whether it's possible to do so without major refactoring. Maybe it doesn't make much sense in general 🙂

Actually, I didn't understand why it's needed :D
The only reason I see is the confusing names of the structures.

The only reason I see is the confusing names of the structures.

Actually yes, it's confusing because a specific struct represents table and table's item at the same time 🙂

shamardy

LGTM! Great Work 🔥

# Conflicts: # mm2src/lp_native_dex.rs # mm2src/lp_ordermatch.rs # mm2src/mm2_tests.rs

sergeyboyko0791 added 10 commits June 28, 2021 18:58

Refactor indexed_db

4f55b99

* Move the low-level Indexed DB interface to the separate 'db_driver' module * Remove channel logic from tx_history_db (tx_history_db communicates with Indexed DB directly)

Refactor IndexedDb to allow multiple MmCtx at the same time

a07ffb6

* Add DbNamespaceId

Add 'SwapLock' to 'lp_swap'

21ee710

* Add 'SwapsContext::swap_db' * Add 'SwapLockTable'

Integrate 'SwapLock' to maker/taker swaps

4c57f4a

* Remove 'SwapLock::last_timestamp'

Add missing swap_db.rs file

799b342

Merge branch 'dev' into wasm-swaps

01ea97f

Add P2P in-memory transport

17669d7

* Add 'NodeType::LightInMemory' and 'NodeType::RelayInMemory' variants * Refactor parsing relay addresses

Add warn logs to 'PeersExchange'

307cb42

Merge branch 'dev' into wasm-swaps

abefcf4

sergeyboyko0791 linked an issue Jul 7, 2021 that may be closed by this pull request

Web browser support (WASM). #928

Closed

sergeyboyko0791 marked this pull request as draft July 7, 2021 13:23

sergeyboyko0791 self-assigned this Jul 7, 2021

sergeyboyko0791 added 2 commits July 8, 2021 13:30

Set panic_hook when 'mm2_main' is called in WASM

4f3e27b

* Do not add relays to 'PeersExchange' in WASM

Minor changes

e5745f2

sergeyboyko0791 marked this pull request as ready for review July 8, 2021 07:18

artemii235 reviewed Jul 8, 2021

View reviewed changes

mm2src/mm2_tests.rs Outdated Show resolved Hide resolved

mm2src/mm2_libp2p/src/relay_address.rs Show resolved Hide resolved

sergeyboyko0791 added 13 commits July 13, 2021 19:13

Set up mm2_tests to run in a browser

58d17d3

* Refactor 'MarketMakerIt' * Add 'common::log_tag' macro * Configure the 'trade_test_rick_and_morty' and 'test_qrc20_history' tests to run in WASM

Merge branch 'dev' into wasm-swaps

04f0ab8

# Conflicts: # mm2src/mm2_tests.rs

Obtain passphrase from env instead of env_file

9e6edc9

Add BOB_PASSPHRASE, ALICE_PASSPHRASE envs to CI wasm tests

395e1bd

Fix test_check_balance_on_post_order

a289265

Implement saving and loading 'SavedSwap' for WASM

0b13f2e

* Move 'SavedSwap' to the separate 'lp_swap/saved_swap' module * Add 'get_item_by_unique_index', 'replace_item_by_unique_index', 'delete_item_by_unique_index' methods to 'DbTable' * Refactor 'for_tests::register_wasm_log'

Add the wallet 'rmd160' to DbIdentifier

d745f3d

Remove defined-in-js.js

147514d

* Add 'common::wasm_http' mod with 'FetchRequest'

Add the 'Build MM2 WASM Debug' step to the Linux CI script

85be85a

Fix artifacts folder path

3f35df2

Separate the 'db_driver' module into parts

a5a851e

Merge branch 'dev' into wasm-swaps

09552e7

sergeyboyko0791 added 7 commits August 3, 2021 18:30

Port 'import_swaps' and 'recover_funds_of_swap' to WASM

f7d1c05

* Add 'spawn_highload_future' to dispatcher_legacy.rs

Fix swap kick-start in WASM

0c919ff

Order my_recent_swaps by started_at

2373ffa

Add common::fs module

6b449e4

* Move common::file_lock to common::fs::file_lock * Move slurp, safe_slurp, file_lock, remove_file, write, read_dir_async, read_dir, json_dir_entries to common::fs

Fix WASM build

fc5fff5

Implement MyOrdersStorage in WASM except 'select_orders_by_filter'

50195cd

* Add 'OrdermatchContext::ordermatch_db' * Add 'MyActive[Maker/Taker]OrdersTable', 'MyHistoryOrdersTable', 'MyFilteringHistoryOrdersTable' * Kick-start active maker/taker orders on restart

sergeyboyko0791 changed the title ~~[WASM] Fixing, refactoring, and implementing P2P, IndexedDB, SwapLock~~ [WASM] Fixing, refactoring, and implementing P2P, IndexedDB, SwapLock, SavedSwap, MySwaps, MyOrdersStorage Aug 11, 2021

sergeyboyko0791 added 3 commits August 11, 2021 13:49

Revert bumped versions of Cargo.lock dependencies

4d46877

* Enable 'order_status' RPC call in WASM

Minor changes

20ca5dd

* Rename 'my_swaps_db::MySwaps' to 'my_swaps_storage::MySwapsStorage' * Rename swap_db.rs to swap_wasm_db.rs

Merge branch 'dev' into wasm-swaps

7f04928

sergeyboyko0791 requested a review from shamardy August 11, 2021 11:28

artemii235 reviewed Aug 30, 2021

View reviewed changes

mm2src/coins/lp_coins.rs Outdated Show resolved Hide resolved

mm2src/lp_ordermatch/my_orders_storage.rs Outdated Show resolved Hide resolved

mm2src/lp_ordermatch/my_orders_storage.rs Outdated Show resolved Hide resolved

mm2src/common/log.rs Outdated Show resolved Hide resolved

sergeyboyko0791 added 2 commits August 30, 2021 20:31

Merge branch 'dev' into wasm-swaps

ebbeaef

Small fixes

a7e435c

* Remove excessive trait bounds of the `load_history_from_file_impl` function * Refactor error messages

Replace error_if, warn_if macros with a trait

b8f64bd

* Add the 'LogOnError' trait

artemii235 previously approved these changes Sep 1, 2021

View reviewed changes

shamardy previously approved these changes Sep 1, 2021

View reviewed changes

artemii235 added 2 commits September 13, 2021 18:32

Merge remote-tracking branch 'origin/dev' into wasm-swaps

4a9c6dc

# Conflicts: # mm2src/lp_native_dex.rs # mm2src/lp_ordermatch.rs # mm2src/mm2_tests.rs

Fixes after merge.

7c560ed

artemii235 dismissed stale reviews from shamardy and themself via 7c560ed September 13, 2021 11:48

Fix Build MM2 WASM Debug by partial cargo clean.

159eb4c

artemii235 approved these changes Sep 13, 2021

View reviewed changes

artemii235 merged commit 42d76e6 into dev Sep 13, 2021

artemii235 deleted the wasm-swaps branch September 13, 2021 12:17

artemii235 mentioned this pull request Sep 15, 2021

Web browser support (WASM). #928

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WASM] Fixing, refactoring, and implementing P2P, IndexedDB, SwapLock, SavedSwap, MySwaps, MyOrdersStorage #1007

[WASM] Fixing, refactoring, and implementing P2P, IndexedDB, SwapLock, SavedSwap, MySwaps, MyOrdersStorage #1007

sergeyboyko0791 commented Jul 7, 2021 •

edited

artemii235 left a comment

sergeyboyko0791 commented Aug 11, 2021

artemii235 left a comment

artemii235 commented Aug 30, 2021

sergeyboyko0791 commented Aug 30, 2021 •

edited

artemii235 left a comment

artemii235 Sep 1, 2021

sergeyboyko0791 Sep 1, 2021

artemii235 Sep 1, 2021

sergeyboyko0791 Sep 1, 2021

artemii235 Sep 2, 2021

shamardy left a comment

[WASM] Fixing, refactoring, and implementing P2P, IndexedDB, SwapLock, SavedSwap, MySwaps, MyOrdersStorage #1007

[WASM] Fixing, refactoring, and implementing P2P, IndexedDB, SwapLock, SavedSwap, MySwaps, MyOrdersStorage #1007

Conversation

sergeyboyko0791 commented Jul 7, 2021 • edited

artemii235 left a comment

Choose a reason for hiding this comment

sergeyboyko0791 commented Aug 11, 2021

artemii235 left a comment

Choose a reason for hiding this comment

artemii235 commented Aug 30, 2021

sergeyboyko0791 commented Aug 30, 2021 • edited

artemii235 left a comment

Choose a reason for hiding this comment

artemii235 Sep 1, 2021

Choose a reason for hiding this comment

sergeyboyko0791 Sep 1, 2021

Choose a reason for hiding this comment

artemii235 Sep 1, 2021

Choose a reason for hiding this comment

sergeyboyko0791 Sep 1, 2021

Choose a reason for hiding this comment

artemii235 Sep 2, 2021

Choose a reason for hiding this comment

shamardy left a comment

Choose a reason for hiding this comment

sergeyboyko0791 commented Jul 7, 2021 •

edited

sergeyboyko0791 commented Aug 30, 2021 •

edited