fix(nft): add log_index to history table and use in PK #1926

shamardy · 2023-08-01T00:59:06Z

fixes https://github.com/KomodoPlatform/komodo-wallet/issues/1239#issuecomment-1658128185
related comment https://github.com/KomodoPlatform/komodo-wallet/issues/1239#issuecomment-1658670188

This PR fixes transactions that transfer multiple NFT tokens in db. These transactions cause errors when adding due to the db constraint on tx hash uniqueness. To solve this, the PR uses log_index as part of the transfers history table primary key.
nft_tx_history table is now called nft_transfer_history and tx/txs are renamed to transfer/transfers throughout the NFT code since what's added/retrieved from DB is NFT transfers not transactions (Multiple NFT transfers can be in one transaction). By renaming the table, there are no need for db migrations due to the addition of log_index column. Although NFT is not used in production yet, if anybody used it, transfers will be re-fetched and saved to the new DB table when using the related API methods.

…r_history_table

shamardy · 2023-08-01T01:23:04Z

@anarkiVan can you please check if this PR fixes this error https://github.com/KomodoPlatform/komodo-wallet/issues/1239#issuecomment-1658128185 or not?

anarkiVan · 2023-08-01T04:55:14Z

@anarkiVan can you please check if this PR fixes this error KomodoPlatform/komodo-wallet#1239 (comment) or not?

@shamardy the error is gone. I was able to see nfts and transactions using seed from the issue

onur-ozkan

LGTM.

onur-ozkan · 2023-08-01T05:27:49Z

mm2src/coins/nft/storage/sql_storage.rs

@@ -63,7 +64,8 @@ fn create_tx_history_table_sql(chain: &Chain) -> MmResult<String, SqlError> {
    collection_name TEXT,
    image_url TEXT,
    token_name TEXT,
-    details_json TEXT
+    details_json TEXT,
+    PRIMARY KEY (transaction_hash, log_index, chain)


note: I don't know about the frequency of write operations this table will get, but this PK design will definetly slow down these operations a lot specially when there are too many rows.

I can remove this primary key all together, we already avoid writing duplicate transfers that have the same chain/transaction_hash/log_index by using the last block number when requesting new transfers that will be saved. It will be difficult though to detect problems/bugs like the one that this PR fixes if I did that.

we already avoid writing duplicate transfers that have the same chain/transaction_hash/log_index by using the last block number when requesting new transfers that will be saved.

If it's done with the thread-safe way, seems fine yeah.

It will be difficult though to detect problems/bugs like the one that this PR fixes if I did that.

The error seems quite clear if I am not missing something.

I don't have hard feelings to remove this PK, just wanted to leave a note on it. I leave it up to you :)

https://github.com/KomodoPlatform/komodo-wallet/issues/1239#issuecomment-1658128185 seems quite clear if I am not missing something.

It was a db error due to the unique constraint on transaction hash, if moralis duplicates data for some reason or there are problems with our implementation that we are trying to save duplicate data it won't be caught without some kind of constraint. Adding a constraint on these values instead of a primary key will also slow down operations afaik.

I don't have hard feelings to remove this PK, just wanted to leave a note on it. I leave it up to you :)

Let's leave it as is then and see if there are performance problems in the future :)

Adding a constraint on these values instead of a primary key will also slow down operations afaik.

The idea is not merging multiple values as PK/constrait because on each write each row in the table will need to check all these fields. The best way to deal with this would be creating another field(maybe id?) and inserting the values with pre-calculated id(which can be combination of the values used as PK atm) from mm2.

Let's leave it as is then and see if there are performance problems in the future :)

Totally fine 👍

Is the column necessary?

It's not, but I guess it was added to make this

komodo-defi-framework/mm2src/coins/nft/storage/sql_storage.rs

Lines 201 to 209 in be413ca

fn nft_from_row(row: &Row<'_>) -> Result<Nft, SqlError> {

let json_string: String = row.get(0)?;

json::from_str(&json_string).map_err(|e| SqlError::FromSqlConversionFailure(0, Type::Text, Box::new(e)))

}

fn transfer_history_from_row(row: &Row<'_>) -> Result<NftTransferHistory, SqlError> {

let json_string: String = row.get(0)?;

json::from_str(&json_string).map_err(|e| SqlError::FromSqlConversionFailure(0, Type::Text, Box::new(e)))

}

deserialize to the structs straight away. I can change the function to transfer_history_from_row_and_chain and insert the chain value into the json object before returning it or just leave it as is. What do you think is better?

I think rather than keeping a field all with the same values, inserting the value while deserializing better.

Inserting the value after deserialization and removing it before serialization would complicate the code a lot. Additionally, adding the chain field after fetching the row/s requires parsing the JSON string into a serde_json::Value, modifying it, and then deserializing it into an Nft/NftTransferHistory structs. This process likely has a greater impact on performance than simply fetching the chain value directly from the database and deserializing the entire row in one operation. Ideally, the chain field should not have been included in the Nft/NftTransferHistory structs from the begining, but it cannot be removed now as it is part of the responses used by GUIs.

I can of course change these structs

komodo-defi-framework/mm2src/coins/nft/nft_structs.rs

Lines 218 to 227 in 717e376

#[derive(Clone, Debug, Deserialize, Serialize)]

pub struct Nft {

#[serde(flatten)]

pub(crate) common: NftCommon,

pub(crate) chain: Chain,

pub(crate) block_number_minted: Option<u64>,

pub(crate) block_number: u64,

pub(crate) contract_type: ContractType,

pub(crate) uri_meta: UriMeta,

}

komodo-defi-framework/mm2src/coins/nft/nft_structs.rs

Lines 392 to 405 in 717e376

#[derive(Clone, Debug, Deserialize, Serialize)]

pub struct NftTransferHistory {

#[serde(flatten)]

pub(crate) common: NftTransferCommon,

pub(crate) chain: Chain,

pub(crate) block_number: u64,

pub(crate) block_timestamp: u64,

pub(crate) contract_type: ContractType,

pub(crate) token_uri: Option<String>,

pub(crate) collection_name: Option<String>,

pub(crate) image_url: Option<String>,

pub(crate) token_name: Option<String>,

pub(crate) status: TransferStatus,

}

to make all fields other than chain part of a new struct that is used for DB operations and use #[serde(flatten)] for this new struct, but the Nft/NftTransferHistory structs will get complicated too. I think I would rather keep the chain column :)

Okay I thought it would be simple. In that case, it can stay as it is.

onur-ozkan · 2023-08-01T05:31:28Z

For the future PRs, we can split the changes into multiple commits(for this PR it would be 1 for the renaming, 1 for the actual fix), it would be very useful for the reviewers. It was quite confusing for me :)

mm2src/coins/nft/storage/wasm/wasm_storage.rs

borngraced

Great work. only one comment

shamardy · 2023-08-01T13:32:02Z

For the future PRs, we can split the changes into multiple commits(for this PR it would be 1 for the renaming, 1 for the actual fix), it would be very useful for the reviewers. It was quite confusing for me :)

Yeah, sorry about that. I thought name changes are gonna be minimal and limited to db methods when I started renaming but it went out of control and I couldn't separate it into a different commit. I will keep that in mind in the future and try to separate commits even if I thought they will be small at the start.

borngraced

LGTM 🚀

caglaryucekaya

LGTM, I only have some questions

mm2src/coins/nft/storage/db_test_helpers.rs

caglaryucekaya · 2023-08-01T20:26:15Z

mm2src/coins/nft/storage/sql_storage.rs

@@ -63,7 +64,8 @@ fn create_tx_history_table_sql(chain: &Chain) -> MmResult<String, SqlError> {
    collection_name TEXT,
    image_url TEXT,
    token_name TEXT,
-    details_json TEXT
+    details_json TEXT,
+    PRIMARY KEY (transaction_hash, log_index, chain)


Aren't there a different transfer history table for each chain, why do we need the chain as part of the primary key?

caglaryucekaya · 2023-08-01T20:36:55Z

komodo-defi-framework/mm2src/coins/nft/storage/db_test_helpers.rs

Lines 382 to 410 in 717e376

    
           pub(crate) async fn test_add_get_nfts_impl() { 
        
               let chain = Chain::Bsc; 
        
               let storage = init_nft_list_storage(&chain).await; 
        
               let nft_list = nft_list(); 
        
               storage.add_nfts_to_list(&chain, nft_list, 28056726).await.unwrap(); 
        
               let token_id = BigDecimal::from_str(TOKEN_ID).unwrap(); 
        
               let nft = storage 
        
                   .get_nft(&chain, TOKEN_ADD.to_string(), token_id) 
        
                   .await 
        
                   .unwrap() 
        
                   .unwrap(); 
        
               assert_eq!(nft.block_number, 28056721); 
        
           } 
        
           pub(crate) async fn test_last_nft_blocks_impl() { 
        
               let chain = Chain::Bsc; 
        
               let storage = init_nft_list_storage(&chain).await; 
        
               let nft_list = nft_list(); 
        
               storage.add_nfts_to_list(&chain, nft_list, 28056726).await.unwrap(); 
        
               let token_id = BigDecimal::from_str(TOKEN_ID).unwrap(); 
        
               let nft = storage 
        
                   .get_nft(&chain, TOKEN_ADD.to_string(), token_id) 
        
                   .await 
        
                   .unwrap() 
        
                   .unwrap(); 
        
               assert_eq!(nft.block_number, 28056721); 
        
           }

@laruh It's not related to this PR, but when looking at it I saw that these two tests are doing the same thing. Did you mean to do something else with the test_last_nft_blocks maybe?

use transaction_hash, log_index, chain as the primary key for transfe…

c121c41

…r_history_table

shamardy added P0 1.0.7-beta in progress Changes will be made from the author under review and removed in progress Changes will be made from the author labels Aug 1, 2023

shamardy requested review from caglaryucekaya, anarkiVan, onur-ozkan and borngraced August 1, 2023 01:21

onur-ozkan previously approved these changes Aug 1, 2023

View reviewed changes

borngraced reviewed Aug 1, 2023

View reviewed changes

mm2src/coins/nft/storage/wasm/wasm_storage.rs Outdated Show resolved Hide resolved

borngraced reviewed Aug 1, 2023

View reviewed changes

make log_index u32

717e376

shamardy dismissed onur-ozkan’s stale review via 717e376 August 1, 2023 16:35

borngraced previously approved these changes Aug 1, 2023

View reviewed changes

caglaryucekaya reviewed Aug 1, 2023

View reviewed changes

chain is not part of primary key of transfer history table

be413ca

shamardy dismissed borngraced’s stale review via be413ca August 2, 2023 01:04

caglaryucekaya approved these changes Aug 4, 2023

View reviewed changes

shamardy merged commit 92372cb into dev Aug 7, 2023
20 of 30 checks passed

shamardy deleted the fix-nft-db-multi-token-tx branch August 7, 2023 03:32

laruh mentioned this pull request Aug 11, 2023

fix(hd-wallet): enable/withdraw using any account'/change/address_index #1933

Merged

laruh mentioned this pull request Aug 30, 2023

chore(release): v1.0.7-beta #1936

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(nft): add log_index to history table and use in PK #1926

fix(nft): add log_index to history table and use in PK #1926

shamardy commented Aug 1, 2023 •

edited

shamardy commented Aug 1, 2023

anarkiVan commented Aug 1, 2023

onur-ozkan left a comment

onur-ozkan Aug 1, 2023

shamardy Aug 1, 2023

onur-ozkan Aug 1, 2023

shamardy Aug 1, 2023

onur-ozkan Aug 1, 2023

shamardy Aug 2, 2023

caglaryucekaya Aug 3, 2023

shamardy Aug 3, 2023

shamardy Aug 3, 2023 •

edited

caglaryucekaya Aug 4, 2023

onur-ozkan commented Aug 1, 2023

borngraced left a comment

shamardy commented Aug 1, 2023

borngraced left a comment

caglaryucekaya left a comment

caglaryucekaya Aug 1, 2023

caglaryucekaya commented Aug 1, 2023

	fn nft_from_row(row: &Row<'_>) -> Result<Nft, SqlError> {
	let json_string: String = row.get(0)?;
	json::from_str(&json_string).map_err(\|e\| SqlError::FromSqlConversionFailure(0, Type::Text, Box::new(e)))
	}

	fn transfer_history_from_row(row: &Row<'_>) -> Result<NftTransferHistory, SqlError> {
	let json_string: String = row.get(0)?;
	json::from_str(&json_string).map_err(\|e\| SqlError::FromSqlConversionFailure(0, Type::Text, Box::new(e)))
	}

	#[derive(Clone, Debug, Deserialize, Serialize)]
	pub struct Nft {
	#[serde(flatten)]
	pub(crate) common: NftCommon,
	pub(crate) chain: Chain,
	pub(crate) block_number_minted: Option<u64>,
	pub(crate) block_number: u64,
	pub(crate) contract_type: ContractType,
	pub(crate) uri_meta: UriMeta,
	}

	#[derive(Clone, Debug, Deserialize, Serialize)]
	pub struct NftTransferHistory {
	#[serde(flatten)]
	pub(crate) common: NftTransferCommon,
	pub(crate) chain: Chain,
	pub(crate) block_number: u64,
	pub(crate) block_timestamp: u64,
	pub(crate) contract_type: ContractType,
	pub(crate) token_uri: Option<String>,
	pub(crate) collection_name: Option<String>,
	pub(crate) image_url: Option<String>,
	pub(crate) token_name: Option<String>,
	pub(crate) status: TransferStatus,
	}

fix(nft): add log_index to history table and use in PK #1926

fix(nft): add log_index to history table and use in PK #1926

Conversation

shamardy commented Aug 1, 2023 • edited

shamardy commented Aug 1, 2023

anarkiVan commented Aug 1, 2023

onur-ozkan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shamardy Aug 3, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

onur-ozkan commented Aug 1, 2023

borngraced left a comment

Choose a reason for hiding this comment

shamardy commented Aug 1, 2023

borngraced left a comment

Choose a reason for hiding this comment

caglaryucekaya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caglaryucekaya commented Aug 1, 2023

shamardy commented Aug 1, 2023 •

edited

shamardy Aug 3, 2023 •

edited