Simple Payment Verification

[shamardy]

This checklist is for what still needs to be implemented to complete SPV implementation

• Block headers storage optimization, needed for webDEX and Mobile. Implement SPV with minimal storage requirements and fast block headers sync time #1483.
• Faster block headers sync time on enabling a coin. Implement SPV with minimal storage requirements and fast block headers sync time #1483.
• Investigate cpu/resources usage when enabling a coin with SPV and downloading block headers. Will iOS kill the application in the background similar to how it used to happen with ARRR activation?
• Implement block headers storage for IndexedDB and re-enable spv in wasm. The PR that disabled spv in wasm [r2r] hotfix to disable spv in wasm #1479
• When fetching a transaction, if a transaction is not found in an electrum, try next electrum in the sequence to avoid transaction hiding. We should have a fail safe in place for the case of if all electrums are malicious.
• Remove malicious electrum from electrum clients list when detected, and use a different client to complete headers download. Should have a fail safe in place in case all electrums become malicious.
• Detect a chain reorganization, if it occurs. Redownload and revalidate the new best chain headers.
• Research how test chain reorgs using electrum and regtest BTC.
• Allow the user to mark some electrums as trusted if they want, no SPV will be done when using these electrums. If these trusted electrums are down, other electrums will be used with SPV.
• BTC difficulty check is implemented. Difficulty checks for other high hashrate POW coins should be added (LTC, Zcash, etc..). A summery of some difficulty algorithms Summary of Difficulty Algorithms zawy12/difficulty-algorithms#50,
• Research and implement SPV for proof of stake coins.
• Investigate if SPV validation error for block header not found in storage is due to transaction appearing in electrums before the header should be moved to the debug level or not.
• Refactor spv_validation module. Will be required to implement Difficulty checks for other high hashrate POW coins should be added(task 9) more efficiently
• Optimize and fix block header storage todos in WASM after this issue extend indexeddb cursor to query with direction #1643
• Reduce numbers of electrum if swaps fail due to timeout so timeout between each electrum retry can be much longer
• Check SPV server current height with swap transactions to make sure the SPV server is in sync

[borngraced]

LITECOIN DIFFCULTY CHECK

Both Litecoin and Bitcoin retarget their mining difficulty every 2016 blocks. However, due to the 4x faster block speed for Litecoin, mining difficulty retargets occur approximately every 3.5 days. This compares to approximately every 14 days for Bitcoin. — https://en.wikipedia.org/wiki/Litecoin#Differences_from_Bitcoin

hence, here are the required configs needed for LTC

/// LTC_TARGET_TIMESPAN_SECONDS is 1/5 of BTC_TARGET_TIMESPAN_SECONDS, so
const LTC_TARGET_TIMESPAN_SECONDS: u32 = 3.5 * 24 * 60 * 60;
const LTC_MIN_TIMESPAN: u32 = LTC_TARGET_TIMESPAN_SECONDS / RETARGETING_FACTOR;
const LTC_MAX_TIMESPAN: u32 = LTC_TARGET_TIMESPAN_SECONDS* RETARGETING_FACTOR;

will keep updating for rest PoW chains cc. @shamardy

[shamardy]

@borngraced did you try a full chain headers sync for LTC with this config? Did it work fine, no problems with headers deserializations, etc...?

[borngraced]

@borngraced did you try a full chain headers sync for LTC with this config? Did it work fine, no problems with headers deserializations, etc...?

yes!, all positive :}

[borngraced]

SPV lib refactoring idea https://github.com/KomodoPlatform/atomicDEX-API/tree/dev/mm2src/mm2_bitcoin/spv_validation.

This will def get complicated as more chains are added to SPV so I have some ideas for refactoring this lib for flexibility.

• remove and move https://github.com/KomodoPlatform/atomicDEX-API/tree/dev/mm2src/mm2_bitcoin/spv_validation lib from mm2-bitcoin to mm2src.
• create chains module in spv_validation crate and add difficulty checks/algorithm for every chain that's needed to be checked

sample demonstration code

#[async_trait]
pub trait DifficultyCheckOps {
    async fn calculate_next_block_bits(
        coin: &str,
        last_block_header: SPVBlockHeader,
        storage: &dyn BlockHeaderStorageOps,
    ) -> Result<BlockHeaderBits, NextBlockBitsError>;

    async fn retarget_bits(
        coin: &str,
        current_block_timestamp: u32,
        last_block_header: SPVBlockHeader,
        storage: &dyn BlockHeaderStorageOps,
        algorithm: &DifficultyAlgorithm,
    ) -> Result<BlockHeaderBits, NextBlockBitsError>;
}

pub mod chains {
    pub mod litecoin {
        use crate::helpers_validation::DifficultyCheckOps;
        use crate::work::{DifficultyAlgorithm, NextBlockBitsError};
        use chain::BlockHeaderBits;

        pub struct LitecoinDifficulty;

        impl LitecoinDifficulty {
            pub fn new() -> Self { Self }
        }

        impl DifficultyCheckOps for LitecoinDifficulty {
            async fn calculate_next_block_bits(
                algorithm: &DifficultyAlgorithm,
                coin: &str,
                last_block_header: SPVBlockHeader,
                storage: &dyn BlockHeaderStorageOps,
            ) -> Result<BlockHeaderBits, NextBlockBitsError> {
                match algorithm {
                    DifficultyAlgorithm::LitecoinDifficulty => {
                        todo!()
                    },
                    DifficultyAlgorithm::LitecoinTestnet => {
                        todo!()
                    },
                    _ => {},
                }
            }

            async fn retarget_bits(
                coin: &str,
                current_block_timestamp: u32,
                last_block_header: SPVBlockHeader,
                storage: &dyn BlockHeaderStorageOps,
                algorithm: &DifficultyAlgorithm,
            ) -> Result<BlockHeaderBits, NextBlockBitsError> {
                match algorithm {
                    DifficultyAlgorithm::LitecoinDifficulty => {
                        todo!()
                    },
                    DifficultyAlgorithm::LitecoinTestnet => {
                        todo!()
                    },
                    _ => {},
                }
            }
        }
    }

    pub mod btc {}
    pub mod zcash {}
}

pub async fn _next_block_bits(
    coin: &str,
    current_block_timestamp: u32,
    last_block_header: SPVBlockHeader,
    storage: &dyn BlockHeaderStorageOps,
    algorithm: &DifficultyAlgorithm,
) -> Result<BlockHeaderBits, NextBlockBitsError> {
    match algorithm {
        DifficultyAlgorithm::BitcoinMainnet => BitcoinDifficulty::calculate_next_block_bits(
            algorithm,
            coin,
            current_block_timestamp,
            last_block_header,
            storage,
        ),
        DifficultyAlgorithm::BitcoinTestnet => {
            BitcoinDifficulty::calculate_next_block_bits(
                algorithm,
                coin,
                current_block_timestamp,
                last_block_header,
                storage,
            )
            .await
        },
        DifficultyAlgorithm::LitecoinDifficulty => {
            LitecoinDifficulty::calculate_next_block_bits(
                algorithm,
                coin,
                current_block_timestamp,
                last_block_header,
                storage,
            )
            .await
        },
        DifficultyAlgorithm::LitecoinTestnet => {
            LitecoinDifficulty::calculate_next_block_bits(
                algorithm,
                coin,
                current_block_timestamp,
                last_block_header,
                storage,
            )
            .await
        },
    }
}

cc. @shamardy

[borngraced]

SPV is now working in WASM again and can be re-enabled for BTC
cc. @tonymorony @ca33 @shamardy

[shamardy]

remove and move https://github.com/KomodoPlatform/atomicDEX-API/tree/dev/mm2src/mm2_bitcoin/spv_validation lib from mm2-bitcoin to mm2src.

not all spv_validation lib will need to be moved since there are parts that are utxo specific, I get the confusion that this crate is called mm2-bitcoin, it should be renamed mm2_utxo since it includes some common implementations for all utxo coins. Only the block headers storage part will need to be moved if it will be used for other non-utxo coins too but I don't think we will start spv implementation for non-utxos now.

In general, I think this refactor needs to be done gradually (with every new added difficulty algorithm), in the example you provided you assumed the LitecoinDifficulty is different from btc but they use the same calculations so adding LTC as a next step shouldn't need a lot of refactoring as discussed in DM before.

SPV is now working in WASM again and can be re-enabled for BTC

@smk762 @cipig @ca333 since this is a long feature implementation I think we shouldn't enable spv for BTC yet (WASM or otherwise), although it's working. The reason behind this is the coin config can change again since we want a generic spv config for all coins and it changes with each new coin addition, so this might require coin config changes with each new PR. We can re-enable spv once the implementation has settled on a unified spv coin config.

[cipig]

@smk762 @cipig @ca333 since this is a long feature implementation I think we shouldn't enable spv for BTC yet (WASM or otherwise), although it's working. The reason behind this is the coin config can change again since we want a generic spv config for all coins and it changes with each new coin addition, so this might require coin config changes with each new PR. We can re-enable spv once the implementation has settled on a unified spv coin config.

BTC has "enable_spv_proof": true in coins file atm: https://github.com/KomodoPlatform/coins/blob/master/coins#L1819
should i do a PR and set it to false?

[shamardy]

should i do a PR and set it to false?

Just remove the field entirely

[shamardy]

For more details @cipig, I tried a complete BTC headers sync in WASM after this PR was merged and it took a very long time, that's why we will use a recent header (maybe a month old or something) in the config in the future to sync the headers faster, this is how the spv config looks currently #1483 (comment).