add payjoin support #156

willowenss · 2023-06-15T20:02:20Z

Adding payjoin support to bdk-cli

src/handlers.rs

notmandatory · 2023-06-15T23:54:53Z

src/handlers.rs

+                .ok_or_else(|| Error::Payjoin("No amount specified in your URI".to_string()))?;
+
+            // creates mutable outputs hashmap with at most one key-value pair.
+            let mut outputs = HashMap::with_capacity(1);


check here for how to use tx_builder to add the recipient.

https://docs.rs/bdk/latest/bdk/wallet/tx_builder/index.html#

notmandatory · 2023-06-16T00:19:13Z

src/handlers.rs

+
+            // all logic below this is temporary and will be deleted or changed
+
+


checkout: https://docs.rs/crate/minreq/latest

In testing we realized minreq has no support for trusting non-root store certificates from the client. So either local testers would have to sudo add the self-signed certificates to their OS root store or get legit CA cert servers for testing. When minreq TLS matches our target MSRV I would definitely consider it again.

For now, reqwest gives us the ability to trust self-signed for testing so it makes more sense to use.

willowenss · 2023-06-19T20:39:44Z

src/handlers.rs


+            // send payjoin request via minreq
+            let response = minreq::post(uri.payjoin_url)


Might need
let response = response.send() after this

rajarshimaitra

Great work. The code looks very clean. I am yet to test it out locally. I like your approach to documenting all the steps. The best way to start writing code is doc everything, even if it's extra it doesn't matter. It's always easier deleting docs later than writing them. 😆

Initial Concept ACK.

I have some minor nit comments, but I will refrain from them until the code finalizes.

I've included one structural comment below for future consideration.

rajarshimaitra · 2023-06-30T06:38:45Z

src/handlers.rs

+        SendPayjoin{
+            uri,
+        } => {
+
+            // convert the bip21 uri into a payjoin uri, and handle error if necessary 
+            let uri = payjoin::Uri::try_from(uri)
+                .map_err(|e| Error::Generic(format!("Unable to convert BIP21 into Payjoin URI: {}", e)))?;
+


I imagine extracting this logic from this match statement into a stand-alone public function could be helpful. If not for export, at least for maintainability.

willowenss · 2023-07-03T18:17:05Z

src/handlers.rs

-            let finalized = wallet      // this returns a bool here
-                .sign(&mut psbt, sign_options);
+            // maybe try allow_grinding?
+            let finalized = wallet.finalize_psbt(&mut psbt, sign_options1)?; // this returns a bool here


When testing, the PSBT response is getting returned as expected from the receiver. However, the bdk-cli sender isn't signing all inputs. I'm getting an ERROR non-mandatory-script-verify-flag when trying to broadcast the transaction because there aren't any signatures. I don't know how to approach this right now, since I thought wallet.sign "Adds to the PSBT all the signatures it can produce".

Can you paste the PSBT here after finalizing it?

From the look of it, the PSBT looks malformed. It has only one witness in one of the inputs, and the other input is empty. There also don't seem to be any PSBT outputs, whereas there are two P2WPKH outputs in the unsigned transaction.

I am not fully sure what's going wrong, but it seems like the payjoin party isn't returning the correct psbt? The one signature in the psbt is probably coming from the wallet itself.

To dig deeper, you probably need to understand psbts a bit better. The easiest thing to do is go by the docs of the psbt structure and try to match why certain components could be missing. https://github.com/rust-bitcoin/rust-bitcoin/blob/83cf389a02d6147a4cd3fe9c42388c61abd40ec2/bitcoin/src/psbt/mod.rs#L44-L63

The Problem

I don't believe the PSBT is malformed, it's partial! I found the problem in BDK: unlike Bitcoin Core's walletprocesspsbt RPC, finalize_psbt only checks if the TxOut script in the PSBT matches the descriptor and does not check whether it has control of the OutPoint specified in the unsigned_tx's TxIn. LND's PSBT signing software does the same thing. While I think BDK should support this lookup in its signer (after 1.0), BIP 78 is also imo flawed by removing the sender's utxo data and ensuring it's not specified. I'll have to keep knocking on @NicolasDorier's door until he believes me.

A Solution

@willowens14 should re-introduce the UTXO (Input Map's TxOut) data to the psbt manually before sending it to the finalizer. Will should be able to copy the Input from the Original PSBT's input map and remove the signature / finalized pieces of data. Once that's in there the finalizer should be able to match the TxO to a descriptor, sign, and finalize.

small note @willowens14 next time you put logs in github you can fence (``` ```) it so it maintains formatting and you don't have to point to a gist if it's short enough

Thanks for the detailed write-up here. That's a deep nuance I missed on BIP78/174 interaction.

src/handlers.rs

DanGould

It works on my machine!

I've asked to clean up a few loose ends and then it's looking ready to merge.

I'm not sure what BDK-CLI's policy is as far as commits go, but this one looks like a contender for squash and merge. @willowens14 for projects who prefer strict control over commit history, you may want to get more acquainted with this kind of commit culture: https://cbea.ms/git-commit/

DanGould · 2023-07-12T14:44:50Z

src/commands.rs

@@ -622,7 +628,7 @@ mod test {
        feature = "compact_filters",
        feature = "rpc"
    ))]
-    use super::OnlineWalletSubCommand::{Broadcast, Sync};
+    use super::OnlineWalletSubCommand::{Broadcast, SendPayjoin, Sync}; // added sendpayjoin here


Thanks for being proactive with comments! Time to get deeper into design review now that you've got payjoin working 🎉

In my experience, comments that describe what changed rather than why can be confusing. The diff tells me SendPayjoin is added here already, so I think there has to be some context that compelled you to write a comment. Why? I think it's better to leave comments describing what changed out. Below, I'll show you what I mean by "why" comments being helpful.

DanGould · 2023-07-12T14:51:22Z

src/handlers.rs

+            // set this to account for original_inputs being of different length than psbt.inputs
+            let mut og = 0;
+
+            // Iterate over the inputs in the PSBT


A for loop comes next, so I know iteration is happening, but I'm not sure why. Isn't the response PSBT processed?

Telling someone why this part of the code exists could help in code comprehension in comparison to "what" comments that can be gleaned from reading the syntax alone. Something like the following should explain why this logic exists.

// BDK only signs scripts that match its target descriptor by iterating through input map. // The BIP 78 spec makes receiver clear sender input map UTXOs, so process_response will fail unless they're cleared. // A PSBT unsigned_tx.input references input OutPoints and not a Script, so the sender signer must either // be able to sign based on OutPoint UTXO lookup or otherwise re-introduce the Script from original_psbt. // Since BDK PSBT signer only checks Input map Scripts for match against its descriptor, it won't sign if they're empty. // Re-add the scripts from the original_psbt in order for BDK to sign properly.

Bonus points for documenting reasons behind particular design decision tradeoffs.

DanGould · 2023-07-12T14:59:43Z

src/handlers.rs

+            // set this to account for original_inputs being of different length than psbt.inputs
+            let mut og = 0;


This comment is exactly what I needed 🙌. It explains why the magic number exists. Without I'd be scratching my head.

DanGould · 2023-07-12T15:49:24Z

src/handlers.rs

+            // set signoptions
+            let mut sign_options = SignOptions::default();
+            sign_options.trust_witness_utxo = true;
+
+            // check + sign psbt
+            let finalized = wallet.sign(&mut psbt, sign_options)?;


Default SignOptions seem to work now that we've got the right data to sign available

Suggested change

// set signoptions

let mut sign_options = SignOptions::default();

sign_options.trust_witness_utxo = true;

// check + sign psbt

let finalized = wallet.sign(&mut psbt, sign_options)?;

let finalized = wallet.sign(&mut psbt, SignOptions::default())?;

DanGould · 2023-07-12T15:49:59Z

src/handlers.rs

+            let mut sign_options1 = SignOptions::default();
+            sign_options1.trust_witness_utxo = true;
+            sign_options1.allow_all_sighashes = true;
+
+            // sign
+            wallet.sign(&mut psbt, sign_options1.clone())?;


Default SignOptions seem to work now that we've got the right data to sign available

Suggested change

let mut sign_options1 = SignOptions::default();

sign_options1.trust_witness_utxo = true;

sign_options1.allow_all_sighashes = true;

// sign

wallet.sign(&mut psbt, sign_options1.clone())?;

wallet.sign(&mut psbt, SignOptions::default())?;

DanGould · 2023-07-12T15:51:01Z

src/handlers.rs

+            // set this to account for original_inputs being of different length than psbt.inputs
+            let mut og = 0;
+
+            // Iterate over the inputs in the PSBT
+            for input in psbt.inputs.iter_mut() {
+                // If this input is some skip it
+                if input.witness_utxo.is_some() || input.non_witness_utxo.is_some() {
+                    continue;
+                }
+                // If index is within the range of original_inputs
+                else if og < original_inputs.len() {
+                    // Reintroduce the utxo from the original_inputs
+                    input.witness_utxo = original_inputs[og].witness_utxo.clone();
+                    input.non_witness_utxo = original_inputs[og].non_witness_utxo.clone();
+
+                    // increment original_index only when an original input is used
+                    og += 1;
+                }
+            }


We should be able to iterate without explicit indexing as in process_response's check_inputs

I suggest renaming psbt to original_psbt and payjoin_psbt respectively. This snippet also requires you to pass a cloned psbt to create_pj_request to keep the original around. It clones slightly more bytes for imo a more readable, maintainable piece of code. I've opted to clone the maps instead of dealing with mutable references in input_pairs for the same reason. I justify this tradeoff because I imagine this BDK-CLI command will be used for individual wallets and sends, and not high throughput enterprise send-a-million-payjoins-at-once scenarios. We need to use mutable references in order to modify the payjoin PSBT. cloning was a bug! PSBTv2 makes this pairing nonsense go away and can't come soon enough!

Payjoin's internal PsbtExt's helpers like this duplicated/modified input_pairs function may be worth exposing publicly in PDK if not rust-bitcoin or BDK. Tagging rust-bitcoin & PDK API expert @Kixunil to weigh in.

Suggested change

// set this to account for original_inputs being of different length than psbt.inputs

let mut og = 0;

// Iterate over the inputs in the PSBT

for input in psbt.inputs.iter_mut() {

// If this input is some skip it

if input.witness_utxo.is_some() || input.non_witness_utxo.is_some() {

continue;

}

// If index is within the range of original_inputs

else if og < original_inputs.len() {

// Reintroduce the utxo from the original_inputs

input.witness_utxo = original_inputs[og].witness_utxo.clone();

input.non_witness_utxo = original_inputs[og].non_witness_utxo.clone();

// increment original_index only when an original input is used

og += 1;

}

}

fn input_pairs(psbt: &mut PartiallySignedTransaction) -> Box<dyn Iterator<Item = (&bdk::bitcoin::TxIn, &mut Input)> + '_> {

Box::new(

psbt.unsigned_tx

.input

.iter()

.zip(&mut psbt.inputs)

)

}

let mut original_inputs = input_pairs(&mut original_psbt).peekable();

for (proposed_txin, mut proposed_psbtin) in input_pairs(&mut payjoin_psbt) {

if let Some((original_txin, original_psbtin)) = original_inputs.peek() {

if proposed_txin.previous_output == original_txin.previous_output {

proposed_psbtin.witness_utxo = original_psbtin.witness_utxo.clone();

proposed_psbtin.non_witness_utxo = original_psbtin.non_witness_utxo.clone();

}

original_inputs.next();

}

}

Yes, for a long time I wanted to move parts from payjoin to rust-bitcoin. This looks like one of them but I don't remember for sure.

DanGould · 2023-07-12T15:56:20Z

src/handlers.rs

+
+            // all logic below this is temporary and will be deleted or changed
+
+


In testing we realized minreq has no support for trusting non-root store certificates from the client. So either local testers would have to sudo add the self-signed certificates to their OS root store or get legit CA cert servers for testing. When minreq TLS matches our target MSRV I would definitely consider it again.

For now, reqwest gives us the ability to trust self-signed for testing so it makes more sense to use.

DanGould · 2023-07-12T16:02:15Z

Note from @notmandatory : "FYI CI is failing on Will’s PR because some dependencies need to be pinned to older versions to work with rust 1.57.. it’s not related to his changes so I’ll need to fix in a different PR"

DanGould · 2023-07-12T19:06:58Z

src/handlers.rs

+
+            // set payjoin params
+            let pj_params = payjoin::send::Configuration::with_fee_contribution(
+                payjoin::bitcoin::Amount::from_sat(100),


This should 1. match the tx_builder's fee rate using min_fee_rate_sat_per_vb or else the receiver can add their input without any fees, reducing the total fee rate and 2. consider adding a fee budget equivalent to receiver adding one input using with_fee_contribution (The recommended fee is size_of_one_input * fee_rate.)

Just use Configuration::non_incentivising for now, we need a utility to identify that recommended fee and change output. Unless there's an easy way to identify the change in BDK already? @notmandatory

I can't think of any easy way to figure out the change output except to look at all of them and use the Wallet.is_mine() function to see which ones are yours, and then take which ever one is not the same as the spending amount, though this is pretty hackey and could fail if both are the same amount. Ideally we should change the Wallet.is_mine() function to return which descriptor the script goes with.

the receiver's change output passed to Wallet.is_mine() will return false? Only true for external account descriptor?

notmandatory · 2023-09-21T01:46:50Z

I screwed up while trying to push a CI fix to this PR. @willowenss please force push 5cc9dd4 from your local branch back to this PR branch. Then we can re-open it.

notmandatory · 2023-09-21T19:34:29Z

Looks like it's passing CI testing! Next steps are:

get commit signing working for your github account
then squash and/or reword your commits to be more descriptive
do final manual testing to confirm it all still works
add any missing tests to make sure it will keep working

https://docs.github.com/en/authentication/managing-commit-signature-verification
https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History

Remove Cargo.lock fixed handlers formatting payjoin 0.8.2 regenerated cargo.lock - pin tokio+tempfile to build with MSRV 1.57.0 Updated pin 8/28 added psbt::input crate

Signed-off-by: willowens14 <will.b.owens@vanderbilt.edu>

notmandatory · 2023-10-18T01:16:05Z

Some new MSRV issues have cropped up, you'll need to amend your last commit to use these cargo update lines:

cargo update -p log --precise 0.4.18
cargo update -p hashlink --precise 0.8.0
cargo update -p tempfile --precise 3.6.0
cargo update -p base64ct --precise 1.5.3
cargo update -p flate2 --precise 1.0.26
cargo update -p h2:0.3.21 --precise 0.3.20
cargo update -p reqwest:0.11.22 --precise 0.11.18
cargo update -p tokio:1.33.0 --precise 1.29.1
cargo update -p tokio-util:0.7.9 --precise 0.7.8
cargo update -p rustls --precise 0.20.8
cargo update -p byteorder --precise "1.4.3"
cargo update -p webpki --precise 0.22.2
cargo update -p cc --precise 1.0.81
cargo update -p os_str_bytes --precise 6.5.1
cargo update -p jobserver --precise 0.1.26

DanGould · 2023-11-22T17:56:13Z

src/handlers.rs

+                        proposed_psbtin.non_witness_utxo = original_psbtin.non_witness_utxo.clone();
+                    }
+                    original_inputs.next();
+                }


Suggested change

proposed_psbtin.non_witness_utxo = original_psbtin.non_witness_utxo.clone();

}

original_inputs.next();

}

proposed_psbtin.non_witness_utxo = original_psbtin.non_witness_utxo.clone();

original_inputs.next();

}

}

Oops! This will skip checking an original_input even if it's not a match, then failing will sign if a match is missed.

DanGould · 2024-01-04T05:03:52Z

Looks like this is failing because it depends on bdk 0.27.1 which depends on an ahash version which was yanked due to a side channel attack vulnerability

I'd really like to close this out, but it looks blocked until #166 is merged or otherwise a 0.27.x patch is released with the new ahash 0.7.7. Tricky situation.

DanGould reviewed Jun 15, 2023

View reviewed changes

src/handlers.rs Outdated Show resolved Hide resolved

notmandatory reviewed Jun 15, 2023

View reviewed changes

notmandatory reviewed Jun 16, 2023

View reviewed changes

willowenss commented Jun 19, 2023

View reviewed changes

willowenss changed the title ~~add payjoin support (very incomplete draft)~~ add payjoin support Jun 26, 2023

rajarshimaitra reviewed Jun 30, 2023

View reviewed changes

willowenss commented Jul 3, 2023

View reviewed changes

rajarshimaitra reviewed Jul 5, 2023

View reviewed changes

src/handlers.rs Outdated Show resolved Hide resolved

DanGould reviewed Jul 6, 2023

View reviewed changes

src/handlers.rs Outdated Show resolved Hide resolved

src/handlers.rs Outdated Show resolved Hide resolved

src/handlers.rs Outdated Show resolved Hide resolved

DanGould requested changes Jul 12, 2023

View reviewed changes

DanGould mentioned this pull request Jul 12, 2023

Fix Payjoin: Add back original PSBT input to payjoin proposal diba-io/bitmask-core#267

Closed

DanGould reviewed Jul 12, 2023

View reviewed changes

This was referenced Jul 20, 2023

ci: pin MSRV dependencies with cargo update and add instructions in README #157

Closed

ci: fix MSRV jobs with cargo update #159

Merged

DanGould assigned willowenss Jul 31, 2023

notmandatory force-pushed the master branch 4 times, most recently from 4042c5a to cef96f5 Compare August 2, 2023 00:21

willowenss force-pushed the master branch 5 times, most recently from 32fad12 to e1d529c Compare August 28, 2023 21:59

willowenss force-pushed the master branch 4 times, most recently from dfa2704 to 5cc9dd4 Compare September 20, 2023 21:58

notmandatory closed this Sep 21, 2023

notmandatory force-pushed the master branch from 5cc9dd4 to 6f61d2e Compare September 21, 2023 00:35

notmandatory mentioned this pull request Sep 21, 2023

Restore for PR 156 #162

Closed

9 tasks

notmandatory reopened this Sep 21, 2023

Add preliminary send payjoin support

1f007db

Remove Cargo.lock fixed handlers formatting payjoin 0.8.2 regenerated cargo.lock - pin tokio+tempfile to build with MSRV 1.57.0 Updated pin 8/28 added psbt::input crate

willowenss force-pushed the master branch from 1323c63 to 7b4f2d0 Compare September 26, 2023 22:46

willowenss force-pushed the master branch from 7b4f2d0 to 5723afc Compare October 16, 2023 22:36

fixing dependencies for MSRV 1.57.0

9ea62be

Signed-off-by: willowens14 <will.b.owens@vanderbilt.edu>

willowenss force-pushed the master branch from 5723afc to 9ea62be Compare October 16, 2023 22:44

DanGould reviewed Nov 22, 2023

View reviewed changes

willowens14 and others added 2 commits December 11, 2023 20:46

fix MSRV issues

2de946d

Format original_inputs.next()

e93f225

DanGould force-pushed the master branch from 33ea38b to e93f225 Compare January 4, 2024 04:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add payjoin support #156

add payjoin support #156

willowenss commented Jun 15, 2023

notmandatory Jun 15, 2023

notmandatory Jun 16, 2023

DanGould Jul 12, 2023

willowenss Jun 19, 2023

rajarshimaitra left a comment

rajarshimaitra Jun 30, 2023

willowenss Jul 3, 2023

rajarshimaitra Jul 3, 2023

rajarshimaitra Jul 5, 2023 •

edited

DanGould Jul 6, 2023 •

edited

rajarshimaitra Jul 7, 2023

DanGould left a comment

DanGould Jul 12, 2023

DanGould Jul 12, 2023

DanGould Jul 12, 2023

DanGould Jul 12, 2023

DanGould Jul 12, 2023

DanGould Jul 12, 2023 •

edited

Kixunil Sep 22, 2023

DanGould Jul 12, 2023

DanGould commented Jul 12, 2023

DanGould Jul 12, 2023 •

edited

notmandatory Jul 19, 2023

DanGould Jul 19, 2023

notmandatory commented Sep 21, 2023 •

edited

notmandatory commented Sep 21, 2023 •

edited

notmandatory commented Oct 18, 2023

DanGould Nov 22, 2023

DanGould commented Jan 4, 2024


		// all logic below this is temporary and will be deleted or changed


		// send payjoin request via minreq
		let response = minreq::post(uri.payjoin_url)

		// set this to account for original_inputs being of different length than psbt.inputs
		let mut og = 0;

add payjoin support #156

Are you sure you want to change the base?

add payjoin support #156

Conversation

willowenss commented Jun 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajarshimaitra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajarshimaitra Jul 5, 2023 • edited

Choose a reason for hiding this comment

DanGould Jul 6, 2023 • edited

Choose a reason for hiding this comment

The Problem

A Solution

Choose a reason for hiding this comment

DanGould left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DanGould Jul 12, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DanGould commented Jul 12, 2023

DanGould Jul 12, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

notmandatory commented Sep 21, 2023 • edited

notmandatory commented Sep 21, 2023 • edited

notmandatory commented Oct 18, 2023

Choose a reason for hiding this comment

DanGould commented Jan 4, 2024

rajarshimaitra Jul 5, 2023 •

edited

DanGould Jul 6, 2023 •

edited

DanGould Jul 12, 2023 •

edited

DanGould Jul 12, 2023 •

edited

notmandatory commented Sep 21, 2023 •

edited

notmandatory commented Sep 21, 2023 •

edited