Add support for auto-compound in `kaspawallet send` #1951

svarogg · 2022-02-20T10:56:15Z

This PR fixes a big problem in kaspawallet - when the number of inputs is greater then one can fit into a single transaction (such as often happens when a big miner wants to send a lot of kaspa) - the transaction generated is rejected by the node.

This PR splits such transaction into multiple smaller transactions, sending the funds into a change address, then adds another, merge transaction, which would send the funds to their final destination.

Note: there are a quite a few redundant serialize/deserialize of PartiallySignedTransaction(s)
This is there because libkaspawallet exposes methods that work with byte-slices rather then PartiallySignedTransaction(s)
I plan to refactor libkaspawallet to expose only methods for deserialized data in an upcoming PR.

…t that it can be used in kaspawallet

…puts

… pay fees

cmd/kaspawallet/daemon/server/split_transaction.go

…puts

… send due to fees

cmd/kaspawallet/daemon/server/create_unsigned_transaction.go

cmd/kaspawallet/daemon/server/split_transaction.go

stasatdaglabs · 2022-02-27T10:45:01Z

cmd/kaspawallet/daemon/server/create_unsigned_transaction.go

@@ -95,3 +102,32 @@ func (s *server) selectUTXOs(spendAmount uint64, feePerInput uint64) (

 	return selectedUTXOs, totalValue - totalSpend, nil
 }
+
+func (s *server) oneMoreUTXOForMergeTransaction(alreadySelectedUTXOs []*libkaspawallet.UTXO, requiredAmount uint64) (*libkaspawallet.UTXO, error) {


Please move this method to split_transaction.go, close to where it's used

stasatdaglabs · 2022-02-27T11:35:46Z

cmd/kaspawallet/daemon/server/split_transaction.go

+	return serialization.DeserializePartiallySignedTransaction(mergeTransactionBytes)
+}
+
+func (s *server) maybeSplitTransaction(transaction *serialization.PartiallySignedTransaction,


This approach of splitting a transaction doesn't sit well with me. I couldn't figure out a concrete example, but I'm certain there could be edge-cases where this could produce split transactions over the mass limit. What more, we never even check that the transactions created by this method are valid

Instead, I propose a far less efficient, but safe approach:

Start a new transaction

Add inputs and for each one check if we're over the limit

Once we're just under the limit, finish this transaction and move on to the next

I feel ya, and also @someone235 has defined said edge case, but I have a feeling what you suggest is an overkill.

It also will probably be quite annoying, clocking at O(n^2) with a possibly high n, and high constants (with cloning of the tx, etc.), in an operation that is supposed to be instantaneous.

I think the way I did it now is much more foolproof.
Please take a careful look in and around splitAndInputPerSplitCounts. (Also suggestions for a better name welcome).

stasatdaglabs · 2022-02-27T12:02:54Z

cmd/kaspawallet/daemon/server/split_transaction.go

+	sentValue := originalTransaction.Tx.Outputs[0].Value
+	utxos := make([]*libkaspawallet.UTXO, len(splitTransactions))
+	for i, splitTransaction := range splitTransactions {
+		output := splitTransaction.Tx.Outputs[0]


We don't appear to be merging the change. Is it not something we want to do at this stage?

The sum of the outputs of the splits is equal to the total of their inputs. This includes the change as well.

Change is re-calculated again in mergeTransaction:

if totalValue > sentValue { payments = append(payments, &libkaspawallet.Payment{ Address: changeAddress, Amount: totalValue - sentValue, })

(Note the change is going to be a bit lower, because there's fee for the mergeTransaction)

codecov · 2022-03-01T08:08:20Z

Codecov Report

Merging #1951 (88857c8) into dev (9fa0844) will decrease coverage by 0.32%.
The diff coverage is 28.16%.

@@            Coverage Diff             @@
##              dev    #1951      +/-   ##
==========================================
- Coverage   57.86%   57.53%   -0.33%     
==========================================
  Files         676      679       +3     
  Lines       32172    32403     +231     
==========================================
+ Hits        18616    18644      +28     
- Misses      10919    11115     +196     
- Partials     2637     2644       +7

Impacted Files	Coverage Δ
cmd/kaspawallet/broadcast.go	`0.00% <0.00%> (ø)`
cmd/kaspawallet/config.go	`0.00% <ø> (ø)`
cmd/kaspawallet/create_unsigned_tx.go	`0.00% <0.00%> (ø)`
cmd/kaspawallet/daemon/server/address.go	`0.00% <0.00%> (ø)`
...allet/daemon/server/create_unsigned_transaction.go	`0.00% <0.00%> (ø)`
cmd/kaspawallet/daemon/server/server.go	`0.00% <0.00%> (ø)`
cmd/kaspawallet/libkaspawallet/sign.go	`65.38% <ø> (ø)`
cmd/kaspawallet/send.go	`0.00% <0.00%> (ø)`
cmd/kaspawallet/sign.go	`0.00% <0.00%> (ø)`
cmd/kaspawallet/transactions_hex_encoding.go	`0.00% <0.00%> (ø)`
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fa0844...88857c8. Read the comment docs.

…nd-rebase

… sign -> broadcast workflow

stasatdaglabs · 2022-03-06T11:56:03Z

cmd/kaspawallet/broadcast.go

+			return err
+		}
+		fmt.Println("Transaction was sent successfully")
+		fmt.Printf("Transaction ID: \t%s\n", response.TxID)


Please somehow combine the above two logs so that the user doesn't see multiple "Transaction was sent successfully" messages on separate lines.

Did something a little bit different, see what I did and lmk if you no like

stasatdaglabs · 2022-03-06T12:01:25Z

cmd/kaspawallet/daemon/server/split_transaction.go

+		}
+	}
+	if totalValueAdded < requiredAmount {
+		return nil, 0, errors.Errorf("Insufficient funds for merge transaction")


Do we really want to deny a user from sending funds just because they don't have enough money for a merge transaction?

What's the alternative?

Not having funds for merge transaction is more-or-less the same as not having enough fees, just send a little bit less and you'll be fine.

stasatdaglabs · 2022-03-06T12:02:00Z

cmd/kaspawallet/daemon/server/split_transaction.go

+
+	if totalValue < sentValue {
+		// sometimes the fees from compound transactions make the total output higher than what's available from selected
+		// utxos, in such cases - find one more UTXO and use it.


Please update this comment

stasatdaglabs · 2022-03-06T12:51:40Z

cmd/kaspawallet/daemon/server/split_transaction.go

+	return splitTransactions, nil
+}
+
+func (s *server) splitAndInputPerSplitCounts(transaction *serialization.PartiallySignedTransaction, transactionMass uint64) (


For the sake of argument, let's assume our transaction contains:

9 pay to public key inputs, each one has mass 100

1 multisig input with mass 900

Mass limit of 1000

A transaction without any inputs has mass 50

I believe this method would produce:

massWithoutInputs = 50

massForInputs = 1800

inputCount = 10

massPerInput = 180

inputPerSplitCount = 5

splitCount = 2

The splitCount is correct, but if we try to have inputPerSplitCount of 5, we'd get:

One transaction with mass 550

Another transaction with mass 1350

1350 is over the mass limit of 1000

We assume that the transaction was created by kaspawallet, thus it wouldn't have inputs with different masses.
Therefore your scenario is impossible.

I think we should add a comment that explains it. It could also be nice to run some sanity test to check if the assumptions are correct.

I've added a comment.
Adding a sanity test means either refactoring the CalculateMass code to have CalculateMassForInput, or using some not-entirely-accurate proxy such as input size, and I'd rather not do either.

You can check if all inputs are multisig with the same number of required signatures, or all of them are p2pk

cmd/kaspawallet/broadcast.go

someone235 · 2022-03-09T15:35:00Z

cmd/kaspawallet/daemon/server/split_transaction.go

+		}
+	}
+	if totalValueAdded < requiredAmount {
+		return nil, 0, errors.Errorf("Insufficient funds for merge transaction")


What's the alternative?

cmd/kaspawallet/sign.go

cmd/kaspawallet/daemon/server/split_transaction.go

cmd/kaspawallet/transactions_hex_encoding.go

…ltiple

app/appmessage/p2p_msgblock_test.go

cmd/kaspawallet/broadcast.go

someone235 · 2022-03-19T12:58:06Z

cmd/kaspawallet/daemon/server/split_transaction.go

+	return splitTransactions, nil
+}
+
+func (s *server) splitAndInputPerSplitCounts(transaction *serialization.PartiallySignedTransaction, transactionMass uint64) (


I think we should add a comment that explains it. It could also be nice to run some sanity test to check if the assumptions are correct.

someone235 · 2022-03-19T13:17:57Z

cmd/kaspawallet/daemon/server/split_transaction.go

+		massPerInput++
+	}
+
+	inputPerSplitCount = int((mempool.MaximumStandardTransactionMass - massWithoutInputs) / massPerInput)


Maybe rename to inputsPerSplitCount?

Each split tx has only one output right? So wouldn't it make more sense to reduce the mass of one output, instead of all outputs?

Each split tx has only one output right? So wouldn't it make more sense to reduce the mass of one output, instead of all outputs?
When there's no change - there's only 1 output in the original transaction. You are right that in most cases (a.k.a. when there is change) massForInputs wouldn't be a tight bound, but to make it tight I'd need some way to calculate CalculateMassForOutput, and this is something we don't have rn (same problem as with CalculateMassForInput)

Using the outputs of the original transaction as an upper bound is confusing from the reader POV (why use such outputs that are unrelated to the actual split transaction), and the calculation might go wrong: e.g. P2SH is 34 bytes long and P2PK is 33 bytes long, so when you're using a multisig wallet and wanna pay to P2PK, and say your original transaction had one output, the output of the split transaction will use a P2SH address, which requires more mass, so your split tx can go above limit.

So if you don't want to export another function for calculating output mass, can't you just make a stub output that will use some change address and then calculate the output mass in the same way you calculated massForInputs?

…ctions

…nd-rebase

…plit transactions

svarogg added 18 commits February 20, 2022 12:51

Add GetUTXOsByBalances command to rpc

bcdf72b

Fix wrong commands in GetBalanceByAddress

8444a9a

Moved calculation of TransactionMass out of TransactionValidator, so …

bccca21

…t that it can be used in kaspawallet

Allow CreateUnsignedTransaction to return multiple transactions

504309a

Start working on split

3a91e21

Implement maybeSplitTransactionInner

d03d918

estimateMassIncreaseForSignatures should multiply by the number of in…

2be2632

…puts

Implement createSplitTransaction

6218cdb

Implement mergeTransactions

c7d0267

Broadcast all transaction, not only 1

4abd3f1

workaround missing UTXOEntry in partially signed transaction

78b19d9

Bugfix in broadcast loop

529a027

Add underscores in some constants

612726e

Make all nets RelayNonStdTxs: false

15369e2

Change estimateMassIncreaseForSignatures to estimateMassAfterSignatures

135c08b

Allow situations where merge transaction doesn't have enough funds to…

7239bc8

… pay fees

Add comments

dbec7e7

A few of renames

ee60333

svarogg force-pushed the kaspwallet-auto-compound-rebase branch from b97f1de to ee60333 Compare February 20, 2022 11:17

svarogg added 2 commits February 20, 2022 13:21

Handle missed errors

3fbf8bc

Fix clone of PubKeySignaturePair to properly clone nil signatures

f21e1be

someone235 requested changes Feb 20, 2022

View reviewed changes

svarogg added 4 commits February 21, 2022 16:56

Add sanity check to make sure originalTransaction has exactly two out…

e90a374

…puts

Re-use change address for splitAddress

84f3ecd

Add one more utxo if the total amount is smaller then what we need to…

761d461

… send due to fees

Fix off-by-1 error in splitTrasnaction

bbb7323

someone235 requested changes Feb 24, 2022

View reviewed changes

stasatdaglabs requested changes Feb 27, 2022

View reviewed changes

svarogg added 2 commits February 28, 2022 19:50

Add a comment to maybeAutoCompoundTransaction

75f7190

Add comment on why we are increasing inputCountPerSplit

cb0d618

svarogg added 3 commits February 28, 2022 20:06

Add comment explaining while originalTransaction has 1 or 2 outputs

7a09ed2

Move oneMoreUTXOForMergeTransaction to split_transaction.go

608bf69

Allow to add multiple utxos to pay fee for mergeTransactions, if needed

f39bac4

svarogg added 3 commits March 1, 2022 11:47

calculate split input counts and sizes properly

85accca

Merge remote-tracking branch 'origin/dev' into kaspwallet-auto-compou…

e4c453c

…nd-rebase

Allow multiple transactions inside the create-unsigned-transaction ->…

c6523bb

… sign -> broadcast workflow

stasatdaglabs requested changes Mar 6, 2022

View reviewed changes

someone235 requested changes Mar 9, 2022

View reviewed changes

svarogg added 4 commits March 16, 2022 16:53

Print the number of transaction which was sent, in case there were mu…

e3bac53

…ltiple

Rename broadcastConfig.Transaction(File) to Transactions(File)

c60c339

Convert alreadySelectedUTXOs to a map

8cc6187

Fix a typo

1cdc3ca

someone235 requested changes Mar 19, 2022

View reviewed changes

svarogg added 2 commits March 21, 2022 12:01

Add comment explaining that we assume all inputs are the same

ecf6b57

Revert over-refactor of rename of config.Transaction -> config.Transa…

73bfb61

…ctions

stasatdaglabs previously approved these changes Mar 21, 2022

View reviewed changes

Rename: inputPerSplitCount -> inputsPerSplitCount

517063e

svarogg dismissed stasatdaglabs’s stale review via 517063e March 21, 2022 12:38

svarogg added 3 commits March 21, 2022 14:45

Add comment for splitAndInputPerSplitCounts

3ffacb0

Merge remote-tracking branch 'origin/dev' into kaspwallet-auto-compou…

92d9a73

…nd-rebase

Use createSplitTransaction to calculate the upper bound of mass for s…

88857c8

…plit transactions

someone235 approved these changes Mar 27, 2022

View reviewed changes

someone235 merged commit 639183b into kaspanet:dev Mar 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for auto-compound in `kaspawallet send` #1951

Add support for auto-compound in `kaspawallet send` #1951

svarogg commented Feb 20, 2022 •

edited

Loading

stasatdaglabs Feb 27, 2022

stasatdaglabs Feb 27, 2022

svarogg Mar 1, 2022

stasatdaglabs Feb 27, 2022

svarogg Feb 28, 2022

codecov bot commented Mar 1, 2022 •

edited

Loading

stasatdaglabs Mar 6, 2022

svarogg Mar 16, 2022

stasatdaglabs Mar 6, 2022

someone235 Mar 9, 2022

svarogg Mar 16, 2022

stasatdaglabs Mar 6, 2022

stasatdaglabs Mar 6, 2022

svarogg Mar 16, 2022

someone235 Mar 19, 2022

svarogg Mar 21, 2022

someone235 Mar 26, 2022

someone235 Mar 9, 2022

someone235 Mar 19, 2022

someone235 Mar 19, 2022

someone235 Mar 19, 2022

svarogg Mar 21, 2022

someone235 Mar 26, 2022

Add support for auto-compound in kaspawallet send #1951

Add support for auto-compound in kaspawallet send #1951

Conversation

svarogg commented Feb 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Mar 1, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add support for auto-compound in `kaspawallet send` #1951

Add support for auto-compound in `kaspawallet send` #1951

svarogg commented Feb 20, 2022 •

edited

Loading

codecov bot commented Mar 1, 2022 •

edited

Loading