multi: send raw tx data in audit and match_status requests #804

buck54321 · 2020-10-31T16:23:25Z

As per off-github conversation. To accommodate SPV clients, who don't have a mempool and so are in the dark until a contract's transaction is mined, we are going to provide the serialized swap transactions to the clients. The server sends sends the data in the msgjson.Audit payload. Server also sends the info in the msgjson.MatchStatusResult, but only for status/side combos for which it is relevant.

The wallet does not make use of the info yet, but it does receive the data though the modified (Wallet).AuditContract method. There are some other changes that will be needed in the way Core handles the audit too, but I think it'll be easier to handle that during work on the wallet.

This work is closely related to #788.

chappjc · 2021-01-20T14:44:43Z

I'll note that not only is this needed for SPV clients, it will make the process more robust in general as slow or generally poor propagation of transactions on the network is less of an issue. I know that requires rebroadcasting the txn at each point, which is not in this PR, but this obviously sets up for that.

chappjc · 2021-01-20T14:53:27Z

client/asset/interface.go

@@ -118,7 +118,7 @@ type Wallet interface {
 	// during a swap. If the coin cannot be found for the coin ID, the
 	// ExchangeWallet should return CoinNotFoundError. This enables the client
 	// to properly handle network latency.
-	AuditContract(coinID, contract dex.Bytes) (AuditInfo, error)
+	AuditContract(coinID, contract, txData dex.Bytes) (AuditInfo, error)


Maybe comment on how txData will be used. I'm actually still wondering how exactly we will robustly get confirmation that the rebroadcast we are planning "succeeded". Presumably we'll decode the txn, validate it (against the provided contract script), and then we'll try to detect sendrawtransaction errors that just indicate the txn already exists, but I'm a little concerned there will be a variety of errors that look different but all stem from the tx already being known, and we may end up declaring audit failures improperly. I guess we'll cross that bridge next.

chappjc · 2021-01-20T15:08:14Z

client/core/core.go

+				counterTxData := metaData.Proof.CounterTxData
+				if len(counterTxData) == 0 {
+					match.swapErr = fmt.Errorf("missing counter-tx data, order %s, match %s", tracker.ID(), match.id)
+					notifyErr("Match status error", "Match %s for order %s is in state %s, but has no maker swap tx data.", dbMatch.Side, tracker.token(), dbMatch.Status)
+					continue


I got the impression from the other changes that this PR would make the client able to work with a server that does not yet provide the CounterTxData. https://github.com/decred/dcrdex/pull/804/files#diff-8962b4c526755f0f98ed9f46ae05827e8edbc43ed5671ea4a2624bdf1617496cR2029 ((*trackedTrade).auditContract). Should this be an error+self-revoke and continue to next match, as you have it, or just pass nil to AuditContract and try to work with it? Ofc AuditContract doesn't use it yet so it will be fine, but I figure this will be the upgrade plan where the server is behind clients.

Yeah. I got a little overzealous here. This is going to be tricky without versioning, because we can't allow the use of SPV wallets on pre-tx-data servers.

itswisdomagain

First pass, looks good overall. Mostly nitpicking below:

client/db/types.go

itswisdomagain · 2021-01-15T01:26:44Z

client/db/types.go

@@ -291,7 +292,8 @@ func (p *MatchProof) Encode() []byte {
 		AddData(auth.RedemptionSig).
 		AddData(uint64Bytes(auth.RedemptionStamp)).
 		AddData(srvRevoked).
-		AddData(selfRevoked)
+		AddData(selfRevoked).
+		AddData(p.CounterTxData)


CounterTx could be ambiguous, other party contract tx or other party redemption tx?
Suggest CounterContractRawTx

client/db/bolt/upgrades.go

dex/msgjson/types.go

server/asset/common.go

server/asset/dcr/dcr.go

server/auth/auth.go

client/db/bolt/upgrades.go

chappjc · 2021-02-23T20:28:57Z

server/swap/swap.go

@@ -1357,6 +1357,7 @@ func (s *Swapper) processInit(msg *msgjson.Message, params *msgjson.Init, stepIn
 		Time:     uint64(swapTimeMs),
 		CoinID:   params.CoinID,
 		Contract: params.Contract,
+		TxData:   contract.TxData(),


Do you think ultimately the client can provide the raw tx in the init request to the server as well? This PR hits the other half of it great though -- supplementing the server-originating audit and match_status requests, so that that client may rebroadcast it in subsequent work.

With our current protocol and assets, we have to fetch the transaction anyway as part of validation. I like the idea of the client providing the details from a standpoint of limiting the server's role as a source of on-chain data, but it's net effect right now would just be more bandwidth. I could be convinced though, if you're adamant.

Nope, just a thought.

JoeGruffins · 2021-02-25T07:38:44Z

server/auth/auth.go

+			var txData, txDataCoin []byte
+			var assetID uint32
+			switch {
+			case status.IsTaker && status.Status == order.MakerSwapCast: // && !status.IsMaker ?


// && !status.IsMaker ?
Is a question? Can't be both maker and taker right?

Can be when trading with self. I've fallen into that mistaken assumption before too. Def. have to consider that situation.

Should uncomment this check then to be safe.

How to handle it correctly depends on the status of the match and how that changes the semantics of TxData. This pertains to the comment in my review:

Regarding the TxData in MatchStatusResult, either we document very clearly that TxData can refer to different transactions depending on the match status (esp. important if user is both maker and taker) or we change to two fields, MakerTxData and TakerTxData, to avoid any ambiguity. What do you think?

The case here also requires MakerSwapCast as the match status, and if taker, that is when you want this particular tx data (the maker's raw swap tx)... and whether it's a self-swap or not doesn't seem to matter, but I could be mistaken.

Since we're all talking about it, I'm suspecting more strongly now that the 'match_status' route's txdata field could use more clarity, even if that means making two separate fields and even setting them when not needed at the current status or recipient if that is clearer.

To clarify, I think the code is right as is. And I don't think sending lots of extra data and when it's not needed is absolutely necessary, but it could make things clearer. Documenting the msgjson.MatchStatusResult extra clearly w.r.t. this field would also be cool.

I'm for having 2 fields for {M,T}akerTxData, even though both need not be set at the same time, similar to the existing {M,T}akerSwap, {M,T}akerContract and {M,T}akerRedeem fields.

JoeGruffins · 2021-02-25T07:55:37Z

client/core/status.go

+	// len(srvData.TxData) == 0 {
+	// 	err = fmt.Errorf("Server is reporting a match in TakerSwapCast with no tx data. %s", logID)
+	// 	return
+	// }
+


Why is this one commented out?

I think this was commented somewhere else, but I guess the client will need to check that the server's version is compatible with itself. An older server would not include the TxData.

This PR is just a part of a few steps to this work (next is presumably using the TxData with bcast), but it seems to me that backward compatibility regarding this TxData field is straightforward for both client and server.

If the TxData is omitted/empty, the client (asset backend) should take the legacy approach of searching for the txn

If the TxData is present, take the new approach of broadcasting it

I don't believe any versioning is required to handle this elegantly and without any hacks.

As such, I don't think this commented code will ever be uncommented given the legacy handling I'm suggesting if it is len 0.

chappjc

Some minor cleanup on client db upgrade code.

Regarding the TxData in MatchStatusResult, either we document very clearly that TxData can refer to different transactions depending on the match status (esp. important if user is both maker and taker) or we change to two fields, MakerTxData and TakerTxData, to avoid any ambiguity. What do you think?

client/db/bolt/upgrades.go

chappjc · 2021-02-25T17:52:43Z

server/auth/auth.go

+			var txData, txDataCoin []byte
+			var assetID uint32
+			switch {
+			case status.IsTaker && status.Status == order.MakerSwapCast: // && !status.IsMaker ?


Can be when trading with self. I've fallen into that mistaken assumption before too. Def. have to consider that situation.

client/core/core.go

client/core/status.go

itswisdomagain · 2021-02-26T03:04:30Z

client/db/types.go

-		RefundCoin:    pushes[8],
+		Script:          pushes[0],
+		CounterContract: pushes[1],
+		CounterTxData:   pushes[21],


Why not pushes[2] to maintain the order and aid readability.
In decodeMatchProof_v1, instead of appending the nil []byte, can insert:

pushes = append(pushes, nil) copy(pushes[3:], pushes[2:]) pushes[2] = nil

I don't really see how this would improve readability. Simpler is better in these upgrades, and changing the index of every read of pushes for index > 1 is more error-prone.

This works for sure, but the readability improvements I mean is having the order of the fields match the order of data in the pushes. So last field in the struct should write to / read from the last data in pushes. It'd become a bit of a eye turner if for example, the first struct field is populated with the 10th data in pushes, the second field is populated with the data at index 3 and the last field is populated with index 1.

What I'm recommending for clarity is:

Script: pushes[0], CounterContract: pushes[1], CounterTxData: pushes[2], SecretHash: pushes[3], Secret: pushes[4], MakerSwap: pushes[5],

instead of

Script: pushes[0], CounterContract: pushes[1], CounterTxData: pushes[21], SecretHash: pushes[2], Secret: pushes[3], MakerSwap: pushes[4],

itswisdomagain · 2021-02-26T03:07:22Z

server/asset/common.go

@@ -26,6 +26,8 @@ type Backend interface {
 	// the blockchain immediately. The redeem script is required in order to
 	// calculate sigScript length and verify pubkeys.
 	Contract(coinID []byte, redeemScript []byte) (Contract, error)
+	// TxData fetches the same data returned by a Contract's TxData method.


Looks like you missed https://github.com/decred/dcrdex/pull/804/files#r557809395

Suggested change

// TxData fetches the same data returned by a Contract's TxData method.

// TxData fetches the raw transaction data of the specified coin.

itswisdomagain · 2021-02-26T03:19:58Z

server/auth/auth.go

+			var txData, txDataCoin []byte
+			var assetID uint32
+			switch {
+			case status.IsTaker && status.Status == order.MakerSwapCast: // && !status.IsMaker ?


Should uncomment this check then to be safe.

itswisdomagain · 2021-02-28T02:10:10Z

client/db/bolt/upgrades_test.go

 // The indices of the archives here in outdatedDBs should match the first
 // eligible validator for the database.
 var outdatedDBs = []string{
 	"v0.db.gz", "v1.db.gz",
 }


Update doc, add "v2.db.gz" or better still make TestUpgradeDB use dbUpgradeTests but only the filename property to avoid having to define filenames twice.

itswisdomagain · 2021-03-01T20:42:36Z

client/core/core.go

+				// TODO: We will need careful handling here. To be clear, the
+				// client should reject orders at the funding stage when the
+				// server doesn't signal compatibility with the requisite
+				// protocols. e.g spv-compatibility. So if we version the api to


I don't think clients need to reject orders when a server does not provide txdata. If they allow orders still, what happens is that they simply can't find the counter contract until it is mined. Even in full node, a client cannot act until the counter contract is mined. Also, in full node, if a client cannot locate the counter contract tx, it doesn't do anything but wait until the server revokes the match, iirc. Add as @chappjc noted in chat, a full node wallet can fail to locate the counter contract tx and have to wait till it is mined if a user restarts their nodes and the mempool loses the txn.

If the above holds true, then the addition of txdata isn't really something done to aid spv-compatibility.
As noted above whether spv or not, clients will wait for txs to be mined before acting and only abandon matches when the server revokes them, not if they cannot find a particular tx.

The real advantage to adding txdata imo, is to ensure that counter parties play their parts by broadcasting valid contracts, with receiving clients having the ability to also broadcast as a fail safe. With this addition, clients (with spv or full node wallets) can abandon matches earlier if a provided tx is invalid or fails audit checks such as address and amount. This is why I recommended in a previous review to not allude the introduction of txdata to spv compatibility - it started that way but I believe things have taken a turn since the change to allow audits run until the contract tx is mined or the server revokes the match.

I'm not quite sure what you're getting at here. If we weren't doing SPV, we wouldn't be making these changes.

The real advantage to adding txdata imo, is to ensure that counter parties play their parts by broadcasting valid contracts, with receiving clients having the ability to also broadcast as a fail safe.

We already do this verification without tx data from the server though. We only need the tx data so that SPV clients can do the same validation immediately, since they won't have access to the mempool data. The fail-safe definition strikes me wrong too. So what has failed, and why if something has failed, are we progressing the trade for the counter-party? I'm not convinced that broadcasting the counter-party's transaction is a route we want to take as a remedial action.

And even though we are no longer using broadcast timeout to secure the client, I'm not sure it's an option we should throw away. We are not very well protected against a malicious server right now.

With this addition, clients (with spv or full node wallets) can abandon matches earlier if a provided tx is invalid or fails audit checks such as address and amount.

Something we can already do for full node-backed wallets. We only need the tx data so that spv clients can do it too.

I do acknowledge that with our new leniency on broadcast timeouts, having the transaction data before mining gives us little advantage. But without tx data our spv clients will be more vulnerable to attacks from malicious servers.

Which brings us back to

I don't think clients need to reject orders when a server does not provide txdata.

This feels like a new spin on the same conversation we've had many times. We do not want our clients sitting in the dark. A malicious server could effectively lock up the funds of every spv client with just a random hash generator.

We do not want our clients sitting in the dark. A malicious server could effectively lock up the funds of every spv client with just a random hash generator.

But this problem already exists, it's not particular to spv wallets. If a server sends random hashes and a client is unable to locate the tx, funds will remain locked until the server revokes the match. On that basis, sending txdata to combat this issue doesn't only apply to spv wallets.

We already do this verification without tx data from the server though. We only need the tx data so that SPV clients can do the same validation immediately, since they won't have access to the mempool data.

Full node-backed wallets are not always currently able to audit contracts since they may not find the tx in their mempool for different possible reasons such as the user restarting wallet node and dexc after counter party has broadcasted their tx. When this happens, the user has to wait till the contract is mined to perform any validation. Even full node-backed wallets. Sending txdata for full node-backed wallet clients gives clients the ability to perform audits on the provided txdata without needing to find the tx in the blockchain first. Agreed, I expect it'll be less common for full node-backed wallets to not find contract txs in their mempool, but it's a very realistic possibility.

I do acknowledge that with our new leniency on broadcast timeouts, having the transaction data before mining gives us little advantage. But without tx data our spv clients will be more vulnerable to attacks from malicious servers.

The only protection full node wallets currently have over spv wallets is they can perform audit and abandon a match sooner if the contract tx doesn't meet expectations - and this advantage only exists if the full node wallet does find the contract tx before it is mined, which as has been established, may not always happen. On this note, sending txdata offers almost just as much extra advantage for full node wallet as it does for spv wallets - provide the ability to audit contracts regardless of whether they can be found in mempool or not.

I think the meat of my point is:

As noted above whether spv or not, clients will wait for txs to be mined before acting and only abandon matches when the server revokes them, not if they cannot find a particular tx.

This is the key reason why it makes very little difference whether a wallet can find a tx in mempool or not, whether spv or full node-backed. If we say spv wallets should not trade if they won't get txdata from the server, then we should say full node-backed wallets should abandon matches if they can't find a tx in mempool. If we allow full node wallets to wait till the tx is mined or the server revokes the match, why aren't we allowing svp wallets to do the same?

The only scenario where I think full node wallets have an advantage is the combination of the following:

the server sends a hash of a real transaction but that does not meet audit requirements

the wallet finds the tx (mined or unmined) and abandons the match without waiting for the server's revoke.

If that were to happen to an spv wallet that does not have the txdata, the client would not abandon the match until the server revokes it or the wallet finds the mined tx and then performs the audit checks.

Still, if a server really wants to be malicious and blind-side spv wallets, the server can provide a valid txdata that passes audit checks but is a double spend and the client will not know even if the client broadcasts the txdata because spv clients do not really have any validation that a rawtx is accepted into the blockchain until it is mined. So again, they have to wait.

That all makes sense, and I agree. Was it just the mention of spv in the TODO that you were concerned with?

Not that we want more options and switches, but I wonder if it would be helpful to have an option for SPV-only clients to "reject orders at the funding stage when the server doesn't signal compatibility". I think we'd have to evaluate the UX for SPV-only clients is in the absence of this data. That is, will the confusion and suspense waiting for the initial confirm be tolerable?

Even in the absence of the transaction data, we can still link to the transaction in a block explorer, which will presumably have it in mempool.

We could almost use dcrdata as an spv mempool, but let's not.

~~@Wisdom~~ @itswisdomagain I'm just going to delete these TODOs and comments for now, if nobody objects.

You tagged the wrong username 😉 . Life would probably be easier if my username everywhere was just [at]wisdom but we move.

server/market/routers_test.go

buck54321 changed the title ~~implement tx data in audit and match_status requests~~ multi: send raw tx data in audit and match_status requests Oct 31, 2020

buck54321 force-pushed the txdata branch from 46c77ab to 61d1a40 Compare January 20, 2021 10:51

buck54321 marked this pull request as ready for review January 20, 2021 12:52

chappjc reviewed Jan 20, 2021

View reviewed changes

itswisdomagain suggested changes Jan 25, 2021

View reviewed changes

chappjc added this to the 0.2 milestone Jan 30, 2021

buck54321 force-pushed the txdata branch from cb60400 to 68f2df2 Compare February 10, 2021 21:24

chappjc reviewed Feb 23, 2021

View reviewed changes

JoeGruffins approved these changes Feb 25, 2021

View reviewed changes

chappjc requested changes Feb 25, 2021

View reviewed changes

itswisdomagain suggested changes Feb 26, 2021

View reviewed changes

itswisdomagain suggested changes Feb 28, 2021

View reviewed changes

buck54321 force-pushed the txdata branch from 68f2df2 to 4b6cd54 Compare March 1, 2021 17:57

itswisdomagain reviewed Mar 1, 2021

View reviewed changes

buck54321 added 4 commits March 10, 2021 12:07

implement tx data in audit and match_status requests

ee91c96

rebase cleanup

bf7f9f1

review followup

dd85461

review followup 2

03dabb1

buck54321 force-pushed the txdata branch from 0eba90c to 03dabb1 Compare March 10, 2021 18:10

chappjc reviewed Mar 10, 2021

View reviewed changes

server/market/routers_test.go Outdated Show resolved Hide resolved

remove oudated unused test methods

68efb12

chappjc approved these changes Mar 10, 2021

View reviewed changes

chappjc merged commit 3704513 into decred:master Mar 10, 2021

	// TxData fetches the same data returned by a Contract's TxData method.
	// TxData fetches the raw transaction data of the specified coin.

multi: send raw tx data in audit and match_status requests #804

multi: send raw tx data in audit and match_status requests #804

Conversation

buck54321 commented Oct 31, 2020

chappjc commented Jan 20, 2021

chappjc Jan 20, 2021 • edited Loading

Choose a reason for hiding this comment

chappjc Jan 20, 2021 • edited Loading

Choose a reason for hiding this comment

buck54321 Jan 28, 2021 • edited Loading

Choose a reason for hiding this comment

itswisdomagain left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chappjc Feb 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chappjc Feb 25, 2021 • edited Loading

Choose a reason for hiding this comment

chappjc left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

buck54321 Mar 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

buck54321 Mar 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chappjc Jan 20, 2021 •

edited

Loading

chappjc Jan 20, 2021 •

edited

Loading

buck54321 Jan 28, 2021 •

edited

Loading

chappjc Feb 26, 2021 •

edited

Loading

chappjc Feb 25, 2021 •

edited

Loading

chappjc left a comment •

edited

Loading

buck54321 Mar 4, 2021 •

edited

Loading

buck54321 Mar 4, 2021 •

edited

Loading