Cleanups, refactoring, and placeholder for TEX addresses #1423

daira · 2024-06-15T02:30:35Z

This PR includes several other cleanups. Changes related to change strategy will be split out to another PR.
fixes #1424

codecov · 2024-06-15T02:46:22Z

Codecov Report

Attention: Patch coverage is 58.72093% with 71 lines in your changes missing coverage. Please review.

Project coverage is 63.07%. Comparing base (dd3a557) to head (0b7f60d).

Files	Patch %	Lines
zcash_client_backend/src/data_api/wallet.rs	26.66%	11 Missing ⚠️
...ent_backend/src/data_api/wallet/input_selection.rs	28.57%	10 Missing ⚠️
zcash_client_backend/src/fees/orchard.rs	0.00%	6 Missing ⚠️
zcash_keys/src/address.rs	0.00%	6 Missing ⚠️
zcash_primitives/src/transaction/builder.rs	50.00%	5 Missing ⚠️
zcash_client_backend/src/fees/common.rs	81.81%	4 Missing ⚠️
zcash_client_backend/src/proposal.rs	69.23%	4 Missing ⚠️
...mponents/zcash_address/src/kind/unified/address.rs	0.00%	3 Missing ⚠️
zcash_client_sqlite/src/lib.rs	25.00%	3 Missing ⚠️
..._sqlite/src/wallet/init/migrations/ufvk_support.rs	25.00%	3 Missing ⚠️
... and 8 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1423      +/-   ##
==========================================
+ Coverage   63.03%   63.07%   +0.04%     
==========================================
  Files         127      127              
  Lines       14784    14821      +37     
==========================================
+ Hits         9319     9349      +30     
- Misses       5465     5472       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

zcash_client_backend/src/fees.rs

nuttycom

I don't think that OutPoint::NULL needs to be a public constant (and it may not need to be a constant at all. Otherwise LGTM.

zcash_primitives/src/transaction/components/transparent.rs

zcash_client_backend/CHANGELOG.md

requested changes addressed

zcash_client_backend/src/data_api.rs

zcash_client_backend/src/fees/common.rs

str4d

Reviewed 830eb7e.

str4d · 2024-06-17T13:20:51Z

components/zcash_encoding/src/lib.rs

+    /// Returns the serialized size of a vector of `u8` as written by `[Vector::write]`.
+    pub fn serialized_size(vec: &[u8]) -> usize {
+        let length = vec.len();
+        CompactSize::serialized_size(length) + length
+    }


I don't like using Vector::serialized_size (an untyped name) for specifically &[u8]. Instead we should either have some kind of Measurable trait, or this should take two arguments (which can be s.len(), 1) for s: &[u8]):

Suggested change

/// Returns the serialized size of a vector of `u8` as written by `[Vector::write]`.

pub fn serialized_size(vec: &[u8]) -> usize {

let length = vec.len();

CompactSize::serialized_size(length) + length

}

/// Returns the serialized size of a vector.

///

/// - `element_count` is the number of elements in the vector.

/// - `element_size` is the size in bytes of an element encoding as written by the

/// function provided to `[Vector::write]`.

pub fn serialized_size(element_count: usize, element_size: usize) -> usize {

CompactSize::serialized_size(element_count) + element_count * element_size

}

The arithmetic can overflow that way. As I had it, it can't for counts that ever occur in practice (due to MAX_COMPACT_SIZE). Will renaming the method to serialized_size_of_u8_vec do?

The addition in the existing PR diff can also overflow (in fewer circumstances, but still) for very large length; I don't consider that a reason to not have the generic code. If we want to ensure it doesn't, then do a checked mul and panic on overflow (and document that the method panics if element_count * element_size + 9 exceeds usize::MAX.

But if you don't want to do that, then renaming to Vector::serialized_size_for_u8 would be fine.

components/zcash_encoding/src/lib.rs

zcash_primitives/src/legacy.rs

str4d · 2024-06-17T13:51:15Z

zcash_primitives/src/legacy.rs

@@ -322,6 +322,11 @@ impl Script {
        Vector::write(&mut writer, &self.0, |w, e| w.write_u8(*e))
    }

+    /// Returns the length of this script as encoded (including the initial CompactSize).
+    pub fn serialized_size(&self) -> usize {


Note to us for after this PR: we need to bump the zcash_encoding dependency of zcash_primitives to 0.2.1 (once that is released) due to this new method (which uses a method not present in zcash_encoding 0.2.0).

zcash_primitives/src/transaction/fees.rs

zcash_client_backend/src/fees/common.rs

str4d · 2024-06-17T15:51:16Z

zcash_client_backend/src/fees/zip317.rs

-                &Vec::<Infallible>::new()[..],
-            ),
+            &orchard_fees::EmptyBundleView,
+            &DustOutputPolicy::new(DustAction::AllowDustChange, Some(NonNegativeAmount::ZERO)),


Why was the change_with_transparent_payments test altered to use this dust output policy (instead of DustOutputPolicy::default())? AFAICT the test is not transparent-only (it has a Sapling input), and therefore the change to the fee code should not have affected it. Is this change perhaps necessary due to the bug I identified? If so, revert it.

Please move this comment to #1430.

str4d · 2024-06-17T16:14:02Z

zcash_client_backend/src/fees/zip317.rs

+            result,
+            Ok(balance) if
+                balance.proposed_change() == [ChangeValue::sapling(NonNegativeAmount::const_from_u64(1000), None)] &&
+                balance.fee_required() == NonNegativeAmount::const_from_u64(10000)


AFAICT this test passes despite the bug I found, for two reasons:

The proposed change output counts are ignored at change output creation time; the single change output is always created even when single_change_output_policy says no change outputs should be created.

The grace_actions value for ZIP 317 means that the fee calculation allows for the extra logical action that this change output consumes.

Please move this comment to #1430 if it is still relevant.

Shouldn't apply now that the bug has been fixed, assuming tests pass.

nuttycom

Reviewed 830eb7e

nuttycom · 2024-06-17T16:18:13Z

zcash_client_backend/src/fees/zip317.rs

+            result,
+            Ok(balance) if
+                balance.proposed_change() == [ChangeValue::sapling(NonNegativeAmount::const_from_u64(1000), None)] &&
+                balance.fee_required() == NonNegativeAmount::const_from_u64(10000)


Bug: the Sapling builder always pads outputs, so if there is one Sapling output, the builder will always create two, and so this transaction will require a 15,000 ZAT fee: 5000 ZAT for the transparent action, and 10,000 ZAT for the two Sapling actions created by padding. This is why I don't like the padding being part of the low-level builder; everything potentially downstream of it needs to predict that behavior.

Please move this comment to #1430 if it is still relevant.

nuttycom

The issue with dust change and builder padding is blocking.

zcash_client_backend/src/fees/common.rs

nuttycom · 2024-06-18T18:58:22Z

zcash_client_backend/src/fees/common.rs

+                .dust_threshold()
+                .unwrap_or(default_dust_threshold);
+
+            if proposed_change < dust_threshold {


If proposed_change is zero (i.e. total_in == total_out_plus_fee_with_change), there is no actual dust output being proposed. I think that while AllowDustChange and AddDustToFee make sense, the behavior of DustAction::Reject is somewhat nonintuitive in this case, in particular because we want to produce zero-valued change outputs in order to hide the exact input value in the case that a transaction has exactly two real outputs.

Please move this comment to #1430 if it is still relevant.

nuttycom · 2024-06-18T19:01:29Z

zcash_client_backend/src/fees/common.rs

+                    }
+                    DustAction::AllowDustChange => simple_case(),
+                    DustAction::AddDustToFee => {
+                        let fee_with_dust = (total_in - total_out).expect("can't happen");


Suggested change

let fee_with_dust = (total_in - total_out).expect("can't happen");

let fee_with_dust = (total_in - total_out).unwrap();

or else provide a better expectation message than "can't happen"

Please move this comment to #1430.

nuttycom · 2024-06-18T19:05:17Z

zcash_client_backend/src/fees/common.rs

+                        .ok_or_else(overflow)?;
+                        if fee_with_dust > reasonable_fee {
+                            // Defend against losing money by using AddDustToFee with a too-high dust threshold.
+                            simple_case()


This case doesn't make sense to me. If fee_with_dust is that large, won't that result in a change output earlier? Shouldn't this return an error?

Please move this comment to #1430.

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

changing its type from `ShieldedProtocol` to `PoolType`. Also fix compilation errors when the "orchard" feature is used without the "transparent-inputs" feature. Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

if a memo is given for the transparent pool. Use `ChangeValue::shielded` to avoid this error case when creating a `ChangeValue` known to be for a shielded pool. Co-authored-by: Jack Grigg <jack@electriccoin.co> Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

`PoolType::{Shielded(_), Transparent}`. Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

inputs and outputs. Co-authored-by: Kris Nuttycombe <kris@nutty.land> Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

`zcash_primitives::legacy::Script::serialized_size`. Use the latter in `zcash_primitives::transaction::fees::transparent::OutputView::serialized_size`. Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

str4d

utACK 98216be

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

str4d

Re-utACK 0b7f60d

daira

Self-ACK

nuttycom

utACK 0b7f60d

daira requested review from str4d and nuttycom June 15, 2024 02:30

daira mentioned this pull request Jun 15, 2024

Add support for transparent-source-only (TEX) addresses #1257

Merged

daira force-pushed the pre-zip-320-refactoring branch 2 times, most recently from b66e93e to 761e9ae Compare June 15, 2024 04:01

nuttycom reviewed Jun 15, 2024

View reviewed changes

zcash_client_backend/src/fees.rs Outdated Show resolved Hide resolved

nuttycom previously requested changes Jun 15, 2024

View reviewed changes

zcash_primitives/src/transaction/components/transparent.rs Outdated Show resolved Hide resolved

zcash_primitives/src/transaction/components/transparent.rs Outdated Show resolved Hide resolved

zcash_client_backend/CHANGELOG.md Outdated Show resolved Hide resolved

daira force-pushed the pre-zip-320-refactoring branch from 761e9ae to f005ab9 Compare June 17, 2024 00:47

daira requested a review from nuttycom June 17, 2024 00:57

daira force-pushed the pre-zip-320-refactoring branch 2 times, most recently from 9e63bb0 to 830eb7e Compare June 17, 2024 13:44

nuttycom reviewed Jun 17, 2024

View reviewed changes

zcash_client_backend/src/data_api.rs Show resolved Hide resolved

nuttycom reviewed Jun 17, 2024

View reviewed changes

zcash_client_backend/src/fees/common.rs Outdated Show resolved Hide resolved

str4d requested changes Jun 17, 2024

View reviewed changes

nuttycom reviewed Jun 17, 2024

View reviewed changes

nuttycom requested changes Jun 17, 2024

View reviewed changes

daira force-pushed the pre-zip-320-refactoring branch 2 times, most recently from 21cb651 to 783833e Compare June 17, 2024 16:58

daira mentioned this pull request Jun 17, 2024

Add a test for correct calculation of ZIP 317 total output size for mixtures of P2PKH and P2SH outputs #1427

Open

daira force-pushed the pre-zip-320-refactoring branch from 8d4ce74 to 4e42357 Compare June 17, 2024 23:47

nuttycom reviewed Jun 18, 2024

View reviewed changes

daira force-pushed the pre-zip-320-refactoring branch from 4e42357 to 34fbbc8 Compare June 18, 2024 20:02

daira changed the title ~~Cleanups and refactoring; change the fee policy to always use change if there are non-transparent flows~~ Cleanups and refactoring Jun 18, 2024

daira and others added 5 commits June 18, 2024 21:03

Cosmetic fixes to changelog.

1d9fede

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Allow ChangeValue::output_pool to reference the transparent pool by

3582f84

changing its type from `ShieldedProtocol` to `PoolType`. Also fix compilation errors when the "orchard" feature is used without the "transparent-inputs" feature. Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Prefer the PoolType::{SAPLING, ORCHARD, TRANSPARENT} constants to

6c90219

`PoolType::{Shielded(_), Transparent}`. Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Add zcash_keys::Address::Tex.

2337f79

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

daira and others added 6 commits June 18, 2024 21:03

Add zcash_client_backend::fees::{sapling,orchard}::EmptyBundleView.

cefbaf5

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Add OutPoint::fake() helper.

6d35583

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Refactor FeeRule::fee_required to take the sizes of transparent

17af8e3

inputs and outputs. Co-authored-by: Kris Nuttycombe <kris@nutty.land> Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Add zcash_encoding::{CompactSize, Vector}::serialized_size and

9881861

`zcash_primitives::legacy::Script::serialized_size`. Use the latter in `zcash_primitives::transaction::fees::transparent::OutputView::serialized_size`. Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Address a review comment concerning Vector::serialized_size.

6f4a3b1

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Use a simpler way of ascribing a type to an empty slice.

c9c7fa3

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

daira force-pushed the pre-zip-320-refactoring branch from 665c06f to 98216be Compare June 18, 2024 20:14

daira changed the title ~~Cleanups and refactoring~~ Cleanups, refactoring, and placeholder for TEX addresses Jun 18, 2024

str4d previously approved these changes Jun 18, 2024

View reviewed changes

daira added 5 commits June 18, 2024 21:24

Miscellaneous documentation improvements.

9881e81

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

"child index" -> "address index" where applicable.

41c30ef

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Fix a typo in an error message.

dcf0187

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Define an error type alias for brevity.

4f935a7

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

Refactoring of change calculation.

0b7f60d

Signed-off-by: Daira-Emma Hopwood <daira@jacaranda.org>

daira dismissed str4d’s stale review via 0b7f60d June 18, 2024 20:24

daira force-pushed the pre-zip-320-refactoring branch from 98216be to 0b7f60d Compare June 18, 2024 20:24

str4d approved these changes Jun 18, 2024

View reviewed changes

daira commented Jun 18, 2024

View reviewed changes

nuttycom approved these changes Jun 18, 2024

View reviewed changes

nuttycom merged commit 06c0895 into zcash:main Jun 18, 2024
26 of 27 checks passed

daira deleted the pre-zip-320-refactoring branch June 18, 2024 22:04

str4d mentioned this pull request Jun 21, 2024

zcash_client_sqlite: Align handling of transparent UTXOs with that of shielded notes. #1402

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cleanups, refactoring, and placeholder for TEX addresses #1423

Cleanups, refactoring, and placeholder for TEX addresses #1423

daira commented Jun 15, 2024 •

edited

Loading

codecov bot commented Jun 15, 2024 •

edited

Loading

nuttycom left a comment

str4d left a comment

str4d Jun 17, 2024

daira Jun 17, 2024

str4d Jun 17, 2024

str4d Jun 17, 2024

str4d Jun 17, 2024

daira Jun 18, 2024

str4d Jun 18, 2024

str4d Jun 17, 2024

daira Jun 18, 2024

str4d Jun 18, 2024

nuttycom left a comment

nuttycom Jun 17, 2024

daira Jun 18, 2024

nuttycom left a comment •

edited

Loading

nuttycom Jun 18, 2024

daira Jun 18, 2024

nuttycom Jun 18, 2024

daira Jun 18, 2024

nuttycom Jun 18, 2024

daira Jun 18, 2024

str4d left a comment

str4d left a comment

daira left a comment

nuttycom left a comment

-    /// Returns the serialized size of a vector of `u8` as written by `[Vector::write]`.
-    pub fn serialized_size(vec: &[u8]) -> usize {
-        let length = vec.len();
-        CompactSize::serialized_size(length) + length
-    }
+    /// Returns the serialized size of a vector.
+    ///
+    /// - `element_count` is the number of elements in the vector.
+    /// - `element_size` is the size in bytes of an element encoding as written by the
+    ///   function provided to `[Vector::write]`.
+    pub fn serialized_size(element_count: usize, element_size: usize) -> usize {
+        CompactSize::serialized_size(element_count) + element_count * element_size
+    }

	let fee_with_dust = (total_in - total_out).expect("can't happen");
	let fee_with_dust = (total_in - total_out).unwrap();

Cleanups, refactoring, and placeholder for TEX addresses #1423

Cleanups, refactoring, and placeholder for TEX addresses #1423

Conversation

daira commented Jun 15, 2024 • edited Loading

codecov bot commented Jun 15, 2024 • edited Loading

Codecov Report

nuttycom left a comment

Choose a reason for hiding this comment

str4d left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuttycom left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nuttycom left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

str4d left a comment

Choose a reason for hiding this comment

str4d left a comment

Choose a reason for hiding this comment

daira left a comment

Choose a reason for hiding this comment

nuttycom left a comment

Choose a reason for hiding this comment

daira commented Jun 15, 2024 •

edited

Loading

codecov bot commented Jun 15, 2024 •

edited

Loading

nuttycom left a comment •

edited

Loading