New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Randomize JoinSplit input and output orders #1561

Merged
merged 6 commits into from Oct 20, 2016

Conversation

Projects
None yet
7 participants
@str4d
Contributor

str4d commented Oct 18, 2016

Closes #778.

str4d added some commits Oct 18, 2016

@str4d str4d added this to the 1.0.0-rc2 milestone Oct 18, 2016

Show outdated Hide outdated src/primitives/transaction.cpp Outdated
Show outdated Hide outdated src/test/transaction_tests.cpp Outdated
if (outputMap[i] == n) {
return i;
}
}

This comment has been minimized.

@daira

daira Oct 18, 2016

Contributor

No find method? Apparently not. Sigh.

@daira

daira Oct 18, 2016

Contributor

No find method? Apparently not. Sigh.

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

Not for json_spirit::Array, because it doesn't have begin() or end().

@str4d

str4d Oct 18, 2016

Contributor

Not for json_spirit::Array, because it doesn't have begin() or end().

if (jsChange > 0) {
changeOutputIndex = find_output(obj, 1);
}
}
}

This comment has been minimized.

@daira
@daira

daira Oct 18, 2016

Contributor

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

I can't follow that link to anywhere useful.

@str4d

str4d Oct 18, 2016

Contributor

I can't follow that link to anywhere useful.

This comment has been minimized.

@daira

daira Oct 18, 2016

Contributor

Line 504 (after the commit).

@daira

daira Oct 18, 2016

Contributor

Line 504 (after the commit).

@@ -845,11 +867,20 @@ Object AsyncRPCOperation_sendmany::perform_joinsplit(
);
// Generate the proof, this can take over a minute.
JSDescription jsdesc(*pzcashParams,
boost::array<libzcash::JSInput, ZC_NUM_JS_INPUTS> inputs

This comment has been minimized.

@daira

daira Oct 18, 2016

Contributor

This can be inputs(info.vjsin); I think.

@daira

daira Oct 18, 2016

Contributor

This can be inputs(info.vjsin); I think.

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

Not with boost::array AFAICT.

@str4d

str4d Oct 18, 2016

Contributor

Not with boost::array AFAICT.

This comment has been minimized.

@daira

daira Oct 18, 2016

Contributor

Huh, I thought I checked the docs.

@daira

daira Oct 18, 2016

Contributor

Huh, I thought I checked the docs.

This comment has been minimized.

@daira

daira Oct 18, 2016

Contributor
// construct/copy/destruct
template<typename U> array& operator=(const array<U, N>&);

http://www.boost.org/doc/libs/1_62_0/doc/html/boost/array.html

@daira

daira Oct 18, 2016

Contributor
// construct/copy/destruct
template<typename U> array& operator=(const array<U, N>&);

http://www.boost.org/doc/libs/1_62_0/doc/html/boost/array.html

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

That's from another array; info.vjsin is a std::vector<JSInput>.

@str4d

str4d Oct 18, 2016

Contributor

That's from another array; info.vjsin is a std::vector<JSInput>.

This comment has been minimized.

@daira

daira Oct 18, 2016

Contributor

OK, never mind then.

@daira

daira Oct 18, 2016

Contributor

OK, never mind then.

@@ -910,10 +941,21 @@ Object AsyncRPCOperation_sendmany::perform_joinsplit(
encryptedNote2 = HexStr(ss2.begin(), ss2.end());
}
Array arrInputMap;

This comment has been minimized.

@daira

daira Oct 18, 2016

Contributor

This can be arrInputMap(inputMap); instead of the loop, I think.

@daira

daira Oct 18, 2016

Contributor

This can be arrInputMap(inputMap); instead of the loop, I think.

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

Not with json_spirit::Array AFAICT.

@str4d

str4d Oct 18, 2016

Contributor

Not with json_spirit::Array AFAICT.

This comment has been minimized.

@nathan-at-least

nathan-at-least Oct 19, 2016

Contributor

[Comment] Maybe some day we should add the standard iteration API to json_spirit::Array? I personally prefer shorter conventional idioms instead of repeated larger code blocks.

@nathan-at-least

nathan-at-least Oct 19, 2016

Contributor

[Comment] Maybe some day we should add the standard iteration API to json_spirit::Array? I personally prefer shorter conventional idioms instead of repeated larger code blocks.

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Oct 18, 2016

Contributor

ut(ACK+cov) modulo the stale comment at 2eeb6be#diff-63ec75bfb71c2d4e26f2a95d5168f5e5R504 . Addressing other comments is optional. Note that I wrote the first two commits with @str4d, so another ACK is needed.

Contributor

daira commented Oct 18, 2016

ut(ACK+cov) modulo the stale comment at 2eeb6be#diff-63ec75bfb71c2d4e26f2a95d5168f5e5R504 . Addressing other comments is optional. Note that I wrote the first two commits with @str4d, so another ACK is needed.

str4d added some commits Oct 18, 2016

@str4d

This comment has been minimized.

Show comment
Hide comment
@str4d

str4d Oct 18, 2016

Contributor

@daira comments addressed.

Contributor

str4d commented Oct 18, 2016

@daira comments addressed.

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Oct 18, 2016

Contributor

ut(ACK+cov). Still needs another ACK.

Contributor

daira commented Oct 18, 2016

ut(ACK+cov). Still needs another ACK.

@ebfull

This comment has been minimized.

Show comment
Hide comment
@ebfull

ebfull Oct 18, 2016

Contributor

This looks good to me, cannot review it extensively enough to ACK though.

Contributor

ebfull commented Oct 18, 2016

This looks good to me, cannot review it extensively enough to ACK though.

@@ -25,6 +26,29 @@ int GetRandInt(int nMax);
uint256 GetRandHash();
/**
* Rearranges the elements in the range [first,first+len) randomly, assuming

This comment has been minimized.

@bitcartel

bitcartel Oct 18, 2016

Contributor

) should be ] ?

@bitcartel

bitcartel Oct 18, 2016

Contributor

) should be ] ?

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

No, the end of the range is exclusive.

@str4d

str4d Oct 18, 2016

Contributor

No, the end of the range is exclusive.

This comment has been minimized.

@bitcartel

bitcartel Oct 19, 2016

Contributor

Hmm, ok, maybe it's a C++ thing to use interval notation in documentation. I can't recall seeing this style for other languages.

@bitcartel

bitcartel Oct 19, 2016

Contributor

Hmm, ok, maybe it's a C++ thing to use interval notation in documentation. I can't recall seeing this style for other languages.

* that gen is a uniform random number generator. Follows the same algorithm as
* std::shuffle in C++11 (a Durstenfeld shuffle).
*
* The elements in the range [mapFirst,mapFirst+len) are rearranged according to

This comment has been minimized.

@bitcartel

bitcartel Oct 18, 2016

Contributor

) should be ] ?

@bitcartel

bitcartel Oct 18, 2016

Contributor

) should be ] ?

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

No, the end of the range is exclusive.

@str4d

str4d Oct 18, 2016

Contributor

No, the end of the range is exclusive.

* The elements in the range [mapFirst,mapFirst+len) are rearranged according to
* the same permutation, enabling the permutation to be tracked by the caller.
*
* gen takes an integer n and produces a uniform random output in [0,n).

This comment has been minimized.

@bitcartel

bitcartel Oct 18, 2016

Contributor

) should be ] ?

@bitcartel

bitcartel Oct 18, 2016

Contributor

) should be ] ?

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

No, the end of the range is exclusive.

@str4d

str4d Oct 18, 2016

Contributor

No, the end of the range is exclusive.

std::function<int(int)> gen)
{
for (size_t i = len-1; i > 0; --i) {
auto r = gen(i+1);

This comment has been minimized.

@bitcartel

bitcartel Oct 18, 2016

Contributor

function gen could return a value which is out of bounds for the iterator.

@bitcartel

bitcartel Oct 18, 2016

Contributor

function gen could return a value which is out of bounds for the iterator.

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

The definition of gen is that it returns a value in [0,n); I will add assertions that this is the case.

@str4d

str4d Oct 18, 2016

Contributor

The definition of gen is that it returns a value in [0,n); I will add assertions that this is the case.

@@ -429,6 +446,10 @@ bool AsyncRPCOperation_sendmany::main_impl() {
}
obj = perform_joinsplit(info, outPoints);
if (jsChange > 0) {
changeOutputIndex = find_output(obj, 1);

This comment has been minimized.

@bitcartel

bitcartel Oct 18, 2016

Contributor

This should never return -1 right? Since we always have two output notes?

@bitcartel

bitcartel Oct 18, 2016

Contributor

This should never return -1 right? Since we always have two output notes?

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

It should never return -1 because outputmap is always length 2 and contains 1. If we want to have a stronger assertion, I can add it.

@str4d

str4d Oct 18, 2016

Contributor

It should never return -1 because outputmap is always length 2 and contains 1. If we want to have a stronger assertion, I can add it.

This comment has been minimized.

@nathan-at-least

nathan-at-least Oct 19, 2016

Contributor

[Comment] I would like an assertion here so that with minimal effort a new reader doesn't have to know those "remote invariants", eg: always length 2. (Does not block.)

@nathan-at-least

nathan-at-least Oct 19, 2016

Contributor

[Comment] I would like an assertion here so that with minimal effort a new reader doesn't have to know those "remote invariants", eg: always length 2. (Does not block.)

This comment has been minimized.

@daira

daira Oct 19, 2016

Contributor

We could raise std::logic_error instead of returning -1; that would avoid the risk of an out-of-bounds access if the -1 were used.

@daira

daira Oct 19, 2016

Contributor

We could raise std::logic_error instead of returning -1; that would avoid the risk of an out-of-bounds access if the -1 were used.

@@ -645,6 +662,10 @@ bool AsyncRPCOperation_sendmany::main_impl() {
}
obj = perform_joinsplit(info, witnesses, jsAnchor);
if (jsChange > 0) {
changeOutputIndex = find_output(obj, 1);

This comment has been minimized.

@bitcartel

bitcartel Oct 18, 2016

Contributor

Ditto above, this should never be -1 right?

@bitcartel

bitcartel Oct 18, 2016

Contributor

Ditto above, this should never be -1 right?

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

Yep.

@str4d

str4d Oct 18, 2016

Contributor

Yep.

template <typename RandomAccessIterator, typename MapRandomAccessIterator>
void MappedShuffle(RandomAccessIterator first,
MapRandomAccessIterator mapFirst,
size_t len,

This comment has been minimized.

@bitcartel

bitcartel Oct 18, 2016

Contributor

The len parameter can be removed. Both iterators should span the same distance, so you can compute the len value in the body of the function with something like size_t len = std::distance(first.begin(), first.end()). Could use an assert to ensure both iterators are of the same len,

@bitcartel

bitcartel Oct 18, 2016

Contributor

The len parameter can be removed. Both iterators should span the same distance, so you can compute the len value in the body of the function with something like size_t len = std::distance(first.begin(), first.end()). Could use an assert to ensure both iterators are of the same len,

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

Or just last-first, which is what the equivalence code in std::shuffle uses. But that means we have to pass in both begin() and end() for both collections (which is why @daira and I opted at the time for len). If you prefer that style, I can change it.

@str4d

str4d Oct 18, 2016

Contributor

Or just last-first, which is what the equivalence code in std::shuffle uses. But that means we have to pass in both begin() and end() for both collections (which is why @daira and I opted at the time for len). If you prefer that style, I can change it.

This comment has been minimized.

@daira

daira Oct 19, 2016

Contributor

I think this is clearer as it is.

@daira

daira Oct 19, 2016

Contributor

I think this is clearer as it is.

This comment has been minimized.

@bitcartel

bitcartel Oct 19, 2016

Contributor

I don't mind.

@bitcartel

bitcartel Oct 19, 2016

Contributor

I don't mind.

// Randomize the order of the inputs and outputs
inputMap = {0, 1};
outputMap = {0, 1};
MappedShuffle(inputs.begin(), inputMap.begin(), ZC_NUM_JS_INPUTS, gen);

This comment has been minimized.

@bitcartel

bitcartel Oct 18, 2016

Contributor

Assert that inputs and inputMap are the same length? And see comment below about not needing the ZC_NUM_JS_INPUTS parameter.

@bitcartel

bitcartel Oct 18, 2016

Contributor

Assert that inputs and inputMap are the same length? And see comment below about not needing the ZC_NUM_JS_INPUTS parameter.

This comment has been minimized.

@daira

daira Oct 19, 2016

Contributor

You can already locally see that they must be the same length, from the declarations, so I don't think that an assertion adds anything.

@daira

daira Oct 19, 2016

Contributor

You can already locally see that they must be the same length, from the declarations, so I don't think that an assertion adds anything.

@@ -13,7 +13,7 @@
using ::testing::Return;
ZCJoinSplit* params = ZCJoinSplit::Unopened();
extern ZCJoinSplit* params;

This comment has been minimized.

@bitcartel

bitcartel Oct 18, 2016

Contributor

Why is this changing? Thanks.

@bitcartel

bitcartel Oct 18, 2016

Contributor

Why is this changing? Thanks.

This comment has been minimized.

@str4d

str4d Oct 18, 2016

Contributor

Because otherwise g++ complains that params is defined twice.

@str4d

str4d Oct 18, 2016

Contributor

Because otherwise g++ complains that params is defined twice.

@bitcartel

This comment has been minimized.

Show comment
Hide comment
@bitcartel

bitcartel Oct 18, 2016

Contributor

First review pass complete and ran tests. Will review a second time after feedback on comments.

Contributor

bitcartel commented Oct 18, 2016

First review pass complete and ran tests. Will review a second time after feedback on comments.

@nathan-at-least

ACK code review.

Not-yet-ACK'd test coverage.

@@ -25,6 +26,29 @@ int GetRandInt(int nMax);
uint256 GetRandHash();
/**
* Rearranges the elements in the range [first,first+len) randomly, assuming
* that gen is a uniform random number generator. Follows the same algorithm as
* std::shuffle in C++11 (a Durstenfeld shuffle).

This comment has been minimized.

@nathan-at-least

nathan-at-least Oct 19, 2016

Contributor

[comment] I'm curious why we don't use std::shuffle if we're reimplementing the same algorithm. If we std::shuffle a list of pairs of (i, x) where i is the original index, could we achieve the same result while reusing std? (I can't tell if that'd be nicer or worse for ergonomics, yet.)

@nathan-at-least

nathan-at-least Oct 19, 2016

Contributor

[comment] I'm curious why we don't use std::shuffle if we're reimplementing the same algorithm. If we std::shuffle a list of pairs of (i, x) where i is the original index, could we achieve the same result while reusing std? (I can't tell if that'd be nicer or worse for ergonomics, yet.)

This comment has been minimized.

@daira

daira Oct 19, 2016

Contributor

@bitcartel originally used that approach, but the specification of std::shuffle and URNG are not strong enough to guarantee determinism (needed for first and firstMap to be permuted in the same way), use of a secure RNG, and a uniform distribution over permutations.

@daira

daira Oct 19, 2016

Contributor

@bitcartel originally used that approach, but the specification of std::shuffle and URNG are not strong enough to guarantee determinism (needed for first and firstMap to be permuted in the same way), use of a secure RNG, and a uniform distribution over permutations.

This comment has been minimized.

@bitcartel

bitcartel Oct 19, 2016

Contributor

I think you mean @str4d ?

@bitcartel

bitcartel Oct 19, 2016

Contributor

I think you mean @str4d ?

Show outdated Hide outdated src/random.h Outdated
@@ -429,6 +446,10 @@ bool AsyncRPCOperation_sendmany::main_impl() {
}
obj = perform_joinsplit(info, outPoints);
if (jsChange > 0) {
changeOutputIndex = find_output(obj, 1);

This comment has been minimized.

@nathan-at-least

nathan-at-least Oct 19, 2016

Contributor

[Comment] I would like an assertion here so that with minimal effort a new reader doesn't have to know those "remote invariants", eg: always length 2. (Does not block.)

@nathan-at-least

nathan-at-least Oct 19, 2016

Contributor

[Comment] I would like an assertion here so that with minimal effort a new reader doesn't have to know those "remote invariants", eg: always length 2. (Does not block.)

@@ -910,10 +941,21 @@ Object AsyncRPCOperation_sendmany::perform_joinsplit(
encryptedNote2 = HexStr(ss2.begin(), ss2.end());
}
Array arrInputMap;

This comment has been minimized.

@nathan-at-least

nathan-at-least Oct 19, 2016

Contributor

[Comment] Maybe some day we should add the standard iteration API to json_spirit::Array? I personally prefer shorter conventional idioms instead of repeated larger code blocks.

@nathan-at-least

nathan-at-least Oct 19, 2016

Contributor

[Comment] Maybe some day we should add the standard iteration API to json_spirit::Array? I personally prefer shorter conventional idioms instead of repeated larger code blocks.

@str4d

This comment has been minimized.

Show comment
Hide comment
@str4d

str4d Oct 19, 2016

Contributor

Pending concerns addressed.

Contributor

str4d commented Oct 19, 2016

Pending concerns addressed.

@bitcartel

This comment has been minimized.

Show comment
Hide comment
@bitcartel

bitcartel Oct 20, 2016

Contributor

ACK (have not checked code coverage)

Contributor

bitcartel commented Oct 20, 2016

ACK (have not checked code coverage)

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Oct 20, 2016

Contributor

ACK aa36398. I'm happy with coverage, so @zkbot r+

Contributor

daira commented Oct 20, 2016

ACK aa36398. I'm happy with coverage, so @zkbot r+

@zkbot

This comment has been minimized.

Show comment
Hide comment
@zkbot

zkbot Oct 20, 2016

Contributor

📌 Commit aa36398 has been approved by daira

Contributor

zkbot commented Oct 20, 2016

📌 Commit aa36398 has been approved by daira

zkbot pushed a commit that referenced this pull request Oct 20, 2016

zkbot
Auto merge of #1561 - str4d:778-randomise-note-order, r=daira
Randomize JoinSplit input and output orders

Closes #778.
@zkbot

This comment has been minimized.

Show comment
Hide comment
@zkbot

zkbot Oct 20, 2016

Contributor

⌛️ Testing commit aa36398 with merge 0dfe612...

Contributor

zkbot commented Oct 20, 2016

⌛️ Testing commit aa36398 with merge 0dfe612...

@zkbot

This comment has been minimized.

Show comment
Hide comment
@zkbot

zkbot Oct 20, 2016

Contributor

☀️ Test successful - zcash

Contributor

zkbot commented Oct 20, 2016

☀️ Test successful - zcash

@zkbot zkbot merged commit aa36398 into zcash:master Oct 20, 2016

1 check passed

homu Test successful
Details
@Zenitur

This comment has been minimized.

Show comment
Hide comment
@Zenitur

Zenitur commented Oct 29, 2016

@bitcartel

This comment has been minimized.

Show comment
Hide comment
@bitcartel

bitcartel Oct 29, 2016

Contributor

@Zenitur What version of g++ are you using? What's your OS environment?

Contributor

bitcartel commented Oct 29, 2016

@Zenitur What version of g++ are you using? What's your OS environment?

@Zenitur

This comment has been minimized.

Show comment
Hide comment
@Zenitur

Zenitur Oct 29, 2016

@bitcartel, Debian Linux 7.0 Wheezy, x86_64 arch

I have been copied binaries compiled with GCC 4.9.2 from another PC. Now I just tell you about an error in code

Zenitur commented Oct 29, 2016

@bitcartel, Debian Linux 7.0 Wheezy, x86_64 arch

I have been copied binaries compiled with GCC 4.9.2 from another PC. Now I just tell you about an error in code

@Zenitur

This comment has been minimized.

Show comment
Hide comment
@Zenitur

Zenitur Oct 30, 2016

After applying #838 (comment) there is no compiling errors anymore

Zenitur commented Oct 30, 2016

After applying #838 (comment) there is no compiling errors anymore

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment