Low memory prover #2670

arielgabizon · 2017-10-18T11:56:20Z

This PR integrates @ebfull 's low memory changes: https://github.com/zcash/zcash/pull/2243/commits
on top of @str4d 's work bringing in libsnark as a subtree
4699d0e

str4d · 2017-10-19T04:39:30Z

Nice! I'm going to review the PR later this week, but one initial comment: please rebase this PR to edit the first commit to restore its original commit message (use git rebase -i master, change the first commit from pick to r, change the commit message, and then git push -f to update this PR).

arielgabizon · 2017-10-19T09:38:54Z

Done.

daira · 2017-10-21T10:03:20Z

The commit message "Delete libsnark tests that we've merged into gtest." does not match the contents of that commit.

…ated invariants.

arielgabizon · 2017-10-21T12:44:23Z

My mistake. Should be fine now. Though rebase still makes me a little uncomfortable so would be happy if someone verified.

daira · 2017-10-21T10:13:54Z

src/snark/src/zk_proof_systems/ppzksnark/r1cs_ppzksnark/r1cs_ppzksnark.tcc

+        const size_t chunks = omp_get_max_threads(); // to override, set OMP_NUM_THREADS env var or call omp_set_num_threads()
+    #else
+        const size_t chunks = 1;
+    #endif


Suggestion for another PR: refactor this to const size_t chunks = MULTICORE_CHUNKS(); and move the preprocessor conditional to a header.

daira · 2017-10-21T10:26:31Z

src/snark/src/zk_proof_systems/ppzksnark/r1cs_ppzksnark/r1cs_ppzksnark.tcc

@@ -439,10 +513,6 @@ r1cs_ppzksnark_proof<ppT> r1cs_ppzksnark_prover(const r1cs_ppzksnark_proving_key
 {
    enter_block("Call to r1cs_ppzksnark_prover");

-#ifdef DEBUG
-    assert(constraint_system.is_satisfied(primary_input, auxiliary_input));
-#endif


Please retain this; we will remove it in #2005 (when verification succeeds), but I want to see the effect of the rest of the PR on performance/memory usage independently of the effect of #2005.

Is there some git trick to add this to the commit where it was deleted? Or should I make a new commit?

git rebase -i <COMMIT_ID_BEFORE_THIS>

Select this commit to be edited (change pick to e)

Edit the file to add this back in (copy from this diff, and remove the leading -s)

Add this change

git commit --amend

git diff HEAD^..HEAD and check this part of the diff is no longer there.

git rebase --continue to finish.

Alternatively:

Make a new commit that exactly reverses this one change.

git rebase -i <COMMIT_ID_BEFORE_THIS>

Move the new commit line underneath this commit's line, and change it from pick to s (squash)

Great explanation!

daira · 2017-10-22T10:00:30Z

src/snark/src/zk_proof_systems/ppzksnark/r1cs_ppzksnark/r1cs_ppzksnark.tcc

-    assert(pk.C_query.domain_size() == qap_wit.num_variables()+2);
-    assert(pk.H_query.size() == qap_wit.degree()+1);
-    assert(pk.K_query.size() == qap_wit.num_variables()+4);
-#endif


These debug assertions have been dropped. Is this intentional?

They could be left in.

I think this can't be left in cause it needs all the proving key at once.

Most of this could be split across the various functions. The for loop would be harder though, and given that we don't set DEBUG, I'm fine with leaving these out.

daira · 2017-10-22T10:04:54Z

src/snark/src/zk_proof_systems/ppzksnark/r1cs_ppzksnark/r1cs_ppzksnark.tcc

-    const Fr<ppT> t = Fr<ppT>::random_element();
-    qap_instance_evaluation<Fr<ppT> > qap_inst = r1cs_to_qap_instance_map_with_evaluation(constraint_system, t);
-    assert(qap_inst.is_satisfied(qap_wit));
-#endif


Is dropping this debug assertion intentional?

It was intentional when we did it. I can't remember the reason why though. It could be left in -- after all, we don't enable DEBUG anyway.

daira · 2017-10-22T10:09:04Z

src/zcash/JoinSplit.cpp


-        return ZCProof(r1cs_ppzksnark_prover<ppzksnark_ppT>(
-            pk,
+        // TODO


What is to do here?

I can't remember, it should be removed.

daira · 2017-10-22T10:10:07Z

src/zcash/JoinSplit.cpp

-            pk,
+        // TODO
+        if(!fh.is_open()) {
+            throw std::runtime_error((boost::format("could not load param file at %s") % pkPath).str());


We don't use boost::format, we use tinyformat.

See #2243 (comment)

also second instance

str4d

Concept ACK and preliminary utACK. I've checked that the refactor appears to be a no-op. Daira's comments need addressing.

ebfull · 2017-10-26T04:09:59Z

src/snark/src/zk_proof_systems/ppzksnark/r1cs_ppzksnark/r1cs_ppzksnark.tcc

-    const Fr<ppT> t = Fr<ppT>::random_element();
-    qap_instance_evaluation<Fr<ppT> > qap_inst = r1cs_to_qap_instance_map_with_evaluation(constraint_system, t);
-    assert(qap_inst.is_satisfied(qap_wit));
-#endif


It was intentional when we did it. I can't remember the reason why though. It could be left in -- after all, we don't enable DEBUG anyway.

ebfull · 2017-10-26T04:13:23Z

src/snark/src/zk_proof_systems/ppzksnark/r1cs_ppzksnark/r1cs_ppzksnark.tcc

-    assert(pk.C_query.domain_size() == qap_wit.num_variables()+2);
-    assert(pk.H_query.size() == qap_wit.degree()+1);
-    assert(pk.K_query.size() == qap_wit.num_variables()+4);
-#endif


They could be left in.

ebfull · 2017-10-26T04:13:59Z

The math is good, the code and logic is good, I like the API changes, nice and clean.

Just might as well put those DEBUG assertions back even if we're not enabling them ever. (I consider that kind of pre-processor-macro-debug-assertion to be an anti-pattern though, since it doesn't expand templates and create static assertions and things like that.)

bitcartel · 2017-10-27T19:33:14Z

There might be a memory leak. Tops out with virtual memory at 9GB and actual usage spiking to 7GB.

VIRT      RES
978988  26544  (after ./zcashd) 
2936904 1.471g (after ./zcash-cli zcbenchmark createjoinsplit 1)
4960356 3.080g (repeat...)
6983808 4.689g
9007260 6.298g
9007260 6.462g
9007260 6.462g (repeat... final 6th invocation)

The above was conducted with zcash in regtest mode.

arielgabizon · 2017-10-27T20:05:12Z

@bitcartel I wonder if it's related to this comment 929b9db#r120993316

If I understand what @daira said there, it's not a problem?

arielgabizon · 2017-10-27T20:06:25Z

Btw, wouldn't you get a similar thing with the regular joinsplit before these changes? I remember memory not going down at the end of a joinsplit when checking with htop

arielgabizon · 2017-10-27T20:23:18Z

corrected link to Daira's comment #2243 (comment)

tromer · 2017-10-27T20:24:31Z

@arielgabizon This depends on the libc implementation (did the allocator use the heap pool or dedicated pages? if it's the heap pool, how fragmented is it?). This is non-deterministic platform-specific voodoo, and I would not be surprised if "often" the memory remains reserved as far as the OS can see. To be sure it's really freed up, we'd have to take over and do the mmap() by ourselves.

arielgabizon · 2017-10-27T20:29:48Z

@tromer I don't know this stuff, and not sure how you do it yourself; would be happy to fix it if someone explained it to me. @daira said here #2679 that it currently allocates on the stack and we should change that. Though she also said in the comment I linked above that it can wait for a future PR.

arielgabizon · 2017-10-27T20:30:41Z

Ohh you perhaps were talking about the joinsplit before this code change.

tromer · 2017-10-27T20:40:57Z

Yes. @daira is right that it should be on the heap, and that's all that matters for the current PR.

An additional issue, which should not be addressed in the current PR, is that even when using the heap (whether it's the old prover or the new prover), the memory may remain reserved unless you use a special allocator. Fixing that would probably be platform-specific and a bit tricky.

arielgabizon · 2017-10-27T21:36:23Z

@bitcartel you might want to try this version I wrote. https://github.com/arielgabizon/zcash/tree/lowmemmovetoheap
It moves the main variable to the heap, and you can see the more (though not all) memory reclaimed after z_sendmany

bitcartel · 2017-10-27T21:55:00Z

@arielg I tried the other branch, same results with zcbenchmark createjoinsplit. Anyway, I don't think this is a blocker, given that this issue already exists and has been documented.

arielgabizon · 2017-10-27T22:03:43Z

Well glad it's not a blocker. I see more memory released in the other branch when using z_sendmany directly. I wonder if the memory increase is cause the benchmark uses a stack variable.
https://github.com/arielgabizon/zcash/blob/lowmemmovetoheap/src/zcbenchmarks.cpp#L118

zkbot · 2017-11-02T09:20:42Z

💔 Test failed - pr-merge

str4d · 2017-11-02T10:19:58Z

One of the Boost tests is failing. The error message:

unknown location(0): fatal error: in "rpc_wallet_tests/rpc_z_sendmany_internals": memory access violation at address: 0x00000848: no mapping at fault address
test/rpc_wallet_tests.cpp(1171): last checkpoint

daira · 2017-11-02T11:57:41Z

Ugh, address looks like an offset from NULL.

bitcartel · 2017-11-04T06:30:40Z

Reproducible. test/test_bitcoin -t rpc_wallet_tests fails in this branch, but passes fine on master branch.

arielgabizon · 2017-11-04T13:05:52Z

Looks like that test fails only with the last commit that puts the ifstream object on the head, which increases running time by 0.5 seconds according to @str4d , maybe just abort that commit for now?

str4d · 2017-11-05T15:05:41Z

Force-pushed to remove 2f598a5 from this PR.

@zkbot try

zkbot · 2017-11-05T15:05:45Z

⌛ Trying commit 4305a56 with merge 94e19023bc8ca82661ae015c3b898dd6b5fbde48...

str4d · 2017-11-05T15:27:07Z

Still seeing that failure:

unknown location(0): fatal error: in "rpc_wallet_tests/rpc_z_sendmany_internals": memory access violation at address: 0x00000270: no mapping at fault address
test/rpc_wallet_tests.cpp(1171): last checkpoint
*** 1 failure is detected in the test module "Bitcoin Test Suite"
test_bitcoin: /home/zcbbworker/latent-0/debian8/build/depends/x86_64-unknown-linux-gnu/share/../include/boost/thread/pthread/condition_variable_fwd.hpp:102: boost::condition_variable::~condition_variable(): Assertion `!ret' failed.

zkbot · 2017-11-05T15:29:57Z

💔 Test failed - pr-try

str4d · 2017-11-05T16:39:29Z

src/test/test_bitcoin.cpp

 BasicTestingSetup::BasicTestingSetup()
 {
-        assert(init_and_check_sodium() != -1);
-        ECC_Start();
-        pzcashParams = ZCJoinSplit::Unopened();


This is the cause of the crash. Previously TestingSetup had access to pzcashParams because it inherits from BasicTestingSetup. With this change, it doesn't have that defined any more, leading to a null pointer error in rpc_wallet_tests (which uses TestingSetup). I think the best solution is to have TestingSetup inherit from JoinSplitTestingSetup.

Oh, and if we do this, the test that was triggering this bug then fails:

test/rpc_wallet_tests.cpp(1377): error: in "rpc_wallet_tests/rpc_z_shieldcoinbase_internals": check string(e.what()).find("JoinSplit verifying key not loaded")!= string::npos has failed

str4d · 2017-11-05T16:55:17Z

src/zcash/JoinSplit.cpp

@@ -154,10 +110,6 @@ class JoinSplitCircuit : public JoinSplit<NumInputs, NumOutputs> {
        uint64_t vpub_new,
        const uint256& rt
    ) {
-        if (!vk || !vk_precomp) {
-            throw std::runtime_error("JoinSplit verifying key not loaded");


I'm going to just alter the test to check for the error we actually get now, given that this error has been removed.

str4d · 2017-11-05T17:03:27Z

@zkbot try

zkbot · 2017-11-05T17:03:32Z

⌛ Trying commit bef1b5c with merge 9e03db9...

@ebfull

Low memory prover This PR integrates @ebfull 's low memory changes: https://github.com/zcash/zcash/pull/2243/commits on top of @str4d 's work bringing in libsnark as a subtree 4699d0e

zkbot · 2017-11-05T18:00:52Z

☀️ Test successful - pr-try
State: approved= try=True

daira

ACK bef1b5c

daira · 2017-11-06T12:22:05Z

@zkbot r+

zkbot · 2017-11-06T12:22:06Z

📌 Commit bef1b5c has been approved by daira

zkbot · 2017-11-06T12:22:11Z

⌛ Testing commit bef1b5c with merge 6f9f09d...

@ebfull

Low memory prover This PR integrates @ebfull 's low memory changes: https://github.com/zcash/zcash/pull/2243/commits on top of @str4d 's work bringing in libsnark as a subtree 4699d0e

zkbot · 2017-11-06T15:02:38Z

☀️ Test successful - pr-merge
Approved by: daira
Pushing 6f9f09d to master...

karelbilek · 2017-11-07T13:30:53Z

Thanks for merging! I will be able to run actual release soon :)

arielgabizon · 2017-12-01T13:52:02Z

Interestingly, the memory peak now and previously, is not during proof generation but during creating the joinsplit gadget https://github.com/zcash/zcash/blob/master/src/zcash/JoinSplit.cpp#L276.
@madars and other libsnark authors might want to know about this.

str4d added this to the 1.0.13 milestone Oct 19, 2017

arielgabizon force-pushed the lowmemprover branch from 4d2246d to 66d08c3 Compare October 19, 2017 09:37

daira mentioned this pull request Oct 21, 2017

Speed up JoinSplit proving by running is_satisfied() only if verification fails #2005

Closed

Remove crusty old "loadVerifyingKey"/"loadProvingKey" APIs and associ…

1a9543d

…ated invariants.

arielgabizon force-pushed the lowmemprover branch from 66d08c3 to 87703d3 Compare October 21, 2017 12:43

daira reviewed Oct 22, 2017

View reviewed changes

daira previously requested changes Oct 22, 2017

View reviewed changes

This was referenced Oct 22, 2017

Low memory proving integration #2243

Closed

Stream the proving key to reduce memory usage #2235

Closed

Ensure that large prover data structures can be released without process memory retention #2679

Closed

str4d requested changes Oct 23, 2017

View reviewed changes

arielgabizon force-pushed the lowmemprover branch from 87703d3 to 3a0034b Compare October 24, 2017 10:28

ebfull previously requested changes Oct 26, 2017

View reviewed changes

arielgabizon force-pushed the lowmemprover branch from 3a0034b to 8eeda60 Compare October 26, 2017 20:37

str4d force-pushed the lowmemprover branch from 2f598a5 to 4305a56 Compare November 5, 2017 15:04

str4d reviewed Nov 5, 2017

View reviewed changes

Fix NPE in rpc_wallet_tests

bef1b5c

daira approved these changes Nov 6, 2017

View reviewed changes

zkbot merged commit bef1b5c into zcash:master Nov 6, 2017

This was referenced Nov 6, 2017

Experimental feature: remote proving service #2120

Closed

Bitcoin 0.12 P2P/Net PRs 1 #2534

Draft

str4d mentioned this pull request Nov 10, 2017

RPC call for loading/unloading proving #789

Closed

Low memory prover #2670

Low memory prover #2670

Conversation

arielgabizon commented Oct 18, 2017

str4d commented Oct 19, 2017

arielgabizon commented Oct 19, 2017

daira commented Oct 21, 2017

arielgabizon commented Oct 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

str4d Oct 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

str4d left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebfull commented Oct 26, 2017 • edited Loading

bitcartel commented Oct 27, 2017 • edited Loading

arielgabizon commented Oct 27, 2017

arielgabizon commented Oct 27, 2017

arielgabizon commented Oct 27, 2017

tromer commented Oct 27, 2017

arielgabizon commented Oct 27, 2017

arielgabizon commented Oct 27, 2017

tromer commented Oct 27, 2017

arielgabizon commented Oct 27, 2017

bitcartel commented Oct 27, 2017 • edited Loading

arielgabizon commented Oct 27, 2017

zkbot commented Nov 2, 2017

str4d commented Nov 2, 2017

daira commented Nov 2, 2017

bitcartel commented Nov 4, 2017

arielgabizon commented Nov 4, 2017

str4d commented Nov 5, 2017

zkbot commented Nov 5, 2017

str4d commented Nov 5, 2017

zkbot commented Nov 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

str4d commented Nov 5, 2017

zkbot commented Nov 5, 2017

zkbot commented Nov 5, 2017

daira left a comment

Choose a reason for hiding this comment

daira commented Nov 6, 2017

zkbot commented Nov 6, 2017

zkbot commented Nov 6, 2017

zkbot commented Nov 6, 2017

karelbilek commented Nov 7, 2017

arielgabizon commented Dec 1, 2017

arielgabizon commented Oct 21, 2017 •

edited

Loading

str4d Oct 23, 2017 •

edited

Loading

ebfull commented Oct 26, 2017 •

edited

Loading

bitcartel commented Oct 27, 2017 •

edited

Loading

bitcartel commented Oct 27, 2017 •

edited

Loading