Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use `MmapStore` for `MerkleTree` generation #529

Merged
merged 1 commit into from Mar 22, 2019

Conversation

@schomatis
Copy link
Contributor

commented Mar 7, 2019

Blocked on filecoin-project/merkle_light#13, after that is merged the dependency here should point to merkle_light/master.


This is now dropping the allocated memory an order of magnitude:

# master:

Total: 309.4 MB
   309.0  99.9%  99.9%    309.0  99.9% ::allocate_in (inline) /rustc/14997d56a550f4aa99fe737593cd2758227afc56/src/liballoc/raw_vec.rs:109
     0.3   0.1% 100.0%      0.3   0.1% ::reserve_internal (inline) /rustc/14997d56a550f4aa99fe737593cd2758227afc56/src/liballoc/raw_vec.rs:683
     0.1   0.0% 100.0%      0.1   0.0% sapling_crypto::jubjub::JubjubBls12::new (inline) ??:0

# This PR with mmap'ed MT:

Total: 19.4 MB
    13.0  67.1%  67.1%     13.0  67.1% ::allocate_in (inline) /rustc/14997d56a550f4aa99fe737593cd2758227afc56/src/liballoc/raw_vec.rs:109
     4.3  22.0%  89.1%      4.3  22.0% ::reserve_internal (inline) /rustc/14997d56a550f4aa99fe737593cd2758227afc56/src/liballoc/raw_vec.rs:683
     2.0  10.3%  99.4%      2.0  10.3% rayon::iter::collect::special_extend@1594e0 (inline) :0

@schomatis schomatis self-assigned this Mar 7, 2019

@schomatis

This comment was marked as resolved.

Copy link
Contributor Author

commented Mar 7, 2019

Labeling as blocked until #528 lands (and we now the new version of MerkleTree is working as expected).

@schomatis

This comment was marked as resolved.

Copy link
Contributor Author

commented Mar 11, 2019

Currently failing due to a bug I introduced in filecoin-project/merkle_light@ff18b53 where I'm calling new_from_slice with the wrong length (of bytes instead of Elements), although fixing that still fails now with invalid replica node so I'll have to keep digging.

@schomatis

This comment was marked as resolved.

Copy link
Contributor Author

commented Mar 12, 2019

MT upstream branch fixed, triggering another build.

@schomatis schomatis closed this Mar 12, 2019

@schomatis schomatis reopened this Mar 12, 2019

@schomatis schomatis force-pushed the feat/dgraph/use-mmap-merkle branch from 96ccc64 to bcb9634 Mar 14, 2019

@schomatis schomatis changed the title [WIP] `dgraph::Graph`: use `MmapStore` for `MerkleTree` generation use `MmapStore` for `MerkleTree` generation Mar 14, 2019

@schomatis

This comment was marked as resolved.

Copy link
Contributor Author

commented Mar 14, 2019

Currently working but has now considerable differences with the current master, need to do a careful rebase.

@schomatis schomatis force-pushed the feat/dgraph/use-mmap-merkle branch 2 times, most recently from f19a16f to 26833f0 Mar 15, 2019

@schomatis schomatis requested a review from dignifiedquire Mar 15, 2019

@schomatis schomatis referenced this pull request Mar 19, 2019

@schomatis schomatis force-pushed the feat/dgraph/use-mmap-merkle branch 2 times, most recently from da14888 to 21124c9 Mar 19, 2019

@nicola nicola referenced this pull request Mar 21, 2019
@porcuquine
Copy link
Collaborator

left a comment

This seems fine when the rest is sorted out (might be already).

@schomatis schomatis force-pushed the feat/dgraph/use-mmap-merkle branch from 21124c9 to 5d3ec83 Mar 22, 2019

@schomatis schomatis merged commit e4fdb86 into master Mar 22, 2019

11 checks passed

ci/circleci: bench_nightly Your tests passed on CircleCI!
Details
ci/circleci: build_darwin_release Your tests passed on CircleCI!
Details
ci/circleci: build_linux_release Your tests passed on CircleCI!
Details
ci/circleci: cargo_fetch Your tests passed on CircleCI!
Details
ci/circleci: clippy Your tests passed on CircleCI!
Details
ci/circleci: ffi_regression Your tests passed on CircleCI!
Details
ci/circleci: rustfmt Your tests passed on CircleCI!
Details
ci/circleci: test Your tests passed on CircleCI!
Details
ci/circleci: test_ignored_release Your tests passed on CircleCI!
Details
ci/circleci: test_nightly Your tests passed on CircleCI!
Details
ci/circleci: test_release Your tests passed on CircleCI!
Details

@schomatis schomatis deleted the feat/dgraph/use-mmap-merkle branch Mar 22, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.