Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Boost PreloadAsync() #420

Merged
merged 2 commits into from Aug 27, 2019

Conversation

@longfin
Copy link
Member

commented Aug 9, 2019

This patch enhances the performance of Swarm.PreloadAsync() using two tricks as below.

  • Instead of using RecentStates.StateReferences directly, PreloadAsync became to translate this message as Dictionary<HashDigest<SHA256>, ISet<Address>> to reduce file I/O.
    • To accomplish this, IStore.StoreStateReference became to take blockHash and blockIndex, not whole Block.
  • PreloadAsync() are currently fork its chain to prevent corruption by unexpected termination. this patch reduces its file I/O by IStore.ForkIndexes()
@longfin longfin force-pushed the longfin:feature/boost-ibd branch from 6d0e141 to 9bba9ab Aug 9, 2019
@longfin longfin self-assigned this Aug 12, 2019
@longfin longfin force-pushed the longfin:feature/boost-ibd branch 7 times, most recently from 4049d43 to 00197ca Aug 13, 2019
@longfin longfin force-pushed the longfin:feature/boost-ibd branch 3 times, most recently from 13b17c5 to f08577b Aug 21, 2019
@longfin longfin requested review from dahlia, earlbread, moreal and limebell and removed request for dahlia Aug 21, 2019
@longfin longfin marked this pull request as ready for review Aug 21, 2019
@longfin longfin force-pushed the longfin:feature/boost-ibd branch from f08577b to fc4bb70 Aug 21, 2019
@longfin

This comment has been minimized.

Copy link
Member Author

commented Aug 21, 2019

I've rebased. PTAL.

@codecov

This comment has been minimized.

Copy link

commented Aug 21, 2019

Codecov Report

Merging #420 into master will increase coverage by 0.03%.
The diff coverage is 93.82%.

@@            Coverage Diff            @@
##           master    #420      +/-   ##
=========================================
+ Coverage   87.97%     88%   +0.03%     
=========================================
  Files         195     195              
  Lines       13675   13729      +54     
=========================================
+ Hits        12030   12082      +52     
- Misses       1332    1334       +2     
  Partials      313     313
Impacted Files Coverage Δ
Libplanet/Store/StoreExtension.cs 95.55% <ø> (ø) ⬆️
Libplanet/Net/StateReferenceDownloadState.cs 0% <ø> (ø) ⬆️
Libplanet/Store/BaseStore.cs 81.25% <ø> (ø) ⬆️
Libplanet.Tests/Blockchain/BlockChainTest.cs 98.4% <100%> (ø) ⬆️
Libplanet/Blockchain/BlockChain.cs 94.53% <100%> (-0.01%) ⬇️
Libplanet.Tests/Store/StoreTest.cs 98.24% <100%> (+0.07%) ⬆️
Libplanet/Store/LiteDBStore.cs 89.33% <100%> (+0.16%) ⬆️
Libplanet.Tests/Store/StoreExtensionTest.cs 100% <100%> (ø) ⬆️
Libplanet.Tests/Store/StoreTracker.cs 47.85% <33.33%> (-1.41%) ⬇️
Libplanet/Net/Swarm.cs 80.23% <93.75%> (+0.42%) ⬆️
... and 2 more
@longfin longfin added this to the 0.6.0 milestone Aug 22, 2019
@longfin longfin dismissed stale reviews from moreal and earlbread via c111e56 Aug 23, 2019
@longfin longfin force-pushed the longfin:feature/boost-ibd branch from fc4bb70 to c111e56 Aug 23, 2019
@longfin

This comment has been minimized.

Copy link
Member Author

commented Aug 23, 2019

I've rebased to append it to 0.6.0.

@longfin longfin force-pushed the longfin:feature/boost-ibd branch from c111e56 to b46fb2c Aug 26, 2019
CHANGES.md Outdated Show resolved Hide resolved
CHANGES.md Outdated Show resolved Hide resolved
Libplanet/Store/IStore.cs Show resolved Hide resolved
@@ -53,6 +53,26 @@ public interface IStore

bool DeleteIndex(string @namespace, HashDigest<SHA256> hash);

/// <summary>
/// Forks block indexes from

This comment has been minimized.

Copy link
@dahlia

dahlia Aug 27, 2019

Member

Does the term “block indexes” mean a sequence (chain) of block hashes and its index numbers? Or does it include state references? If it's latter the plural term seems natural for me as well, but if it's former I believe the singular term (i.e., “block index”) would listen more natural IMHO (as like the way other related methods are named, e.g., AppendIndex, DeleteIndex, IterateIndex).

This comment has been minimized.

Copy link
@longfin

longfin Aug 27, 2019

Author Member

It's former. but I've confused about that because….

  • AppendIndex() and DeleteIndex() take only one argument.
  • Other iteration methods use plural form. (e.g. IterateBlockHashes(), IterateStateReferences())

So, how about keep ForkIndexes() in this PR and rename IterateIndex() to IterateIndexes() in other PR?

This comment has been minimized.

Copy link
@dahlia

dahlia Aug 27, 2019

Member

IMO in the case of this a singular noun “index” is more natural than a plural noun “indexes”/“indices,” in the same manner we say “a set of blocks” instead of “sets of blocks” to mean ISet[Block] in verbal language.

This comment has been minimized.

Copy link
@dahlia

dahlia Aug 27, 2019

Member

I thought about this terminology once more, and it turns out the plural term has its own advantage, and I agree your suggestion to rename these method names to use the plural term “indexes” instead of the singular “index.” Some reasons:

  • Although relational databases tend to say “an index” for values in a column, we also say “indices” in books.
  • We already name a block's height by Block.Index, so it's natural to say a collection of Block.Indexes as the plural “indexes.”
@longfin longfin force-pushed the longfin:feature/boost-ibd branch from b46fb2c to eb59e7c Aug 27, 2019
longfin added 2 commits Aug 9, 2019
- Use temporary memory to organize staterefs for avoid too many file writings
- IStore.StoreStateReferences() became to take blockHash and blockIndex instead of the block itself.
- Remove ReceivedAddress from StateReferenceDownloadState
- Add ForkBlockIndexes to IStore and LiteDBStore for fast fork
@longfin longfin force-pushed the longfin:feature/boost-ibd branch from eb59e7c to 2d81e58 Aug 27, 2019
@longfin longfin requested review from dahlia, earlbread and moreal Aug 27, 2019
@moreal
moreal approved these changes Aug 27, 2019
@dahlia
dahlia approved these changes Aug 27, 2019
@longfin longfin merged commit aa5e357 into planetarium:master Aug 27, 2019
17 checks passed
17 checks passed
WIP Ready for review
Details
changelog This contains self-describing changelog.
Details
codecov/patch 93.82% of diff hit (target 87.97%)
Details
codecov/project 88% (+0.03%) compared to a079b95
Details
docs Libplanet docs generated by DocFX
Details
license/cla Contributor License Agreement is signed.
Details
planetarium.libplanet Build #20190827.3 succeeded
Details
planetarium.libplanet (Linux_Mono) Linux_Mono succeeded
Details
planetarium.libplanet (Linux_NETCore) Linux_NETCore succeeded
Details
planetarium.libplanet (Windows_Mono) Windows_Mono succeeded
Details
planetarium.libplanet (Windows_NETCore) Windows_NETCore succeeded
Details
planetarium.libplanet (Windows_NETCore_Benchmark) Windows_NETCore_Benchmark succeeded
Details
planetarium.libplanet (Windows_NETCore_coverage) Windows_NETCore_coverage succeeded
Details
planetarium.libplanet (Windows_NETFramework) Windows_NETFramework succeeded
Details
planetarium.libplanet (macOS_Mono) macOS_Mono succeeded
Details
planetarium.libplanet (macOS_NETCore) macOS_NETCore succeeded
Details
planetarium.libplanet (macOS_Unity) macOS_Unity succeeded
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.