New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perf: Massively improved the performance of the replayMilestones method #1197

Merged
merged 22 commits into from Nov 29, 2018

Conversation

Projects
None yet
4 participants
@hmoog
Copy link
Contributor

hmoog commented Nov 26, 2018

Description

This PR refactors the replayMilestones method to first accumulate all changes and then apply them in a single call. This allows us to modify the milestone without having to create a copy of the state first. Since the Snapshot and its corresponding SnapshotState are huge datastructures, this change reduces the time required for a rebuild of the ledger state after a IRI restart from a few hours to a few seconds (for a non-snapshotted database).

Type of change

  • Performance refactor (a non-breaking change)

Checklist:

  • My code follows the style guidelines for this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@hmoog hmoog requested review from alon-e and GalRogozinski Nov 26, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@GalRogozinski
Copy link
Member

GalRogozinski left a comment

Awesome!

Show resolved Hide resolved src/main/java/com/iota/iri/service/snapshot/impl/SnapshotImpl.java Outdated
Objects.equals(metaData, ((SnapshotImpl) obj).metaData) &&
Objects.equals(skippedMilestones, ((SnapshotImpl) obj).skippedMilestones);
}

This comment has been minimized.

@GalRogozinski

GalRogozinski Nov 28, 2018

Member

It will also be nice to have a toString() here.
The toString() may only print SnapshotMetadata` toString() to not make it a reasonable size (index, hash, hasSolidEntryPoints). You can also decide to have it print the state, but it will be long.

Whatever you think will help with debugging

This comment has been minimized.

@hmoog

hmoog Nov 29, 2018

Contributor

We can add that in another PR - maybe create an issue to check if all models have these toString methods.

Show resolved Hide resolved src/main/java/com/iota/iri/service/snapshot/impl/SnapshotServiceImpl.java
Show resolved Hide resolved src/main/java/com/iota/iri/service/snapshot/impl/SnapshotServiceImpl.java
Show resolved Hide resolved src/main/java/com/iota/iri/service/snapshot/impl/SnapshotStateImpl.java
@alon-e

alon-e approved these changes Nov 28, 2018

Copy link
Member

alon-e left a comment

great PR and clever idea!

Show resolved Hide resolved src/main/java/com/iota/iri/service/snapshot/impl/SnapshotServiceImpl.java Outdated
@@ -112,6 +113,10 @@ public void applyStateDiff(SnapshotStateDiff diff) throws SnapshotException {
if (balances.computeIfPresent(addressHash, (hash, aLong) -> balance + aLong) == null) {
balances.putIfAbsent(addressHash, balance);
}

if (balances.get(addressHash) == 0) {

This comment has been minimized.

@alon-e

hmoog added some commits Nov 28, 2018

@GalRogozinski GalRogozinski merged commit 6ee68bc into iotaledger:dev-localsnapshots Nov 29, 2018

5 checks passed

buildkite/iri-build-jar-prs Build #258 passed (30 minutes, 57 seconds)
Details
buildkite/iri-build-jar-prs/build-iri-oracle8 Passed (3 minutes, 12 seconds)
Details
buildkite/iri-build-jar-prs/f10c4cf1-caf9-4e15-87ca-9a864ebc038d Passed (25 minutes, 52 seconds)
Details
buildkite/iri-build-jar-prs/pull-from-repo Passed (1 minute, 50 seconds)
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment