New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Added an option to fast forward the ledger state #1196

Merged
merged 11 commits into from Nov 29, 2018

Conversation

Projects
None yet
4 participants
@hmoog
Copy link
Contributor

hmoog commented Nov 26, 2018

Description

This PR introduces a way to fast forward the ledger state after a restart of IRI. Previously we had to rescan all milestones to rebuild the ledger state. Especially for nodes that start off with a non-snapshotted database, this could take a really long time (a few hours). In combination with the upcoming refactor of the replayMilestones method this allows us to massively decrease the time and resources that are required for this operation (to a few seconds rather than hours).

Type of change

  • Enhancement (a non-breaking change which adds functionality)

Checklist:

  • My code follows the style guidelines for this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@hmoog hmoog requested review from alon-e and GalRogozinski Nov 26, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 27, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 27, 2018

@iotaledger iotaledger deleted a comment from hmoog Nov 27, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 28, 2018

@GalRogozinski
Copy link
Member

GalRogozinski left a comment

Very good!

Liked your mock utils :-)

Only one change request...

Show resolved Hide resolved ...t/java/com/iota/iri/service/milestone/impl/MilestoneServiceImplTest.java Outdated
Show resolved Hide resolved src/main/java/com/iota/iri/service/ledger/impl/LedgerServiceImpl.java Outdated
Show resolved Hide resolved ...com/iota/iri/service/milestone/impl/LatestSolidMilestoneTrackerImpl.java
Show resolved Hide resolved ...com/iota/iri/service/milestone/impl/LatestSolidMilestoneTrackerImpl.java Outdated
Show resolved Hide resolved ...com/iota/iri/service/milestone/impl/LatestSolidMilestoneTrackerImpl.java Outdated

@Before
public void setUp() {
SnapshotMockUtils.mockSnapshotProvider(snapshotProvider);

This comment has been minimized.

@GalRogozinski

GalRogozinski Nov 28, 2018

Member

why not put this in the constructor?

This comment has been minimized.

@hmoog

hmoog Nov 28, 2018

Contributor

Because the mocked properties are NULL in the constructor.

public void setUp() {
SnapshotMockUtils.mockSnapshotProvider(snapshotProvider);

MilestoneViewModel.clear();

This comment has been minimized.

@GalRogozinski

GalRogozinski Nov 28, 2018

Member

shouldn't this be in After?

This comment has been minimized.

@hmoog

hmoog Nov 28, 2018

Contributor

Other tests don't clean up the cached MilestoneViewModels so it can happen that we start this test with some cache Milestones, but you are right this should ALSO go into after the test.

public class SnapshotMockUtils {
private static final int DEFAULT_MILESTONE_START_INDEX = 70000;

private static final Hash DEFAULT_GENESIS_HASH = Hash.NULL_HASH;

This comment has been minimized.

@GalRogozinski

GalRogozinski Nov 28, 2018

Member

why not use Hash.NULL_HASH directly?

This comment has been minimized.

@hmoog

hmoog Nov 28, 2018

Contributor

To make it easier to reason about the logic and allow the "user" of this utils class to easily determine the important values for the mocked objects.

Otherwise anybody who uses this class would have to examine the code of the specific method to know which values are used for the mocked objects.

I would even consider making them public so tests can check against these values without reading them from the mocked objects (that might for example be usefull when testing the rollback functionality - atm I read them from the mocked object and assume that the value was not magically modified during any other call).

But this will be refined ans grow as I add more tests.

Show resolved Hide resolved src/test/java/com/iota/iri/service/snapshot/impl/SnapshotMockUtils.java Outdated
@alon-e

alon-e approved these changes Nov 28, 2018

Copy link
Member

alon-e left a comment

👌

@iotaledger iotaledger deleted a comment from codacy-bot Nov 29, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 29, 2018

@iotaledger iotaledger deleted a comment from hmoog Nov 29, 2018

@iotaledger iotaledger deleted a comment from hmoog Nov 29, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 29, 2018

@iotaledger iotaledger deleted a comment from GalRogozinski Nov 29, 2018

@iotaledger iotaledger deleted a comment from hmoog Nov 29, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 29, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 29, 2018

@iotaledger iotaledger deleted a comment from hmoog Nov 29, 2018

@iotaledger iotaledger deleted a comment from hmoog Nov 29, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 29, 2018

@iotaledger iotaledger deleted a comment from codacy-bot Nov 29, 2018

@iotaledger iotaledger deleted a comment from hmoog Nov 29, 2018

@iotaledger iotaledger deleted a comment from hmoog Nov 29, 2018

@iotaledger iotaledger deleted a comment from hmoog Nov 29, 2018

Optional<MilestoneViewModel> milestone = milestoneService.findLatestProcessedSolidMilestoneInDatabase();
if (milestone.isPresent()) {
snapshotService.replayMilestones(snapshotProvider.getLatestSnapshot(), milestone.get().index());
}

This comment has been minimized.

@GalRogozinski

GalRogozinski Nov 29, 2018

Member

looks nicer :-)

@GalRogozinski GalRogozinski merged commit fec0935 into iotaledger:dev-localsnapshots Nov 29, 2018

5 of 6 checks passed

Codacy/PR Quality Review Hang in there, Codacy is reviewing your Pull request.
Details
buildkite/iri-build-jar-prs Build #252 passed (12 minutes, 12 seconds)
Details
buildkite/iri-build-jar-prs/build-iri-oracle8 Passed (3 minutes, 20 seconds)
Details
buildkite/iri-build-jar-prs/cc997ff8-e995-4ed7-808f-9b1daebfc592 Passed (7 minutes, 5 seconds)
Details
buildkite/iri-build-jar-prs/pull-from-repo Passed (1 minute, 42 seconds)
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment