Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Update Milestone Solidification #1674

Open
DyrellC opened this issue Nov 28, 2019 · 3 comments
Open

Feature: Update Milestone Solidification #1674

DyrellC opened this issue Nov 28, 2019 · 3 comments

Comments

@DyrellC
Copy link
Contributor

@DyrellC DyrellC commented Nov 28, 2019

Description

The current milestone solidification logic relies on a 3 step process that is rather disjointed. Milestones are found in the LatestMilestoneTracker (LMT) through an address search and then processed through milestone/bundle validity checks before being placed into the MilestoneSolidifier (MS) where it will then go through a separate process of transaction requesting and solidity checking to fetch missing reference transactions for milestone candidates. When a milestone is solidified through this process a milestone object is created in the DB. Lastly the LatestSolidMilestoneTracker (LSMT) searches sequentially for the next milestone object by index and updates the nodes solidity index by index. This strikes me as an unnecessarily disconnected process. With the new TransactionProcessingPipeline(TPP) following the last network rewrite, we can utilise the processing stages to more directly handle milestones and milestone reference requests to streamline milestone solidification and process them in a more connected order that carries the hash references straight through to the LSMT checks. Currently the LMS checks rely on transaction addresses and hash references to pass to the MS, whereas the LSMT is reliant on the milestone index, which is only created when the MS successfully solidifies the transaction. The MS does not rely on index while solidifying and may produce the occasional false INVALID or INCOMPLETE which then stops a specific Milestone from solidifying correctly.

Motivation

To process milestone objects more efficiently and improve solidification and request logic.

Requirements

  • New transaction requesting logic using the TPP
  • New milestone stages and payload processing through the TPP
  • Rewrite/Consolidate milestone solidification pipeline with better milestone hash/index mapping

Open Questions (optional)

Am I planning to do it myself with a PR?

Yes

@GalRogozinski

This comment has been minimized.

Copy link
Member

@GalRogozinski GalRogozinski commented Dec 2, 2019

Relates to #1448

@DyrellC

This comment has been minimized.

Copy link
Contributor Author

@DyrellC DyrellC commented Dec 10, 2019

I would like to outline my proposal for changes to milestone processing. With the TPP in place and the new solidification process coming into play, we now have a good opportunity to consolidate behaviour into a less disjointed flow. The solidification of milestones can be handled from within the TPP rather than as an external set of threads which each run independently of one another.
As is currently, the processing flow looks something like this:
image

With the addition of the solidification stage it's starting to become more streamlined as follows:
image

But in both of these cases, milestones are still handled as a separate entity and process, and within this disconnected process, we recreate numerous objects over and over to analyse and determine the milestone state of the tangle/node. The new flow I would like to propose is as follows:
image

In this flow, transactions that enter the received stage will have their address checked to see if it matches that of the coordinator. If it does it will then check if the potential milestone has already been seen, and if not, it will run a check to pull the index of the milestone. If the milestone index is within the range of what the node is looking for (higher than the latest snapshot index) then the transaction will be placed into the MilestoneStage where it will undergo milestone validation. If the milestone is registered as valid, it will then be placed into the new MilestoneSolidifier (might want to rename it) which will act similar to the TransactionRequester as a temporary holding space for milestone mappings. In the same stage, a check for the latest solid milestone will be conducted, and the oldest milestone in the mapping will be pulled from the MilestoneSolidifier and passed forward to the SolidifyStage. In this way milestones will be processed within the same pipeline as all other transactions, and we reduce the 3 thread process to a 1 stage process handling all three tasks.

Beyond the milestone side I would also like to investigate the potential of migrating the transaction requesting process away from the BroadcastStage and into its own stage so that we aren't dependent on broadcasting for transaction requesting, which in my opinion is a little hacky in its own way.

@GalRogozinski

This comment has been minimized.

Copy link
Member

@GalRogozinski GalRogozinski commented Dec 12, 2019

How do you handle out of order milestones?
What do you do if a milestone was received stored and then the node got shut off? So the milestone is in the DB but never went through milestone processing?

Beyond the milestone side I would also like to investigate the potential of migrating the transaction requesting process away from the BroadcastStage and into its own stage so that we aren't dependent on broadcasting for transaction requesting, which in my opinion is a little hacky in its own way.

This can be resolved by implementing the STING protocol that the Hornet people use. Since we need to make the nodes compatible, it should be the solution...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.