New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimized Delegated Byzantine Fault Tolerance (ODBFT) - Part I: commit phase + regeneration strategy + minnor message p2p route optimizing #426

Open
wants to merge 141 commits into
base: master
from

Conversation

7 participants
@vncoelho
Member

vncoelho commented Oct 24, 2018

Last update: 21th November

Optimized Delegated Byzantine Fault Tolerance (ODBFT)

  • Part I: commit phase + regeneration strategy + minnor message p2p route optimizing
    • Finish the original commit phase with partial signatures;
    • Propose a new regeneration strategy able to recover lost/failled nodes;
    • Use the maximum amount of benefits from the current Payloads, extracting all information (communication is the most expensive part of the descentralized system). Thus, let's process the maximum we can locally and extract good conclusions.

Part II, III and IV will be dealt in different Pull Requests.

  • Part II: Novel network routing strategies, a trade-off between payloads and communication (Q4 2018 - Q1 2019): https://github.com/igormcoelho/neo/tree/optimized-dbft-part-ii
    • Implement novel efficient strategies for optimizing message routing.
  • Part III: Efficient prediction and optimization strategies for fine-tune the dBFT (Q2/Q3 2019)
    • Fine-tune times and predict the cost of the next blocks in order to ensure accomplishment of desired block times.
  • Part IV: Metaheuristics an efficient selection of pending transactions (Q3 2019 - Q1/Q2 2020)
    • Investigate the use of a Greedy Randomized Adaptive Search Procedure for FillingContext

Part I: Highlights of the proposal (which is a consolidation of previous knowledge discussed in other issues and PRs)

  • Complete signatures are only send after commit phase:
    • At least, M partials signatures should be obtained before sending a complete signature of the context.makeHeader;
    • Partials signatures are produced by signing the PrepareRequest Payload. One peculiarity is that the Speaker/Primary signs the PrepareRequest Payload_1 with an empty vector of his signature (as one could expected), while the Backups signs the complete PrepareRequest Payload_Complete that contains Speaker signature of the aforementioned PrepareRequest Payload_1;
    • It should be noticed that, in essence, the Speaker/Primary partial signature could even be removed, but we are making use of this in order to speed up the P2P share of the PrepareRequest Payload since every PrepareResponse Payload is now containing the PrepareRequest Payload. Thus, any CN that receives the PrepareResponse Payload (even without receiving the PrepareRequest) can validate both of them together if his local PrepareRequest Payload is still null (for this purpose, the partial signature is important for ensuring that a third-entity does not modify primary payload content);
  • When a node Complete Signature is sent (flag CommitAgreement is activated) we can lock Change View. The reasoning is that M nodes already showed their agreement with the proposed Block.
    • When M partial signatures are achieved, a given node signed the complete header and relay to the network.
    • This node can not anymore increase its view level (Change View);
    • An event called OnRegeneration is called every time that a node asks for ChangeView , it contains a Payload with all Partial Signatures (SignedPayloads) plus the original PrepareRequestPayload. It calls a MakeRegenaration event that will be spread throughout the network. That node (that asked for changeview) will be able to verify the OnRegeneration and even reduce its View number if it sees that Commit phase was already been achieved in a lower view.

Part I main contributors: @erikzhang, @shargon, @igormcoelho, @vncoelho, @belane, @wanglongfei88, @edwardz246003

Analogy of the proposal

An analogy of the proposal is the indefatigable miners problem (defined here):

  1. The speaker is a Geological Engineering (my hometown is pioneer in it and one of the cities of more diversity of minerals in the world) and he is searching for a place to dig for Kryptonite;
  2. He proposes a geographic location (coordinates to dig);
  3. The majority (M guys) agrees with the coordinates (with their partial signatures);
  4. Time for digging: they will now dig until they really find Kryptonite (no other place will be accepted to be dig until Kryptonite is found). Kryptonite is an infinite divisible crystal, thus, as soon as one finds he will share the kryptonite so that everyone will have a piece for finishing their contract 3.;
  5. If one of them dies, when it resurrects it will see its previous signed agreement (3.) and it will automatically start to dig again. The other minority will suffer the same, they will be fulfilled with hidden messages saying that they should also dig.

@vncoelho vncoelho referenced this pull request Oct 24, 2018

Merged

fix consensus #425

@vncoelho

This comment has been minimized.

Member

vncoelho commented Dec 1, 2018

@erikzhang, since here we have no problems with forks we removed SignatureSent flag (last commit) because the idea of storing Signatures and replicating the FillContext from previous Views was complicated to handle, there were 2-3 detected problems.

In fact, we could handle it, but it sounded simpler to just remove this minor speed up after change view. Furthermore, we believe that in the future we gonna have few/none changeviews.

Thanks to @wanglongfei88 for the suggestions and support.

@vncoelho

This comment has been minimized.

Member

vncoelho commented Dec 1, 2018

@canesin, @PeterLinX and @wanglongfei88 have been testing this PR, I believe that other members of NGD are also verifying it.

Without putting pressure, but I think that it should be merged for Testnet as soon as possible.
It contains critical and interesting modifications, with this we can move to another level of consensus.

In addition, it will open possibilities for several other improvements (I am not saying it has problems...aehauehauea).

vncoelho added some commits Dec 3, 2018

@vncoelho

This comment has been minimized.

Member

vncoelho commented Dec 4, 2018

@igormcoelho, we reached another level now with #475.
Congratulations for everyone involved.

We may need now to double check this last merge that was done and fix the Unit Tests for this PR.

@vncoelho vncoelho force-pushed the igormcoelho:new_commit_phase branch from 11489a5 to 37e7f1b Dec 4, 2018

@vncoelho vncoelho force-pushed the igormcoelho:new_commit_phase branch from 37e7f1b to 238b51a Dec 4, 2018

vncoelho added some commits Dec 4, 2018

@vncoelho

This comment has been minimized.

Member

vncoelho commented Dec 5, 2018

Preview-Release V2.1.1 with neo-cli 2.9.3
https://github.com/NeoResearch/neo-tests/raw/master/releases/neo-release-2.9.3-all-plugins-odbft-v2.1.1.zip

Congratulations everyone for the recent contributions, in particular:
@jsolman and @erikzhang for the great fix on P2P stability on the Neo Core library! Neo-cli 2.9.3 is much more stable in terms of syncing blocks when delayed.
@igormcoelho, for the great insights with the Units Tests! (we still need to perform some adjust here).

vncoelho added some commits Dec 5, 2018

@shargon

This comment has been minimized.

Member

shargon commented Dec 7, 2018

@erikzhang could you review the code, there is something wrong?

@erikzhang

This comment has been minimized.

Member

erikzhang commented Dec 8, 2018

NGD is testing it.

shargon and others added some commits Dec 8, 2018

@shargon

shargon approved these changes Dec 11, 2018 edited

First of all, happy birthday @vncoelho :)

I have a present for your birthday: my approval to this amazing work, in my tests, works fine (again).

My proposal:

  • Create a pre-release and use this version on testnet for almost 1 month.
  • Work in P2P Plugin for allow to drop message, this should be great to replicate fork issues, and try to see the powerful of this phase.
  • Give great applause to @igormcoelho and @vncoelho for take my PR, and doing it better 🥇

image

@igormcoelho

This comment has been minimized.

Contributor

igormcoelho commented Dec 11, 2018

Congratulations brother @vncoelho and to you @shargon, for taking the (giant) first step and NGD that is being fundamental to make this happen. I feel that Neo will shine much more and evolve much faster in the next few months.

@erikzhang erikzhang self-requested a review Dec 12, 2018

@vncoelho

This comment has been minimized.

Member

vncoelho commented Dec 12, 2018

Thanks brother @igormcoelho, without you everything would be nearly impossible to me.
Furthermore, thanks @shargon for this nice birthday message and recognition.
It has been a great pleasure to live these last months learning with all of you and, in particular, @erikzhang.

"discovering truth by building on previous discoveries" - Standing on the shoulders of giants

I think that the important think of all this is the partnership in building something that is each time more concrete.
In this sense, try to always consider a trade-off between simplicity, reliability and efficiency.
That is why I believe in keep improving the template of the pBFT/dBFT designed by Erik and all NEO team.
Our efforts here are surely already a standard for multi-agent secure, simple and efficient communication.
Improvements, fails and new ideas will always appear, that is why we all like challenges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment