Validator: Connect to all quorum set validators #643

AndrejMitrovic · 2020-03-10T04:24:50Z

~~Requires #648~~

codecov · 2020-03-10T04:32:06Z

Codecov Report

Merging #643 into v0.x.x will increase coverage by 0.02%.
The diff coverage is 95%.

@@            Coverage Diff             @@
##           v0.x.x     #643      +/-   ##
==========================================
+ Coverage   91.09%   91.12%   +0.02%     
==========================================
  Files          63       63              
  Lines        4786     4821      +35     
==========================================
+ Hits         4360     4393      +33     
- Misses        426      428       +2

Flag	Coverage Δ
#integration	`54.41% <93.75%> (+0.17%)`	⬆️
#unittests	`89.88% <92.5%> (+0.03%)`	⬆️

Impacted Files	Coverage Δ
source/agora/network/NetworkManager.d	`80.12% <100%> (+1.14%)`	⬆️
source/agora/test/Base.d	`89.14% <100%> (+0.25%)`	⬆️
source/agora/test/NetworkDiscovery.d	`88.88% <88.88%> (ø)`	⬆️
source/agora/node/Node.d	`98.94% <88.88%> (-1.06%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 91943ce...4d6e10b. Read the comment docs.

Geod24 · 2020-03-10T04:42:20Z

Isn't it going against some of our goals to have validator special casing in network manager ?

AndrejMitrovic · 2020-03-10T04:43:39Z

Right, so we could move the list of required peers to the Node class (and ValidatorNode class once that's implemented). I'll rework it.

AndrejMitrovic · 2020-03-10T08:02:12Z

Actually, I've realized that the issue itself isn't so clear-cut.

The node doesn't have to be connected to all of its quorum peers in order to reach consensus. For example, let's say we have this node quorum layout:

A => (B, C, D)
D => (B, C)

A has (B, C, D) in its quorum set, while D has (B, C).

If A loses or does not establish its connection to B, it does not mean it won't receive its messages. As long as A is connected to D, then D will propagate B's messages back to A.

So changing minPeersConnected is incorrect here, although it's safer (but works against liveness). Changing peerLimitReached however is correct because we should always strive towards connecting to all of our quorum peers, even if their messages can be forwarded by other nodes in the network.

AndrejMitrovic · 2020-03-12T01:35:40Z

#648 was merged, this is now unblocked.

I've decided to keep the safe behavior by default. The validating node will attempt to connect to all of its quorum peers before discover() finishes. Later we can fine-tune this and make it have more liveness vs safety.

AndrejMitrovic · 2020-03-12T02:03:28Z

Updated with better documentation. @bpfkorea/core ready for review.

TrustHenry · 2020-03-13T00:48:34Z

source/agora/test/Base.d

+
+        auto quorum_keys =
+            node_confs
+                .filter!(conf => conf.is_validator)


If conf.is_validator is false, an exception may occur.

Module tests failed: agora.test.NetworkDiscovery object.Exception@submodules/localrest/source/geod24/LocalRest.d-mixin-716(770): "Request timed-out" ---------------- /usr/local/opt/dmd/include/dlang/dmd/core/internal/array/equality.d:33 @safe void geod24.LocalRest.RemoteAPI!(agora.test.Base.TestAPI).RemoteAPI.start() [0x10e7876da] /usr/local/opt/dmd/include/dlang/dmd/core/internal/array/equality.d:33 @safe void agora.test.Base.TestAPIManager.start().__lambda1!(agora.test.Base.NodePair).__lambda1(agora.test.Base.NodePair) [0x10e7965e0] /usr/local/opt/dmd/include/dlang/dmd/core/internal/array/equality.d:33 @safe std.typecons.Flag!("each").Flag std.algorithm.iteration.each!(agora.test.Base.TestAPIManager.start().__lambda1).each!(agora.test.Base.NodePair[]).each(ref agora.test.Base.NodePair[]) [0x10e796657] /usr/local/opt/dmd/include/dlang/dmd/core/internal/array/equality.d:33 void agora.test.Base.TestAPIManager.start() [0x10e783b7f] /usr/local/opt/dmd/include/dlang/dmd/core/internal/array/equality.d:33 void agora.test.NetworkDiscovery.__unittest_L69_C1() [0x10e7ab21a] source/agora/node/main.d:40 void agora.test.NetworkDiscovery.__modtest() [0x10e4edf4a] /usr/local/opt/dmd/include/dlang/dmd/core/internal/array/equality.d:33 int agora.test.Base.customModuleUnitTester().__foreachbody3(ref agora.test.Base.customModuleUnitTester().ModTest) [0x10e7837df] /usr/local/opt/dmd/include/dlang/dmd/core/internal/array/equality.d:33 void std.parallelism.ParallelForeach!(agora.test.Base.customModuleUnitTester().ModTest[]).ParallelForeach.opApply(scope int delegate(ref agora.test.Base.customModuleUnitTester().ModTest)).doIt() [0x10e566203]

if none of the node configs' is_validator is true, it will fail because map will try to invoke popFront on the empty range. But this makeMinimalNetQuorum should only be called for this specific test-case where all the nodes are validators.

Ah it definitely does not throw. I wonder where I got this error from.

I'll add an assert.

TrustHenry · 2020-03-13T00:53:25Z

source/agora/node/Node.d

+
+        // build the list of required quorum peers to connect to if we're a Validator
+        Set!PublicKey required_peer_keys;
+        if (this.config.node.is_validator)


If this.config.node.is_validator is false, a unit test results in an error.

Module tests failed: agora.test.NetworkDiscovery core.exception.AssertError@source/agora/test/NetworkDiscovery.d(89): Node 0 has 2 peers: ["GAYDGXP436SZ7LV6B5EXRZX27TB2D2GQRFV3SRN775RU6OQPMJT347F5", "GDD62CVWBKJE4NXYMELIV6GHZ5AKJ2TNY6MFASHUQY24RFAKJ4EZHCUF"]

Where are you setting it to false? Can you show me this test-case?

Geod24

This seems to only work for the initial connection, doesn't it ?
What happens if a required peer drops off, then a non-required peer connects, taking the "slot", and when the required peer comes back online, it wouldn't be accepted, would it ?

source/agora/test/Base.d

@@ -744,6 +749,31 @@ public APIManager makeTestNetwork (APIManager : TestAPIManager = TestAPIManager)
        return conf;
    }

+    // for disovery testing: full quorum, but only 1 node in the 'network' section


source/agora/test/Base.d

+
+        auto quorum_keys =
+            node_confs
+                .filter!(conf => conf.is_validator)


AndrejMitrovic · 2020-03-18T04:51:43Z

What happens if a required peer drops off

True. The current design of NetworkManager does not take into account dropped connections. It's not really obvious how to deal with this while using the RestInterfaceClient. Maybe after #203 is implemented it will be much clearer how to handle connections?

Geod24 · 2020-03-18T04:53:39Z

Maybe after #203 is implemented it will be much clearer how to handle connections?

Then this is your lucky day.

AndrejMitrovic · 2020-03-18T05:01:44Z

I think reworking the NetworkManager class could be a good issue to tackle in one of the next sprints. #587 and #644

AndrejMitrovic · 2020-03-19T04:03:32Z

Then this is your lucky day.

Maybe today..?

AndrejMitrovic · 2020-03-19T04:48:39Z

This seems to only work for the initial connection, doesn't it ?
What happens if a required peer drops off, then a non-required peer connects, taking the "slot", and when the required peer comes back online, it wouldn't be accepted, would it ?

I'll try to redesign the networking class to make it more flexible. Can we make this part of future work in #587?

A validator currently does not attempt to connect to all of its quorum peers before it's ready to start nominating.

Geod24 · 2020-04-01T09:01:35Z

source/agora/test/Base.d

@@ -767,11 +771,13 @@ public APIManager makeTestNetwork (APIManager : TestAPIManager = TestAPIManager)
        return conf;
    }

-    // for disovery testing: full quorum, but only 1 node in the 'network' section
+    // for dissovery testing: full quorum, but only 1 node in the 'network' section


LOL. Also wrong commit.

Previously the network discovery would be considered complete even if the quorum which a node depends on has not been connected to.

AndrejMitrovic added the type-enhancement An improvement of existing functionalities label Mar 10, 2020

AndrejMitrovic mentioned this pull request Mar 11, 2020

Network discovery is broken (due to lack of a proper test) #647

Closed

AndrejMitrovic force-pushed the wait-validators branch from 2710f44 to 5a1a3a4 Compare March 11, 2020 06:36

AndrejMitrovic added status-blocked Another issue/PR/external dependency is blocking completion of this PR/issue and removed status-blocked Another issue/PR/external dependency is blocking completion of this PR/issue labels Mar 11, 2020

AndrejMitrovic force-pushed the wait-validators branch from 5a1a3a4 to 54a5312 Compare March 12, 2020 01:34

AndrejMitrovic mentioned this pull request Mar 12, 2020

Send & receive SCP quorum sets to fix SCP message propagation #621

Closed

AndrejMitrovic force-pushed the wait-validators branch 2 times, most recently from ae2421a to 2c87ef9 Compare March 12, 2020 02:03

TrustHenry reviewed Mar 13, 2020

View reviewed changes

Geod24 suggested changes Mar 18, 2020

View reviewed changes

AndrejMitrovic force-pushed the wait-validators branch from 2c87ef9 to ac12f45 Compare March 18, 2020 04:59

AndrejMitrovic mentioned this pull request Mar 18, 2020

Network manager needs to be reworked #587

Closed

AndrejMitrovic force-pushed the wait-validators branch from ac12f45 to 67442a9 Compare April 1, 2020 08:17

Add test-case for current quorum networking behavior

5c8b4e9

A validator currently does not attempt to connect to all of its quorum peers before it's ready to start nominating.

AndrejMitrovic force-pushed the wait-validators branch from 67442a9 to 6d44321 Compare April 1, 2020 08:24

AndrejMitrovic added a commit to AndrejMitrovic/agora that referenced this pull request Apr 1, 2020

[REMOVE] Merge bosagora#643

d39ad92

Geod24 reviewed Apr 1, 2020

View reviewed changes

Validator: Connect to all quorum set validators

4d6e10b

Previously the network discovery would be considered complete even if the quorum which a node depends on has not been connected to.

AndrejMitrovic force-pushed the wait-validators branch from 6d44321 to 4d6e10b Compare April 1, 2020 09:04

AndrejMitrovic mentioned this pull request Apr 1, 2020

Consensus failure test-case #716

Closed

Geod24 approved these changes Apr 5, 2020

View reviewed changes

Geod24 merged commit 0c45867 into bosagora:v0.x.x Apr 5, 2020

AndrejMitrovic mentioned this pull request Apr 6, 2020

Design and implement quorum balancing #240

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validator: Connect to all quorum set validators #643

Validator: Connect to all quorum set validators #643

AndrejMitrovic commented Mar 10, 2020 •

edited

Loading

codecov bot commented Mar 10, 2020 •

edited

Loading

Geod24 commented Mar 10, 2020

AndrejMitrovic commented Mar 10, 2020 •

edited

Loading

AndrejMitrovic commented Mar 10, 2020 •

edited

Loading

AndrejMitrovic commented Mar 12, 2020

AndrejMitrovic commented Mar 12, 2020

TrustHenry Mar 13, 2020 •

edited

Loading

AndrejMitrovic Mar 13, 2020 •

edited

Loading

This comment was marked as resolved.

AndrejMitrovic Mar 18, 2020

TrustHenry Mar 13, 2020 •

edited

Loading

AndrejMitrovic Mar 16, 2020

Geod24 left a comment

This comment was marked as resolved.

This comment was marked as resolved.

AndrejMitrovic commented Mar 18, 2020

Geod24 commented Mar 18, 2020

AndrejMitrovic commented Mar 18, 2020

AndrejMitrovic commented Mar 19, 2020

AndrejMitrovic commented Mar 19, 2020

Geod24 Apr 1, 2020

Validator: Connect to all quorum set validators #643

Validator: Connect to all quorum set validators #643

Conversation

AndrejMitrovic commented Mar 10, 2020 • edited Loading

codecov bot commented Mar 10, 2020 • edited Loading

Codecov Report

Geod24 commented Mar 10, 2020

AndrejMitrovic commented Mar 10, 2020 • edited Loading

AndrejMitrovic commented Mar 10, 2020 • edited Loading

AndrejMitrovic commented Mar 12, 2020

AndrejMitrovic commented Mar 12, 2020

TrustHenry Mar 13, 2020 • edited Loading

Choose a reason for hiding this comment

AndrejMitrovic Mar 13, 2020 • edited Loading

Choose a reason for hiding this comment

This comment was marked as resolved.

AndrejMitrovic Mar 18, 2020

Choose a reason for hiding this comment

TrustHenry Mar 13, 2020 • edited Loading

Choose a reason for hiding this comment

AndrejMitrovic Mar 16, 2020

Choose a reason for hiding this comment

Geod24 left a comment

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

AndrejMitrovic commented Mar 18, 2020

Geod24 commented Mar 18, 2020

AndrejMitrovic commented Mar 18, 2020

AndrejMitrovic commented Mar 19, 2020

AndrejMitrovic commented Mar 19, 2020

Geod24 Apr 1, 2020

Choose a reason for hiding this comment

AndrejMitrovic commented Mar 10, 2020 •

edited

Loading

codecov bot commented Mar 10, 2020 •

edited

Loading

AndrejMitrovic commented Mar 10, 2020 •

edited

Loading

AndrejMitrovic commented Mar 10, 2020 •

edited

Loading

TrustHenry Mar 13, 2020 •

edited

Loading

AndrejMitrovic Mar 13, 2020 •

edited

Loading

TrustHenry Mar 13, 2020 •

edited

Loading