Add the quorum intersection checker code from stellar-core #664

AndrejMitrovic · 2020-03-16T09:08:52Z

It also includes tests.

Requires:

codecov · 2020-03-16T09:18:33Z

Codecov Report

Merging #664 into v0.x.x will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           v0.x.x     #664      +/-   ##
==========================================
- Coverage   91.34%   91.32%   -0.02%     
==========================================
  Files          65       65              
  Lines        5162     5167       +5     
==========================================
+ Hits         4715     4719       +4     
- Misses        447      448       +1

Flag	Coverage Δ
#integration	`51.20% <ø> (+0.94%)`	⬆️
#unittests	`90.17% <ø> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
source/agora/test/Base.d	`88.31% <0.00%> (-0.47%)`	⬇️
source/agora/consensus/EnrollmentManager.d	`94.67% <0.00%> (-0.05%)`	⬇️
source/agora/node/Ledger.d	`96.58% <0.00%> (+<0.01%)`	⬆️
source/agora/consensus/ValidatorSet.d	`87.34% <0.00%> (+0.32%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2b787c8...5bf37db. Read the comment docs.

AndrejMitrovic · 2020-03-25T01:31:44Z

The part that assigns unassigned nodes needs to be removed. Not all nodes need to be in other nodes' quorums.

Geod24 · 2020-04-05T18:51:35Z

I rebased it on v0.x.x, let's see if it passes on Windows

Geod24 · 2020-04-05T23:55:48Z

Looks nice (and green). Will give it a review today.

AndrejMitrovic

Completed self-review.

AndrejMitrovic · 2020-04-06T08:12:38Z

source/scpd/quorum/QuorumTracker.d

+    ref const(QuorumMap) getQuorum() const;
+}
+
+private void quorum_tracker_rebuild (void* quorum_tracker, void* cb, void* dg);


Use ref and the proper type instead of void*

source/scpd/quorum/QuorumTracker.d

+        quorum_tracker_rebuild(cast(void*)this, &call, cast(void*)&lookup);
+    }
+
+    // returns the current known quorum


AndrejMitrovic · 2020-04-06T08:13:42Z

source/scpd/quorum/QuorumTracker.d

+    bool expand(const ref NodeID id, SCPQuorumSetPtr qSet);
+
+    // replaces 'rebuild' taking an std::function so we can use it with D delegates
+    extern (D) public final void rebuild (SCPQuorumSetPtr delegate(const ref NodeID) lookup)


Suggested change

extern (D) public final void rebuild (SCPQuorumSetPtr delegate(const ref NodeID) lookup)

extern (D) public final void rebuild (SCPQuorumSetPtr delegate(const ref NodeID) lookup) @trusted

And other attributes.

source/scpd/quorum/QuorumTracker.d

+
+extern (C++, `stellar`):
+
+// helper class to help track the overall quorum over time


source/scpp/src/quorum/QuorumTracker.cpp

+// workaround to be able to pass D delegates to C++
+void quorum_tracker_rebuild (void* qt, void* cb, void* dg)
+{
+    typedef SCPQuorumSetPtr (*Callback)(void*, NodeID const&);


source/scpd/quorum/QuorumIntersectionChecker.d

+    extern(D) static shared_ptr!QuorumIntersectionChecker create (
+        QuorumTracker.QuorumMap map) nothrow pure @trusted @nogc
+    {
+        return create(*cast(void**)&map);


source/scpd/quorum/QuorumIntersectionChecker.d

+    Bindings for quorum/QuorumIntersectionChecker.h
+
+    Copyright:
+        Copyright (c) 2019 BOS Platform Foundation Korea


source/scpd/quorum/QuorumIntersectionChecker.d

+extern(C++, class) public abstract class QuorumIntersectionChecker
+{
+  public:
+    extern(D) static shared_ptr!QuorumIntersectionChecker create (


AndrejMitrovic · 2020-04-07T12:23:04Z

source/scpd/quorum/QuorumTracker.d

+            return (*dg)(node_id);
+        }
+
+        quorum_tracker_rebuild(cast(void*)this, &call, cast(void*)&lookup);


Replace pointers with references.

Geod24 · 2020-04-15T13:33:10Z

source/scpd/quorum/QuorumTracker.d

+    //     `id` was known and didn't have a quorumset
+    // returns false on failure
+    // if expand fails, the caller should instead use `rebuild`
+    bool expand(const ref NodeID id, SCPQuorumSetPtr qSet);


This was not addressed

Suggested change

bool expand(const ref NodeID id, SCPQuorumSetPtr qSet);

bool expand (const ref NodeID id, SCPQuorumSetPtr qSet);

source/scpp/README.md

@@ -15,6 +15,8 @@ Some files have been modified after porting:

 - Some `private` declarations were changed to `public`. This allows us to test size & layout checks for our D glue layer, for example: https://github.com/bpfkorea/agora/blob/76e4ca7d77b123ef867d3ce99c1c1b26afab761d/source/scpp/src/crypto/ByteSlice.h#L21 used in https://github.com/bpfkorea/agora/blob/76e4ca7d77b123ef867d3ce99c1c1b26afab761d/source/scpp/extra/DSizeChecks.cpp and https://github.com/bpfkorea/agora/blob/76e4ca7d77b123ef867d3ce99c1c1b26afab761d/source/scpp/extra/DLayoutChecks.cpp.

+- The `src/quorum` folder adapts the Quorum intersection tool which is used in Stellar's Herder implementation (not SCP). The `update.d` script does not track these files yet, so they have to be kept manually in sync when updating the bindings.


AndrejMitrovic · 2020-04-07T12:27:27Z

I suggest adding a in-between commit which just checks in all the intersection checker code as-is, but without bindings. And change the commit author to be the Stellar devs. Otherwise it appears like the code was forged or stolen, and we don't want to leave that kind of impression on the community.

source/scpd/quorum/QuorumIntersectionTests.d

+const shared_ptr!SCPQuorumSet nullquorum;
+
+// we replace the use of std::pair with a fixed-length array
+size_t first (size_t[2] pair) { return pair[0]; }


source/scpd/quorum/QuorumIntersectionTests.d

+        (((n * pct) - size_t(1)) / size_t(100)));
+}
+
+NodeID toStellarKey (PublicKey address)


source/scpd/quorum/QuorumIntersectionTests.d

+    auto qic = QuorumIntersectionChecker.create(qm);
+    assert(qic.networkEnjoysQuorumIntersection());
+
+    //CircularAppender().printConsole();


source/scpd/quorum/QuorumIntersectionTests.d

+}
+
+//TEST_CASE("quorum intersection 6-node with subquorums",
+//          "[herder][quorumintersection]")


AndrejMitrovic · 2020-04-13T06:24:17Z

Completed fixes after self-review.

AndrejMitrovic · 2020-04-13T06:26:03Z

With regards to Make Quorum checker buildable, note that doing this upstream isn't easy because the bindings we're targeting here is for v11.2.0, whereas the one in stellar-core master branch contains even more specialized code which we can't easily bind to. For example, they use interrupts to stop the quorum checker (because sometimes it takes too long to finish).

source/scpp/README.md

Geod24

Woah... Sorry it took me so long to review this. So I left a few comments. The most important ones are w.r.t. class vs struct, and the need for size checks which uncovered #753 .

A few more things:

I didn't port the SCP tests when I imported the SCP code. The reasoning was that little change would be required to SCP (if any), and none of that will be of substance. Any substantial change should be submitted upstream, where its sanity could be checked. So basically piggybacking on upstream having proper tests. I was a bit surprised when I saw the porting of the tests, and would have not originally asked for it, unless you had major plans to change the code being imported (which I guess you do). But the reasoning I just mentioned was also intended to save us time, and if the work is done, I see no reason not to pull it.
The last two commit should be merged. I appreciate the effort to make the diff readable (and it was much more pleasant that it could have been), but since dub compiles everything, there will be a commit that doesn't build. Also, I believe git (/github) has greatly improved their rename tracking, so the diff might not be so bad (did you test it?). But atomic commit trumps readability in that case.
As mentioned above, dub compiles everything. You might want to add the tests to the excludedSourceFiles so is only compiled on unittests.

Given that we use the fake wrapper you mentioned in #753 this is good to go after the fixes. Just add the commented out size check and a comment explaining why it is not currently correct.

Geod24 · 2020-04-15T13:29:54Z

source/scpd/quorum/QuorumIntersectionChecker.d

+extern (C++, `stellar`):
+
+/// Ditto
+extern(C++, class) public abstract class QuorumIntersectionChecker


Suggested change

extern(C++, class) public abstract class QuorumIntersectionChecker

public abstract class QuorumIntersectionChecker

Geod24 · 2020-04-15T13:30:32Z

source/scpd/quorum/QuorumIntersectionChecker.d

+import scpd.types.Stellar_types;
+import scpd.quorum.QuorumTracker;


Suggested change

import scpd.types.Stellar_types;

import scpd.quorum.QuorumTracker;

import scpd.quorum.QuorumTracker;

import scpd.types.Stellar_types;

source/scpd/quorum/QuorumTracker.d

Geod24 · 2020-04-15T13:33:10Z

source/scpd/quorum/QuorumTracker.d

+    //     `id` was known and didn't have a quorumset
+    // returns false on failure
+    // if expand fails, the caller should instead use `rebuild`
+    bool expand(const ref NodeID id, SCPQuorumSetPtr qSet);


This was not addressed

Suggested change

bool expand(const ref NodeID id, SCPQuorumSetPtr qSet);

bool expand (const ref NodeID id, SCPQuorumSetPtr qSet);

Geod24 · 2020-04-15T13:34:19Z

source/scpd/quorum/QuorumTracker.d

+
+    Bindings for quorum/QuorumTracker.h
+
+    Note: D code does not need to use anything in this class except the


Suggested change

Note: D code does not need to use anything in this class except the

D code does not need to use anything in this class except the

Having Note: will create a section which will look bad in the doc IIRC. If you really intended this, perhaps

Note: D code does not [...]

is more appropriate

Geod24 · 2020-04-15T14:07:04Z

source/scpd/quorum/QuorumTracker.d

+    this (SCP* scp);
+
+    /// returns true if id is in transitive quorum for sure
+    bool isNodeDefinitelyInQuorum(const ref NodeID id);


This is not correct, and neither are the following ones. In D those functions are virtual, while in C++ they are not. You can make either the affected functions final, or, if the class doesn't have a single virtual function, make the class itself final.

From my experience, things declared as class with no virtual method => struct. And even more so if the class is stored as a member of another type.

In D those functions are virtual

Oh damn. How come it worked?

From my experience, things declared as class with no virtual method => struct.

What about classes which inherit other classes? This one specifically is defined as class QuorumTracker : public NonMovableOrCopyable, so it looks like it was never supposed to be used as a POD type.

class QuorumTracker : public NonMovableOrCopyable

Sounds like the idiomatic way to do this in D would be:

mixin template NonMovableOrCopyable (T /* = typeof(this) */) // Not sure if the default arg works { @disable this(this); @disable ref T opAssign () (auto ref T other); } public struct QuorumTracker { mixin NonMovableOrCopyable!(QuorumTracker); }

That might be a bit of an overreaching statement, but my experience is that good, principled C++ code ends up being idiomatic D code, and trying to use the same construct in D as one does in C++ might lead one astray.

Oh damn. How come it worked?

Depends on the usage. The code called from C++ doesn't have an issue, obviously. The code called from D may or may not crash, depending on whether or not the compiler devirtualize the call. And since we dropped DMD, there is a good chance it does :)

Note, the above snippet doesn't fully capture the C++ intent though, since in D, you cannot declare a struct are non movable. Should probably include a copy constructor too but it currently triggers a cascade of bugs.

And since we dropped DMD, there is a good chance it does :)

LOL.

Yeah also I dislike virtual by default in D. Where is Manu when you need him.

Geod24 · 2020-04-15T14:13:54Z

source/scpd/quorum/QuorumTracker.d

+    bool expand(const ref NodeID id, SCPQuorumSetPtr qSet);
+
+    /// Lookup function to find a mapping from Node ID => Quorum set
+    private alias CallbackType = SCPQuorumSetPtr delegate(const ref NodeID) @trusted @nogc nothrow;


This is not used

source/scpd/quorum/QuorumIntersectionChecker.d

Geod24 · 2020-04-15T14:39:25Z

source/scpd/quorum/QuorumTracker.d

+    SCP* mSCP;
+    QuorumMap mQuorum;
+
+public:


I don't mind the usage of :-attributes, but can we merge the two public: together ?

Actually I dislike : attributes personally. I'll fix up the mistake though.

Geod24 · 2020-04-15T15:00:46Z

source/scpd/quorum/QuorumIntersectionTests.d

+import scpd.quorum.QuorumIntersectionChecker;
+import scpd.quorum.QuorumTracker;


Wrong order

AndrejMitrovic · 2020-04-16T05:22:54Z

I didn't port the SCP tests when I imported the SCP code.

I wanted to verify the binding was correct. It wasn't a big effort tho, and it did catch bugs in my binding.

That being said I guess we don't really need it anymore. We will have separate tests for the quorums we generate, so maybe we can remove this code?

It might even be complicated to have something which is Stellar-licensed in our codebase, but I'm not lawyer.

but since dub compiles everything, there will be a commit that doesn't build.

Yeah this was purely to make it easy to review. I'll squash them.

As mentioned above, dub compiles everything. You might want to add the tests to the excludedSourceFiles so is only compiled on unittests.

Oh right. Will fix.

Geod24 · 2020-04-16T05:51:24Z

It might even be complicated to have something which is Stellar-licensed in our codebase, but I'm not lawyer.

We already do (SCP), and it's Apache licensed (and we have LICENSE-APACHE.txt included and haven't removed any authorship / license information), so .

Geod24 · 2020-04-16T06:10:56Z

agora.obj : error LNK2001: unresolved external symbol "public: virtual __cdecl stellar::QuorumIntersectionChecker::~QuorumIntersectionChecker(void)" (??1QuorumIntersectionChecker@stellar@@UEAA@XZ)

AndrejMitrovic · 2020-04-16T06:12:17Z

Also the block storage checksum thing failed on Windows again.

Like I said in #731, why not just version it out on Windows and fix it later?

AndrejMitrovic · 2020-04-16T06:15:17Z

Can I just get rid of the destructor? I don't know why it's defined on the C++ side:

virtual ~QuorumIntersectionChecker(){};

Geod24 · 2020-04-16T06:17:47Z

It's defined so that there is an entry in the vtable for the destructor. Otherwise you end up with things like this.
Except in C++ it is inline for the base class. Just provide an empty destructor in the D binding and you should be 👍

AndrejMitrovic · 2020-04-16T06:25:10Z

I don't understand where the submodule update is coming from..

See also: stellar/stellar-core#2127 Also made changes to make the Quorum checker buildable In stellar-core v11.2.0 the quorum intersection checker depends on Stellar's Config struct. This dependency was removed here so we can use the code. Additionally shared_ptr was removed from one of the parameters as it was causing linker issues, plus we do not have a complete binding to shared_ptr with the correct semantics yet. Co-authored-by: MonsieurNicolas <nicolas@stellar.org>

This allowed us to catch a few bugs with our bindings. Co-authored-by: AndrejMitrovic <andrej.mitrovich@gmail.com>

AndrejMitrovic · 2020-04-16T06:28:06Z

Ok I managed to get rid of it.

Geod24

Thanks

AndrejMitrovic added status-blocked Another issue/PR/external dependency is blocking completion of this PR/issue type-feature An addition to the system introducing new functionalities labels Mar 16, 2020

AndrejMitrovic linked an issue Mar 16, 2020 that may be closed by this pull request

Design and implement quorum balancing #240

Closed

6 tasks

AndrejMitrovic force-pushed the quorum-intersection branch from a314020 to 39eec98 Compare March 18, 2020 01:25

AndrejMitrovic added a commit to AndrejMitrovic/agora that referenced this pull request Mar 20, 2020

[REMOVE ME] Merge bosagora#662 & bosagora#664

7a58a1b

AndrejMitrovic mentioned this pull request Mar 20, 2020

[Draft] Implement a quorum balancing algorithm #684

Closed

AndrejMitrovic added a commit to AndrejMitrovic/agora that referenced this pull request Mar 20, 2020

[REMOVE ME] Merge bosagora#662 & bosagora#664

6b0c713

AndrejMitrovic removed a link to an issue Mar 22, 2020

Design and implement quorum balancing #240

Closed

6 tasks

AndrejMitrovic added a commit to AndrejMitrovic/agora that referenced this pull request Mar 24, 2020

[REMOVE ME] Merge bosagora#662 & bosagora#664

c67e4d1

AndrejMitrovic removed the status-blocked Another issue/PR/external dependency is blocking completion of this PR/issue label Mar 26, 2020

AndrejMitrovic force-pushed the quorum-intersection branch 6 times, most recently from 687622b to f740fb1 Compare March 26, 2020 04:19

AndrejMitrovic added a commit to AndrejMitrovic/agora that referenced this pull request Mar 30, 2020

[REMOVE ME] Merge bosagora#662 & bosagora#664

5eb573b

AndrejMitrovic added a commit to AndrejMitrovic/agora that referenced this pull request Mar 30, 2020

[REMOVE ME] Merge bosagora#662 & bosagora#664

681ea5a

AndrejMitrovic added a commit to AndrejMitrovic/agora that referenced this pull request Mar 30, 2020

[REMOVE ME] Merge bosagora#664

d532f37

Geod24 force-pushed the quorum-intersection branch from f740fb1 to 0d1bf7d Compare April 5, 2020 18:50

AndrejMitrovic mentioned this pull request Apr 6, 2020

Design and implement quorum balancing #240

Closed

6 tasks

AndrejMitrovic commented Apr 6, 2020

View reviewed changes

AndrejMitrovic commented Apr 7, 2020

View reviewed changes

AndrejMitrovic force-pushed the quorum-intersection branch from 0d1bf7d to affed9d Compare April 8, 2020 05:51

AndrejMitrovic force-pushed the quorum-intersection branch 5 times, most recently from 949aa53 to cdb98a1 Compare April 13, 2020 06:16

AndrejMitrovic pushed a commit to AndrejMitrovic/agora that referenced this pull request Apr 13, 2020

[REMOVE ME] Merge bosagora#664

e8c1e44

Geod24 reviewed Apr 14, 2020

View reviewed changes

source/scpp/README.md Show resolved Hide resolved

Geod24 suggested changes Apr 15, 2020

View reviewed changes

AndrejMitrovic force-pushed the quorum-intersection branch from cdb98a1 to 1ca17e1 Compare April 16, 2020 05:47

AndrejMitrovic force-pushed the quorum-intersection branch 3 times, most recently from 6af429b to 5c49f1c Compare April 16, 2020 06:03

AndrejMitrovic force-pushed the quorum-intersection branch from 5c49f1c to 66c7ff6 Compare April 16, 2020 06:19

graydon and others added 3 commits April 16, 2020 15:27

Add D bindings to the quorum intersection code

7d0a484

Port the QuorumIntersectionTests to D

5bf37db

This allowed us to catch a few bugs with our bindings. Co-authored-by: AndrejMitrovic <andrej.mitrovich@gmail.com>

AndrejMitrovic force-pushed the quorum-intersection branch from 66c7ff6 to 5bf37db Compare April 16, 2020 06:27

Geod24 approved these changes Apr 16, 2020

View reviewed changes

Geod24 merged commit 334ea46 into bosagora:v0.x.x Apr 16, 2020

AndrejMitrovic deleted the quorum-intersection branch April 16, 2020 07:37

	extern (D) public final void rebuild (SCPQuorumSetPtr delegate(const ref NodeID) lookup)
	extern (D) public final void rebuild (SCPQuorumSetPtr delegate(const ref NodeID) lookup) @trusted


		extern (C++, `stellar`):

		// helper class to help track the overall quorum over time

	bool expand(const ref NodeID id, SCPQuorumSetPtr qSet);
	bool expand (const ref NodeID id, SCPQuorumSetPtr qSet);

		@@ -15,6 +15,8 @@ Some files have been modified after porting:

		- Some `private` declarations were changed to `public`. This allows us to test size & layout checks for our D glue layer, for example: https://github.com/bpfkorea/agora/blob/76e4ca7d77b123ef867d3ce99c1c1b26afab761d/source/scpp/src/crypto/ByteSlice.h#L21 used in https://github.com/bpfkorea/agora/blob/76e4ca7d77b123ef867d3ce99c1c1b26afab761d/source/scpp/extra/DSizeChecks.cpp and https://github.com/bpfkorea/agora/blob/76e4ca7d77b123ef867d3ce99c1c1b26afab761d/source/scpp/extra/DLayoutChecks.cpp.

		- The `src/quorum` folder adapts the Quorum intersection tool which is used in Stellar's Herder implementation (not SCP). The `update.d` script does not track these files yet, so they have to be kept manually in sync when updating the bindings.

	extern(C++, class) public abstract class QuorumIntersectionChecker
	public abstract class QuorumIntersectionChecker

		import scpd.types.Stellar_types;
		import scpd.quorum.QuorumTracker;


		Bindings for quorum/QuorumTracker.h

		Note: D code does not need to use anything in this class except the

	Note: D code does not need to use anything in this class except the
	D code does not need to use anything in this class except the

		import scpd.quorum.QuorumIntersectionChecker;
		import scpd.quorum.QuorumTracker;

Add the quorum intersection checker code from stellar-core #664

Add the quorum intersection checker code from stellar-core #664

Conversation

AndrejMitrovic commented Mar 16, 2020 • edited Loading

codecov bot commented Mar 16, 2020 • edited Loading

Codecov Report

AndrejMitrovic commented Mar 25, 2020

Geod24 commented Apr 5, 2020

Geod24 commented Apr 5, 2020

AndrejMitrovic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

Choose a reason for hiding this comment

This comment was marked as off-topic.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as outdated.

Choose a reason for hiding this comment

This comment was marked as off-topic.

Choose a reason for hiding this comment

This comment was marked as resolved.

AndrejMitrovic commented Apr 7, 2020

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as resolved.

This comment was marked as resolved.

AndrejMitrovic commented Apr 13, 2020

AndrejMitrovic commented Apr 13, 2020 • edited Loading

Geod24 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrejMitrovic commented Apr 16, 2020

Geod24 commented Apr 16, 2020

Geod24 commented Apr 16, 2020 • edited Loading

AndrejMitrovic commented Apr 16, 2020

AndrejMitrovic commented Apr 16, 2020

Geod24 commented Apr 16, 2020

AndrejMitrovic commented Apr 16, 2020

AndrejMitrovic commented Apr 16, 2020

Geod24 left a comment

Choose a reason for hiding this comment

AndrejMitrovic commented Mar 16, 2020 •

edited

Loading

codecov bot commented Mar 16, 2020 •

edited

Loading

AndrejMitrovic commented Apr 13, 2020 •

edited

Loading

Geod24 commented Apr 16, 2020 •

edited

Loading