Implement catch-up phase #80

AndrejMitrovic · 2019-07-09T09:51:22Z

It's not ready for merging yet, but it can be looked at.

This still needs a bit of refactoring, separating atomic changes into their own commits.

AndrejMitrovic · 2019-07-09T09:52:37Z

Also this only implements catch-up during booting up. The other definitions of done in the Catchup Phase Issue need to be solved.

Geod24 · 2019-07-09T10:02:12Z

source/agora/test/Ledger.d

+    import std.algorithm;
+    import std.conv;
+    import std.format;
+    import std.digest.sha;


Should be removed

damn where did that sneak in from lol

AndrejMitrovic · 2019-07-09T10:47:16Z

I've atomic'ed (atomized?) the commits, it should look much nicer now.

I probably need to think of edge cases to add to the unittests.

source/agora/node/RemoteNode.d

Geod24 · 2019-07-09T14:30:28Z

source/agora/test/Base.d

    ///
    public this (Config config)
    {
        this.mem_metadata = new MemMetadata;
+        this.ledger = new Ledger;


That code doesn't make sense to me. You instantiate a new Ledger just to override it with itself in the parent constructor. You also shadow the parent ledger variable ?

The shadowing was an oversight. I'll simplify this a bit tomorrow.

AndrejMitrovic · 2019-07-10T05:28:23Z

Update: Added periodic catch-up phase, and added a network test for it. This completes all of the tasks and should cover everything in the definition of done.

Geod24

I would merge the last two commits. Also, do you want the ability to arbitrarily delay a node ? Because I can give you that power easily in LocalRest.

Geod24 · 2019-07-10T05:58:57Z

source/agora/test/Base.d

@@ -175,6 +177,9 @@ public interface TestAPI : API

    ///
    public abstract void metaAddPeer(string peer);
+
+    ///
+    public abstract void addBlockToLedger (Block block);


Why do you need this ? I don't see an override anywhere ?

Well, the question is how do you get access to the Ledger from the unittest. As it's required right now. edit: reworded

You never access the internals of a node at the moment.

Well the first test is:

Instantiate the nodes. Don't start networking yet. One node (node A) needs to have 100 blocks, the 3 are empty.

Then, make them connect to each other. The 3 other nodes will retrieve blocks from the 1st node on boot.

But to make this work, I need to add those blocks to node A before it connects to the 3 other nodes.

To be more specific here. I can't just use the sendTransaction API here because this will trigger the gossip protocol to send the transaction over the whole network, and then each node will create their own block. So by the time the node tries to retrieve the blocks from the other nodes, it will already have all the blocks.

Delaying the node might not help. I would need some way to temporarily disable the gossip protocol, but then this becomes very messy to have all these special cases for unittesting.

The more features the node has, the more complicated it is to test a specific feature in the node without triggering an avalanche of other features.

I will be able to replace this code once LocalRest has the ability to filter requests. Then I can do something like:

Use the sendTransaction API to one node

Make the other nodes ignore transaction gossiping

The other nodes will then catch-up using the periodic getBlocksFrom call.

This is way nicer than messing with Node's internals.

Geod24 · 2019-07-10T05:58:57Z

source/agora/test/Ledger.d

+{
+    import core.thread;
+    import std.conv;
+    import vibe.core.log;


Two leftover imports

source/agora/node/RemoteNode.d

AndrejMitrovic · 2019-07-10T06:30:35Z

I've squashed the two commits.

I'm thinking about how to get rid of those methods in the TestAPI now.

AndrejMitrovic · 2019-07-10T06:32:27Z

Also, do you want the ability to arbitrarily delay a node ? Because I can give you that power easily in LocalRest.

Hmm.. that might be very useful.

AndrejMitrovic · 2019-07-10T06:50:36Z

All the Mac builds fail. Hmm..

AndrejMitrovic · 2019-07-10T06:50:59Z

It's a timing issue.

AndrejMitrovic · 2019-07-10T08:06:49Z

Ok I made a more stable version of the tests that shouldn't suffer too much from timing issues. I added polling and a timeout of 4 seconds.

AndrejMitrovic · 2019-07-11T00:53:03Z

Very peculiar that macOS dmd 2.085.1 seems to consistently be failing now..

AndrejMitrovic · 2019-07-11T01:22:06Z

~@Geod24 I'm having consistent time-outs in the MacOS build: https://travis-ci.com/bpfkorea/agora/jobs/214998228

It seems it gets stuck trying to install DMD & Dub.~

And it's green.

Geod24 · 2019-07-11T08:30:00Z

I guess you want to rebase ?

AndrejMitrovic · 2019-07-11T08:56:04Z

I guess you want to rebase ?

Ah I thought this one was rebased.

AndrejMitrovic · 2019-07-11T08:56:15Z

I mean merged.

AndrejMitrovic · 2019-07-11T09:04:53Z

Rebased.

Geod24

As discussed IRL: Bad rebase, some code is missing

Geod24 · 2019-07-12T00:28:03Z

source/agora/node/Network.d

+            // periodic task
+            while (1)
+            {
+                logInfo("Retrieving latest blocks..");


I think this logging should only happen if we are actually catching up

Geod24 · 2019-07-12T00:28:03Z

source/agora/node/RemoteNode.d

+
+    ***************************************************************************/
+
+    public ulong getBlockHeight ( )


Suggested change

public ulong getBlockHeight ( )

public ulong getBlockHeight ()

Geod24 · 2019-07-12T00:28:03Z

source/agora/test/Ledger.d

@@ -22,13 +23,24 @@ import agora.common.Hash;
 import agora.common.Transaction;
 import agora.test.Base;

+/// Returns: true if all node's block heights are equal to the provided height
+private bool allHeightsEqual ( TestAPI[] nodes, size_t height )


Suggested change

private bool allHeightsEqual ( TestAPI[] nodes, size_t height )

private bool allHeightsEqual (TestAPI[] nodes, size_t height)

AndrejMitrovic · 2019-07-13T07:45:53Z

Fixed the suggested changes.

It will be used to add blocks received from other nodes.

AndrejMitrovic · 2019-07-15T01:45:12Z

Rebased to use new LocalRest sleep() feature.

However it's a bit ugly because now I have to store RemoteAPI!TestAPI in TestNetworkManager, not just TestAPI. And then there's the issue of RemoteAPI!TestAPI[] not implicitly converting to TestAPI[].

AndrejMitrovic · 2019-07-15T01:51:09Z

Edit: Pushed a simplification https://github.com/bpfkorea/agora/compare/3c38af6c675d18cab84aa8c89c702d6b3cf8bb7c..bf03c16e9bc1267934450648f55a9f583ffbc9e1

Geod24 · 2019-07-15T02:21:44Z

source/agora/node/Network.d

+    ***************************************************************************/
+
+    private void getBlocksFrom (ulong block_height,
+        void delegate(Block[]) @safe onReceivedBlocks)


Suggested change

void delegate(Block[]) @safe onReceivedBlocks)

scope void delegate(Block[]) @safe onReceivedBlocks)

Geod24 · 2019-07-15T02:21:44Z

source/agora/node/Node.d

+
+    protected Ledger getLedger ()
+    {
+        return new Ledger();


I don't think you need this anymore ?

Geod24 · 2019-07-15T02:22:29Z

source/agora/test/Ledger.d

+{
+    import core.thread;
+    import std.algorithm;
+    import std.format;


Those 2 imports are not necessary anymore

I need each still.

Geod24 · 2019-07-15T02:23:07Z

source/agora/test/Ledger.d

@@ -22,13 +23,24 @@ import agora.common.Hash;
 import agora.common.Transaction;
 import agora.test.Base;

+/// Returns: true if all node's block heights are equal to the provided height
+private bool allHeightsEqual (API)(API[] nodes, size_t height)


Suggestion: isConsistent and check the block headers for equality ?

AndrejMitrovic · 2019-07-15T02:27:06Z

Updated with suggested changes.

Geod24 · 2019-07-15T02:32:02Z

source/agora/test/Ledger.d

+    import core.thread;
+    import std.algorithm;
+    import std.conv;
+    import std.format;


conv and format superfluous

Geod24

LGTM

After a node connects to all its peers, it will retrieve the latest blocks and apply them to its ledger. The node will keep periodically trying to retrieve the latest blocks from its connected peers.

AndrejMitrovic · 2019-07-15T02:41:23Z

Pushed a small update.

AndrejMitrovic added the type-feature An addition to the system introducing new functionalities label Jul 9, 2019

Geod24 reviewed Jul 9, 2019

View changes

AndrejMitrovic force-pushed the catchup-phase branch from 73701f5 to 3c58d26 Compare Jul 9, 2019

Geod24 reviewed Jul 9, 2019

View changes

source/agora/node/RemoteNode.d Outdated Show resolved Hide resolved

Geod24 reviewed Jul 9, 2019

View changes

AndrejMitrovic mentioned this pull request Jul 10, 2019

Simplify creation of metadata in the unittests #82

Merged

AndrejMitrovic force-pushed the catchup-phase branch 2 times, most recently from 3cba60a to ce985fb Compare Jul 10, 2019

AndrejMitrovic changed the title ~~[wip] Implement catch-up phase~~ Implement catch-up phase Jul 10, 2019

AndrejMitrovic force-pushed the catchup-phase branch from ce985fb to 0316951 Compare Jul 10, 2019

Geod24 suggested changes Jul 10, 2019

View changes

AndrejMitrovic force-pushed the catchup-phase branch 2 times, most recently from f2ec64b to 2636b48 Compare Jul 10, 2019

AndrejMitrovic force-pushed the catchup-phase branch 2 times, most recently from dc1e472 to 0cd569a Compare Jul 10, 2019

AndrejMitrovic force-pushed the catchup-phase branch 2 times, most recently from c29d528 to 3ea3762 Compare Jul 10, 2019

AndrejMitrovic force-pushed the catchup-phase branch from 3ea3762 to d71cc65 Compare Jul 11, 2019

AndrejMitrovic force-pushed the catchup-phase branch from d71cc65 to d064264 Compare Jul 11, 2019

Geod24 suggested changes Jul 12, 2019

View changes

AndrejMitrovic force-pushed the catchup-phase branch from d064264 to 2e67838 Compare Jul 13, 2019

AndrejMitrovic added 4 commits Jul 15, 2019

Remove unused import

3287795

Make addNewBlock() public

d78a142

It will be used to add blocks received from other nodes.

Implement getBlockHeight() wrapper in RemoteNode

cc0e5e2

Implement getBlocksFrom() wrapper in RemoteNode

6cad1fe

AndrejMitrovic force-pushed the catchup-phase branch from 2e67838 to 3c38af6 Compare Jul 15, 2019

AndrejMitrovic force-pushed the catchup-phase branch from 3c38af6 to bf03c16 Compare Jul 15, 2019

AndrejMitrovic force-pushed the catchup-phase branch from bf03c16 to 745f098 Compare Jul 15, 2019

Geod24 reviewed Jul 15, 2019

View changes

AndrejMitrovic force-pushed the catchup-phase branch from 745f098 to 758fa8a Compare Jul 15, 2019

Geod24 reviewed Jul 15, 2019

View changes

Geod24 approved these changes Jul 15, 2019

View changes

Implement catchup phase of a node

93cc851

After a node connects to all its peers, it will retrieve the latest blocks and apply them to its ledger. The node will keep periodically trying to retrieve the latest blocks from its connected peers.

AndrejMitrovic force-pushed the catchup-phase branch from 758fa8a to 93cc851 Compare Jul 15, 2019

Geod24 approved these changes Jul 15, 2019

View changes

Geod24 merged commit 450fbba into bosagora:v0.x.x Jul 15, 2019
1 check passed

AndrejMitrovic deleted the catchup-phase branch Jul 15, 2019


		***************************************************************************/

		public ulong getBlockHeight ( )

	public ulong getBlockHeight ( )
	public ulong getBlockHeight ()

	private bool allHeightsEqual ( TestAPI[] nodes, size_t height )
	private bool allHeightsEqual (TestAPI[] nodes, size_t height)

	void delegate(Block[]) @safe onReceivedBlocks)
	scope void delegate(Block[]) @safe onReceivedBlocks)

Implement catch-up phase #80

Implement catch-up phase #80

Conversation

AndrejMitrovic commented Jul 9, 2019

AndrejMitrovic commented Jul 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrejMitrovic commented Jul 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrejMitrovic commented Jul 10, 2019

Geod24 left a comment

Choose a reason for hiding this comment

AndrejMitrovic Jul 10, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrejMitrovic Jul 10, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrejMitrovic commented Jul 10, 2019

AndrejMitrovic commented Jul 10, 2019

AndrejMitrovic commented Jul 10, 2019

AndrejMitrovic commented Jul 10, 2019

AndrejMitrovic commented Jul 10, 2019

AndrejMitrovic commented Jul 11, 2019

AndrejMitrovic commented Jul 11, 2019 • edited

Geod24 commented Jul 11, 2019

AndrejMitrovic commented Jul 11, 2019

AndrejMitrovic commented Jul 11, 2019

AndrejMitrovic commented Jul 11, 2019

Geod24 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrejMitrovic commented Jul 13, 2019

AndrejMitrovic commented Jul 15, 2019

AndrejMitrovic commented Jul 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrejMitrovic commented Jul 15, 2019

Choose a reason for hiding this comment

Geod24 left a comment

AndrejMitrovic commented Jul 15, 2019

AndrejMitrovic Jul 10, 2019 •

edited

AndrejMitrovic Jul 10, 2019 •

edited

AndrejMitrovic commented Jul 11, 2019 •

edited