Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore MemoryPool transactions from saved consensus context. #575

Closed
wants to merge 45 commits into from
Closed
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
999251c
Save consensus context each time we change view (so we will be close …
Jan 25, 2019
2ce8b71
Add extension methods for loading and saving the ConsensusContext to …
Jan 25, 2019
ae56b38
Load MemoryPool initially with transactions from saved consensus.
Jan 25, 2019
232e2fd
Fix tests.
Jan 25, 2019
d930e2c
Merge branch 'consensus/improved_dbft' into consensus/recoveryLog2
vncoelho Jan 25, 2019
4ae84f7
Remove unused 'using' statements.
Jan 25, 2019
947aa53
Merge branch 'consensus/improved_dbft' into consensus/recoveryLog2
Jan 27, 2019
91a2e3a
Merge branch 'consensus/improved_dbft' into consensus/recoveryLog2
shargon Jan 28, 2019
dc7482c
Pass consensus store into the NeoSystem constructor instead of StartC…
Jan 29, 2019
411f905
Add one extra check when restoring transactions from saved consensus …
Jan 29, 2019
f43a5ed
Move loading transactions from the last saved ConsensusContext out of…
Jan 29, 2019
28f05eb
More clean-up.
Jan 29, 2019
91f31de
Further decoupling, don't expose the consensus store from NeoSystem.
Jan 29, 2019
eedeffb
Handle saving and restoring consensus conetext when RequestSent, Requ…
Jan 30, 2019
8eaf669
Make unit tests pass again. Consensus unit tests still need attention…
Jan 30, 2019
240c5bb
Don't write the consensus state after resposne sent to reduce writes.
Jan 30, 2019
92e6987
Remove unneeded commented code.
Jan 31, 2019
3b06474
Fix saving context to include the miner transaction after receiving t…
Jan 31, 2019
323384c
Ensure consensus context has snapshot and keypair set upon loading th…
Jan 31, 2019
d312b2c
Fix loading transactions into the MemoryPool from consensus context.
Jan 31, 2019
6c33b62
More precise check when restoring consensus context.
Jan 31, 2019
1c744ea
Don't need to save consensus state on request sent or received since …
Feb 4, 2019
1d81e54
Merge branch 'consensus/improved_dbft' into consensus/recoveryLog2
Feb 16, 2019
700a0aa
Fix test compile.
Feb 16, 2019
62d5e5a
Fix tests.
Feb 16, 2019
5e29fdc
Don't separate the code for obtaining transactions into a separte met…
Feb 16, 2019
48035df
Merge branch 'consensus/improved_dbft' into consensus/recoveryLog2
erikzhang Feb 17, 2019
9b16fb8
Merge branch 'consensus/improved_dbft' into consensus/recoveryLog2
Feb 19, 2019
f955db5
Write consensus state synchronously upon commit. Move constant for co…
Feb 19, 2019
8aeaf80
Use the passed writeOptions for Put to the store.
Feb 19, 2019
82a0f2b
Decouple restoring memory pool from the consensus state from the Bloc…
Feb 19, 2019
3dd2928
Remove unused using statement.
Feb 19, 2019
2fc2139
Clean-up.
Feb 19, 2019
fc4cfe8
More clean-up. Remove unused using statements.
Feb 19, 2019
14a0a2c
Remove one more unused using statement.
Feb 19, 2019
7e0f518
Merge branch 'consensus/improved_dbft' into consensus/recoveryLog2
jsolman Feb 20, 2019
3d6a2b1
Minor changes
erikzhang Feb 20, 2019
438c105
Add method `PutSync()` to class `Store`
erikzhang Feb 20, 2019
e362f55
Merge branch 'consensus/improved_dbft' into consensus/recoveryLog2
Feb 20, 2019
465e587
Merge branch 'consensus/improved_dbft' into consensus/recoveryLog2
erikzhang Feb 21, 2019
3ebb75e
Renaming
erikzhang Feb 21, 2019
75c81d9
Remove unused variable.
Feb 21, 2019
becf0f5
Optimize the serialization of `ConsensusContext`
erikzhang Feb 21, 2019
0f9856e
Merge branch 'consensus/improved_dbft' into consensus/recoveryLog2
erikzhang Feb 21, 2019
b957d1b
Merge branch 'consensus/improved_dbft' into consensus/recoveryLog2
erikzhang Feb 21, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions neo.UnitTests/UT_MemoryPool.cs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
using Neo.IO.Wrappers;
using Neo.Network.P2P.Payloads;
using Neo.Persistence;
using Neo.Persistence.LevelDB;

namespace Neo.UnitTests
{
Expand Down Expand Up @@ -74,6 +75,7 @@ public void TestSetup()
mockStore.Setup(p => p.GetBlockHashIndex()).Returns(new TestMetaDataCache<HashIndexState>());
mockStore.Setup(p => p.GetHeaderHashIndex()).Returns(new TestMetaDataCache<HashIndexState>());
mockStore.Setup(p => p.GetSnapshot()).Returns(mockSnapshot.Object);
mockStore.Setup(p => p.Get(Prefixes.CN_CONTEXT, It.IsAny<byte[]>())).Returns((byte[]) null);

Console.WriteLine("initialize NeoSystem");
TheNeoSystem = new NeoSystem(mockStore.Object); // new Mock<NeoSystem>(mockStore.Object);
Expand Down
20 changes: 5 additions & 15 deletions neo/Consensus/ConsensusService.cs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
using Neo.Wallets;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;

namespace Neo.Consensus
Expand All @@ -22,8 +21,6 @@ public class Start { }
public class SetViewNumber { public byte ViewNumber; }
internal class Timer { public uint Height; public byte ViewNumber; }

private const byte ContextSerializationPrefix = 0xf4;

private readonly IConsensusContext context;
private readonly IActorRef localNode;
private readonly IActorRef taskManager;
Expand Down Expand Up @@ -107,6 +104,8 @@ private void CheckExpectedView(byte view_number)
if (context.ExpectedView.Count(p => p == view_number) >= context.M)
{
InitializeConsensus(view_number);
// Save our view so if we crash and come back we will be closer to the correct view.
context.WriteContextToStore(store);
}
}

Expand All @@ -117,7 +116,7 @@ private void CheckPreparations()
ConsensusPayload payload = context.MakeCommit();
Log($"send commit");
context.State |= ConsensusState.CommitSent;
store.Put(ContextSerializationPrefix, new byte[0], context.ToArray());
context.WriteContextToStore(store);
jsolman marked this conversation as resolved.
Show resolved Hide resolved
localNode.Tell(new LocalNode.SendDirectly { Inventory = payload });
CheckCommits();
}
Expand Down Expand Up @@ -272,7 +271,6 @@ private void OnPrepareRequestReceived(ConsensusPayload payload, PrepareRequest m
}
else
{

if (Blockchain.Singleton.MemPool.TryGetValue(hash, out tx))
unverified.Add(tx);
}
Expand Down Expand Up @@ -338,19 +336,11 @@ private void OnStart()
{
Log("OnStart");
started = true;
byte[] data = store.Get(ContextSerializationPrefix, new byte[0]);
if (data != null)
{
using (MemoryStream ms = new MemoryStream(data, false))
using (BinaryReader reader = new BinaryReader(ms))
{
context.Deserialize(reader);
}
}
context.LoadContextFromStore(store);
if (context.State.HasFlag(ConsensusState.CommitSent))
CheckPreparations();
else
InitializeConsensus(0);
InitializeConsensus(context.ViewNumber);
}

private void OnTimer(Timer timer)
Expand Down
28 changes: 28 additions & 0 deletions neo/Consensus/Helper.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
using System.IO;
using Neo.IO;
using Neo.Persistence;
using Neo.Persistence.LevelDB;

namespace Neo.Consensus
{
public static class Helper
{
internal static void WriteContextToStore(this IConsensusContext context, Store store)
{
store.Put(Prefixes.CN_CONTEXT, new byte[0], context.ToArray());
}

internal static void LoadContextFromStore(this IConsensusContext context, Store store)
{
byte[] data = store.Get(Prefixes.CN_CONTEXT, new byte[0]);
if (data != null)
{
using (MemoryStream ms = new MemoryStream(data, false))
using (BinaryReader reader = new BinaryReader(ms))
{
context.Deserialize(reader);
}
}
}
}
}
8 changes: 8 additions & 0 deletions neo/Ledger/MemoryPool.cs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
using System.Runtime.CompilerServices;
using System.Threading;
using Akka.Util.Internal;
using Neo.Consensus;
using Neo.Network.P2P;
using Neo.Persistence;
using Neo.Plugins;
Expand Down Expand Up @@ -107,6 +108,13 @@ public MemoryPool(NeoSystem system, int capacity)
_system = system;
Capacity = capacity;
LoadMaxTxLimitsFromPolicyPlugins();
IConsensusContext context = new ConsensusContext(null);
context.LoadContextFromStore(system.store);
if (context.Transactions == null) return;
foreach (var tx in context.Transactions.Values)
{
TryAdd(tx.Hash, tx);
}
}

public void LoadMaxTxLimitsFromPolicyPlugins()
Expand Down
2 changes: 1 addition & 1 deletion neo/NeoSystem.cs
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ public class NeoSystem : IDisposable
public IActorRef Consensus { get; private set; }
public RpcServer RpcServer { get; private set; }

private readonly Store store;
internal readonly Store store;
jsolman marked this conversation as resolved.
Show resolved Hide resolved
private Peer.Start start_message = null;
private bool suspend = false;

Expand Down
1 change: 1 addition & 0 deletions neo/Persistence/LevelDB/Prefixes.cs
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,6 @@ internal static class Prefixes
public const byte IX_CurrentHeader = 0xc1;

public const byte SYS_Version = 0xf0;
public const byte CN_CONTEXT = 0xf4;
jsolman marked this conversation as resolved.
Show resolved Hide resolved
}
}