Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Config] Change default barriers of mainnet to 0 #6225

Merged
merged 17 commits into from
Dec 7, 2023
Merged

[Config] Change default barriers of mainnet to 0 #6225

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
2665609
init
smartprogrammer93 Oct 25, 2023
545e0a7
tests
smartprogrammer93 Oct 25, 2023
a79085d
format
smartprogrammer93 Oct 25, 2023
b0b89cc
fix config tests
smartprogrammer93 Oct 25, 2023
dfd0196
cover more cases
smartprogrammer93 Oct 25, 2023
43b4a8d
Add same logic to SyncProgressResolver with tests
smartprogrammer93 Oct 26, 2023
ed1029a
wip metadatadb usage to detect with what configuration was the sync s…
smartprogrammer93 Oct 28, 2023
3ad31de
add logic to ReceiptsSyncFeed
smartprogrammer93 Nov 3, 2023
8f8a782
fix ReceiptsSyncFeed
smartprogrammer93 Nov 3, 2023
74cc0ea
apply fix
smartprogrammer93 Nov 3, 2023
ca21641
remove useless using
smartprogrammer93 Nov 3, 2023
e25b7d2
Fix bodies not finishing
smartprogrammer93 Nov 4, 2023
245241a
Merge branch 'master' into feature/change-mainnet-default-barriers-to-0
smartprogrammer93 Nov 28, 2023
93f380c
refactor and minor fixes
smartprogrammer93 Nov 28, 2023
810de42
Merge branch 'master' into feature/change-mainnet-default-barriers-to-0
smartprogrammer93 Dec 6, 2023
63a8d16
Merge branch 'master' into feature/change-mainnet-default-barriers-to-0
smartprogrammer93 Dec 6, 2023
91d261b
fix test issue
smartprogrammer93 Dec 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/Nethermind/Nethermind.Db/MetadataDbKeys.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,7 @@ public static class MetadataDbKeys
public const int LowestInsertedBeaconHeaderHash = 7;
public const int FirstPoSHash = 8;
public const int UpdatedPivotData = 9;
public const int ReceiptsBarrierWhenStarted = 10;
public const int BodiesBarrierWhenStarted = 11;
}
}
2 changes: 1 addition & 1 deletion src/Nethermind/Nethermind.Runner.Test/ConfigFilesTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ public void Migrations_are_not_enabled_by_default(string configWildcard)
}

[TestCase("^mainnet", 0)]
[TestCase("mainnet fast", 11052984)]
[TestCase("mainnet fast", 0)]
public void Barriers_defaults_are_correct(string configWildcard, long barrier)
{
Test<ISyncConfig, long>(configWildcard, c => c.AncientBodiesBarrier, barrier);
Expand Down
4 changes: 1 addition & 3 deletions src/Nethermind/Nethermind.Runner/configs/mainnet.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@
"PivotHash": "0x621cd6a2635a8746f0889cfb81a63d24933dca62cd3dd1b7bcc87c73475b1923",
"PivotTotalDifficulty": "58750003716598352816469",
"FastBlocks": true,
"AncientBodiesBarrier": 11052984,
"AncientReceiptsBarrier": 11052984,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to remove these values completely? If user decide to enable them he can set value close to the head and CL will not be able to sync deposit contract. Maybe we can introduce something like Sync.EnableBarrier?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think the user will set the value from his mind. If the user wants to enable them, we can have something in the docs to guide?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair, but I think it's better option to have default value for this. User may decide to set it to current head

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that is desired, we can just make another config file, like mainet-barriers, for example

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd better go with mainnet-light or something but not sure.

"FastSyncCatchUpHeightDelta": "10000000000"
},
"EthStats": {
Expand All @@ -41,4 +39,4 @@
"Merge": {
"Enabled": true
}
}
}
89 changes: 71 additions & 18 deletions src/Nethermind/Nethermind.Synchronization.Test/FastBlocks/BodiesSyncFeedTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,12 @@
using Nethermind.Blockchain;
using Nethermind.Blockchain.Synchronization;
using Nethermind.Core;
using Nethermind.Core.Extensions;
using Nethermind.Core.Test;
using Nethermind.Core.Test.Builders;
using Nethermind.Db;
using Nethermind.Logging;
using Nethermind.Specs;
using Nethermind.Synchronization.FastBlocks;
using Nethermind.Synchronization.ParallelSync;
using Nethermind.Synchronization.Peers;
Expand All @@ -22,9 +25,13 @@ namespace Nethermind.Synchronization.Test.FastBlocks;

public class BodiesSyncFeedTests
{
private BlockTree _syncingFromBlockTree = null!;
private IBlockTree _syncingFromBlockTree = null!;
private IBlockTree _syncingToBlockTree = null!;
private TestMemDb _blocksDb = null!;
private BodiesSyncFeed _syncFeed = null!;
private BodiesSyncFeed _feed = null!;
private ISyncConfig _syncConfig = null!;
private MemDb _metadataDb = null!;
private Block _pivotBlock = null!;

[SetUp]
public void Setup()
Expand All @@ -34,53 +41,56 @@ public void Setup()
.TestObject;

_blocksDb = new TestMemDb();
BlockTree syncingTooBlockTree = Build.A.BlockTree()
_metadataDb = new MemDb();
_syncingToBlockTree = Build.A.BlockTree()
.WithBlocksDb(_blocksDb)
.TestObject;

for (int i = 1; i < 100; i++)
{
Block block = _syncingFromBlockTree.FindBlock(i, BlockTreeLookupOptions.None)!;
syncingTooBlockTree.Insert(block.Header);
_syncingToBlockTree.Insert(block.Header);
}

Block pivot = _syncingFromBlockTree.FindBlock(99, BlockTreeLookupOptions.None)!;
_pivotBlock = _syncingFromBlockTree.FindBlock(99, BlockTreeLookupOptions.None)!;

SyncConfig syncConfig = new SyncConfig()
_syncConfig = new SyncConfig()
{
FastSync = true,
PivotHash = pivot.Hash!.ToString(),
PivotNumber = pivot.Number.ToString(),
PivotHash = _pivotBlock.Hash!.ToString(),
PivotNumber = _pivotBlock.Number.ToString(),
AncientBodiesBarrier = 0,
FastBlocks = true,
DownloadBodiesInFastSync = true,
};

_syncFeed = new BodiesSyncFeed(
syncingTooBlockTree,
_feed = new BodiesSyncFeed(
MainnetSpecProvider.Instance,
_syncingToBlockTree,
Substitute.For<ISyncPeerPool>(),
syncConfig,
_syncConfig,
new NullSyncReport(),
_blocksDb,
_metadataDb,
LimboLogs.Instance,
flushDbInterval: 10
);
_syncFeed.InitializeFeed();
}

[Test]
public async Task ShouldCallFlushPeriodically()
{
BodiesSyncBatch req = (await _syncFeed.PrepareRequest())!;
_feed.InitializeFeed();
BodiesSyncBatch req = (await _feed.PrepareRequest())!;
_blocksDb.FlushCount.Should().Be(1);

async Task HandleAndPrepareNextRequest()
{
req.Response = new OwnedBlockBodies(req.Infos.Take(8).Select((info) =>
_syncingFromBlockTree.FindBlock(info!.BlockNumber, BlockTreeLookupOptions.None)!.Body).ToArray());

_syncFeed.HandleResponse(req);
req = (await _syncFeed.PrepareRequest())!;
_feed.HandleResponse(req);
req = (await _feed.PrepareRequest())!;
}

await HandleAndPrepareNextRequest();
Expand All @@ -99,7 +109,8 @@ async Task HandleAndPrepareNextRequest()
[Test]
public async Task ShouldRecoverOnInsertFailure()
{
BodiesSyncBatch req = (await _syncFeed.PrepareRequest())!;
_feed.InitializeFeed();
BodiesSyncBatch req = (await _feed.PrepareRequest())!;

req.Response = new OwnedBlockBodies(req.Infos.Take(8).Select((info) =>
_syncingFromBlockTree.FindBlock(info!.BlockNumber, BlockTreeLookupOptions.None)!.Body).ToArray());
Expand All @@ -115,11 +126,53 @@ public async Task ShouldRecoverOnInsertFailure()
return true;
};

Func<SyncResponseHandlingResult> act = () => _syncFeed.HandleResponse(req);
Func<SyncResponseHandlingResult> act = () => _feed.HandleResponse(req);
act.Should().Throw<Exception>();

req = (await _syncFeed.PrepareRequest())!;
req = (await _feed.PrepareRequest())!;

req.Infos[0]!.BlockNumber.Should().Be(95);
}

[TestCase(100, false, null, false)]
[TestCase(11052930, false, null, true)]
[TestCase(11052984, false, null, true)]
[TestCase(11052985, false, null, false)]
[TestCase(100, false, 11052984, false)]
[TestCase(11052930, false, 11052984, true)]
[TestCase(11052984, false, 11052984, true)]
[TestCase(11052985, false, 11052984, false)]
[TestCase(100, true, null, false)]
[TestCase(11052930, true, null, false)]
[TestCase(11052984, true, null, false)]
[TestCase(11052985, true, null, false)]
[TestCase(100, false, 0, false)]
[TestCase(11052930, false, 0, false)]
[TestCase(11052984, false, 0, false)]
[TestCase(11052985, false, 0, false)]
public async Task When_finished_sync_with_old_default_barrier_then_finishes_imedietely(
long? lowestInsertedBlockNumber,
bool JustStarted,
long? previousBarrierInDb,
bool shouldfinish)
{
_syncConfig.AncientReceiptsBarrier = 0;
_syncingToBlockTree.LowestInsertedBodyNumber = JustStarted ? _pivotBlock.Number : _pivotBlock.Number - 1;
if (previousBarrierInDb != null)
_metadataDb.Set(MetadataDbKeys.BodiesBarrierWhenStarted, previousBarrierInDb.Value.ToBigEndianByteArrayWithoutLeadingZeros());
_feed.InitializeFeed();
_syncingToBlockTree.LowestInsertedBodyNumber = lowestInsertedBlockNumber;

BodiesSyncBatch? request = await _feed.PrepareRequest();
if (shouldfinish)
{
request.Should().BeNull();
_feed.CurrentState.Should().Be(SyncFeedState.Finished);
}
else
{
request.Should().NotBeNull();
_feed.CurrentState.Should().NotBe(SyncFeedState.Finished);
}
}
}
54 changes: 54 additions & 0 deletions src/Nethermind/Nethermind.Synchronization.Test/FastSync/ReceiptsSyncFeedTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,14 @@
using Nethermind.Blockchain;
using Nethermind.Blockchain.Receipts;
using Nethermind.Blockchain.Synchronization;
using Nethermind.Consensus;
using Nethermind.Core;
using Nethermind.Core.Crypto;
using Nethermind.Core.Extensions;
using Nethermind.Core.Specs;
using Nethermind.Core.Test;
using Nethermind.Core.Test.Builders;
using Nethermind.Db;
using Nethermind.Logging;
using Nethermind.Specs;
using Nethermind.Specs.Forks;
Expand Down Expand Up @@ -73,6 +77,7 @@ public Scenario(ISpecProvider specProvider, int nonEmptyBlocks, int txPerBlock,
private ISyncConfig _syncConfig = null!;
private ISyncReport _syncReport = null!;
private IBlockTree _blockTree = null!;
private IDb _metadataDb = null!;

private static readonly long _pivotNumber = 1024;

Expand All @@ -98,6 +103,7 @@ public void Setup()
{
_receiptStorage = Substitute.For<IReceiptStorage>();
_blockTree = Substitute.For<IBlockTree>();
_metadataDb = new TestMemDb();

_syncConfig = new SyncConfig { FastBlocks = true, FastSync = true };
_syncConfig.PivotNumber = _pivotNumber.ToString();
Expand All @@ -123,6 +129,7 @@ private ReceiptsSyncFeed CreateFeed()
_syncPeerPool,
_syncConfig,
_syncReport,
_metadataDb,
LimboLogs.Instance);
}

Expand All @@ -138,6 +145,7 @@ public void Should_throw_when_fast_blocks_not_enabled()
_syncPeerPool,
_syncConfig,
_syncReport,
_metadataDb,
LimboLogs.Instance));
}

Expand All @@ -151,6 +159,7 @@ public async Task Should_finish_on_start_when_receipts_not_stored()
_syncPeerPool,
_syncConfig,
_syncReport,
_metadataDb,
LimboLogs.Instance);
_feed.InitializeFeed();

Expand Down Expand Up @@ -226,6 +235,50 @@ public async Task When_configured_to_skip_receipts_then_finishes_immediately()
_measuredProgressQueue.HasEnded.Should().BeTrue();
}

[TestCase(100, false, null, false)]
[TestCase(11052930, false, null, true)]
[TestCase(11052984, false, null, true)]
[TestCase(11052985, false, null, false)]
[TestCase(100, false, 11052984, false)]
[TestCase(11052930, false, 11052984, true)]
[TestCase(11052984, false, 11052984, true)]
[TestCase(11052985, false, 11052984, false)]
[TestCase(100, true, null, false)]
[TestCase(11052930, true, null, false)]
[TestCase(11052984, true, null, false)]
[TestCase(11052985, true, null, false)]
[TestCase(100, false, 0, false)]
[TestCase(11052930, false, 0, false)]
[TestCase(11052984, false, 0, false)]
[TestCase(11052985, false, 0, false)]
public async Task When_finished_sync_with_old_default_barrier_then_finishes_imedietely(
long? lowestInsertedReceiptBlockNumber,
bool JustStarted,
long? previousBarrierInDb,
bool shouldfinish)
{
_syncConfig.AncientReceiptsBarrier = 0;
_receiptStorage.HasBlock(Arg.Is(_pivotNumber), Arg.Any<Hash256>()).Returns(!JustStarted);
if (previousBarrierInDb != null)
_metadataDb.Set(MetadataDbKeys.ReceiptsBarrierWhenStarted, previousBarrierInDb.Value.ToBigEndianByteArrayWithoutLeadingZeros());
LoadScenario(_256BodiesWithOneTxEach);
_receiptStorage.LowestInsertedReceiptBlockNumber.Returns(lowestInsertedReceiptBlockNumber);

ReceiptsSyncBatch? request = await _feed.PrepareRequest();
if (shouldfinish)
{
request.Should().BeNull();
_feed.CurrentState.Should().Be(SyncFeedState.Finished);
}
else
{
request.Should().NotBeNull();
_feed.CurrentState.Should().NotBe(SyncFeedState.Finished);
}
_measuredProgress.HasEnded.Should().Be(shouldfinish);
_measuredProgressQueue.HasEnded.Should().Be(shouldfinish);
}

private void LoadScenario(Scenario scenario)
{
LoadScenario(scenario, _syncConfig);
Expand All @@ -244,6 +297,7 @@ private void LoadScenario(Scenario scenario, ISyncConfig syncConfig)
_syncPeerPool,
_syncConfig,
_syncReport,
_metadataDb,
LimboLogs.Instance);
_feed.InitializeFeed();

Expand Down
2 changes: 2 additions & 0 deletions src/Nethermind/Nethermind.Synchronization.Test/ReceiptSyncFeedTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
using Nethermind.Blockchain.Synchronization;
using Nethermind.Core;
using Nethermind.Core.Test.Builders;
using Nethermind.Db;
using Nethermind.Logging;
using Nethermind.Specs;
using Nethermind.Synchronization.FastBlocks;
Expand Down Expand Up @@ -63,6 +64,7 @@ public async Task ShouldRecoverOnInsertFailure()
Substitute.For<ISyncPeerPool>(),
syncConfig,
new NullSyncReport(),
new MemDb(),
LimboLogs.Instance
);
syncFeed.InitializeFeed();
Expand Down
Loading
Loading