ERA history import/export start #6173

ak88 · 2023-10-09T14:38:06Z

R#eceipt decoder

Fixes Closes Resolves #

Please choose one of the keywords above to refer to the issue this PR solves followed by the issue number (e.g. Fixes #000). If no issue number, remove the line. Also, remove everything marked optional that is not applicable. Remove this note after reading.

Changes

List the changes

Types of changes

What types of changes does your code introduce?

Bugfix (a non-breaking change that fixes an issue)
New feature (a non-breaking change that adds functionality)
Breaking change (a change that causes existing functionality not to work as expected)
Optimization
Refactoring
Documentation update
Build-related changes
Other: Description

Testing

Requires testing

Yes
No

If yes, did you write tests?

Yes
No

Notes on testing

Optional. Remove if not applicable.

Documentation

Requires documentation update

Yes
No

If yes, link the PR to the docs update or the issue with the details labeled docs. Remove if not applicable.

Requires explanation in Release Notes

Yes
No

If yes, fill in the details here. Remove if not applicable.

Remarks

!!Work in progress!!

R#eceipt decoder

MarekM25 · 2023-10-10T10:40:51Z

src/Nethermind/Nethermind.GitBook/docs

it shouldn't be updated in this PR, please revert

MarekM25 · 2023-10-10T10:44:14Z

src/Nethermind/Nethermind.Serialization.Rlp/BlockBodyDecoder.cs

+
+namespace Nethermind.Serialization.Rlp
+{
+    public class BlockBodyDecoder : IRlpValueDecoder<BlockBody>, IRlpStreamDecoder<BlockBody>


What is the difference between these decoders and existing one?

I will refactor this.

You could use it in BlockDecoder, also we have a BlockBodyDecoder in the networking part, maybe they are similar.

src/Nethermind/Nethermind.Era1/E2Store.cs

LukaszRozmej · 2023-10-12T15:23:38Z

Related to: ethereum/go-ethereum#26621

into feature/era-history

Test import export unit tests

commit 68c722cad69d082da7e6283c396b4f5706a3540f Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Sun Nov 5 23:38:52 2023 +0100 Verify accumulator commit c48ca9de6ce767173b9674b665ac7ecd0f2e7172 Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Sun Nov 5 15:14:37 2023 +0100 refactor EraReader

ak88 · 2023-11-06T13:02:16Z

src/Nethermind/Nethermind.Era1/EraService.cs

This is not finished. This will be used when integrating with Node CLI.

LukaszRozmej

Some initial comments

LukaszRozmej · 2023-11-07T10:00:06Z

src/Nethermind/Directory.Packages.props

@@ -13,6 +13,7 @@
    <PackageVersion Include="Colorful.Console" Version="1.2.15" />
    <PackageVersion Include="CommandLineParser" Version="2.9.1" />
    <PackageVersion Include="ConcurrentHashSet" Version="1.3.0" />
+    <PackageVersion Include="Cortex.SimpleSerialize" Version="0.2.0" />


Not newer Nethermind.Serialization.Ssz?
@flcl42 Can you advise?

LukaszRozmej · 2023-11-07T10:04:12Z

src/Nethermind/Directory.Packages.props

@@ -68,6 +69,7 @@
    <PackageVersion Include="Seq.Api" Version="2023.3.0" />
    <PackageVersion Include="SharpCompress" Version="0.34.1" />
    <PackageVersion Include="Shouldly" Version="4.2.1" />
+    <PackageVersion Include="Snappier" Version="1.1.1" />


Is it faster or better than then already used Snappy.Standard?

The problem is that era1 uses framing format which the existing lib does not support

src/Nethermind/Nethermind.Core.Test/Builders/TestItem.cs

src/Nethermind/Nethermind.Serialization.Rlp/RlpDecoderExtensions.cs

src/Nethermind/Nethermind.Serialization.Rlp/ReceiptMessageDecoder.cs

LukaszRozmej · 2023-11-07T10:26:03Z

src/Nethermind/Nethermind.Era1/E2Store.cs

+                using SnappyStream snappyStream = new(memoryStream, CompressionMode.Compress, true);
+                await snappyStream.WriteAsync(bytes, cancellation);
+                await snappyStream.FlushAsync();
+                bytes = memoryStream.ToArray();


lot of allocations

src/Nethermind/Nethermind.Era1/E2Store.cs

LukaszRozmej · 2023-11-07T10:30:43Z

src/Nethermind/Nethermind.Era1/E2Store.cs

+        var buf = ArrayPool<byte>.Shared.Rent(HeaderSize);
+        try
+        {
+            _stream!.Position = offset;


Are we doing a lot of jumping/seeking on the stream or is it more of linear scans?

Every iteration will read the offset from the block index, which is appended to the file.
That will probably trigger a lot of random reads.
I thought about loading the whole index instead at the expense of memory WDYT?

commit 836b4aa29eb31f8b903696eef4edfa3f96fb7a05 Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Thu Nov 9 13:35:34 2023 +0100 in params commit 4c7104adfdfc8cf485b7bfe5b23a02e61daeb98c Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Thu Nov 9 13:15:50 2023 +0100 update snappier commit 6d97e87a211f559d1263a4f7a6caf97977a6d25d Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Thu Nov 9 12:57:11 2023 +0100 small optimization commit 8d9e39b0f85eb23bf05b1b7cc47acafe7f684108 Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Wed Nov 8 15:46:44 2023 +0100 format commit d47b9839866676feff5ae2352af793f698f4cd97 Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Wed Nov 8 15:41:57 2023 +0100 changes for PR commit 5c4614c2f68bf104220451d4c945579adfe32575 Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Wed Nov 8 00:28:58 2023 +0100 changes for PR

commit e4bbfed2577d4189637c86a92e6d2848820701fa Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Thu Nov 9 22:57:06 2023 +0100 load bock index commit 9e65cd99544d97610dd96fe5fd406e40be792455 Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Thu Nov 9 22:05:48 2023 +0100 small change commit 8d207c6a7fc0e55f375fd4f169d04f0fa5ac85f7 Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Thu Nov 9 21:43:37 2023 +0100 load whole index commit 126cc0c4ad42dae97a9fdf3951c3e670b71ad2b3 Author: Anders Holmbjerg Kristiansen <anders.holmbjerg@hotmail.com> Date: Thu Nov 9 16:37:38 2023 +0100 small fix

into feature/era-history

smartprogrammer93 · 2024-01-16T13:11:18Z

Closing in favor of #6547

ERA history import/export start

301180c

R#eceipt decoder

ak88 force-pushed the feature/era-history branch from ca6986e to 301180c Compare October 9, 2023 14:43

ak88 added 2 commits October 9, 2023 16:44

Rename in unit test

5a0b8c4

Fix block index not set

04209fb

MarekM25 reviewed Oct 10, 2023

View reviewed changes

src/Nethermind/Nethermind.GitBook/docs Outdated

Copy link

Contributor

MarekM25 Oct 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it shouldn't be updated in this PR, please revert

MarekM25 reviewed Oct 10, 2023

View reviewed changes

Start export

4b10a61

LukaszRozmej reviewed Oct 12, 2023

View reviewed changes

src/Nethermind/Nethermind.Era1/E2Store.cs Show resolved Hide resolved

ak88 added 21 commits October 13, 2023 16:20

Era builder

bfcbfb5

Accumulator calculation

e9d91b6

ERA history import/export

7b88e17

Rename in unit test

1543a39

Fix block index not set

e07cca5

Start export

42131e6

Era builder

551d974

Accumulator calculation

aba6cfb

Builder finalize

65a9225

Merge branch 'feature/era-history' of https://github.com/ak88/nethermind

d0741e5

into feature/era-history

Unit tests

508f1ae

Test import export unit tests

Unit tests

c7636c2

import geth files

ee53da7

Reader with netty

8503f08

1mb read buffer

66eacd1

refactor test and use of netty

8d48264

Big blocks test

904ecab

Change to IByteBuffer

74d14ad

buffer allocater injected

e1e86ae

Merge branch 'NettyBuffers' into feature/era-history

d8f4ad8

ak88 added 9 commits November 6, 2023 00:49

Merge from master

27fc5cd

assert files exist

756b170

reverted

e3a6647

rollback gitbook

f18e742

Merge branch 'master' into feature/era-history

7066e64

revert gitbook

d2656d8

revert launchsettings

5d43047

Delete from this pr

dedba82

format whitespace

f2f03db

ak88 commented Nov 6, 2023

View reviewed changes

src/Nethermind/Nethermind.Era1/EraService.cs

Copy link

Contributor Author

ak88 Nov 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not finished. This will be used when integrating with Node CLI.

ak88 marked this pull request as ready for review November 6, 2023 13:02

small fix

58f8a06

LukaszRozmej reviewed Nov 7, 2023

View reviewed changes

ak88 added 10 commits November 7, 2023 17:49

accumulator test

f089911

ReceiptMessageDecoder cleanup

ce691f0

no seeking to index

c3e6df8

reuse compression streams

c0640ed

format

85dfb85

Performance optimization

6330162

avoid a copy and dispose on ex

3fed302

Merge branch 'feature/era-history' of https://github.com/ak88/nethermind

1900b69

into feature/era-history

ak88 closed this Jan 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ERA history import/export start #6173

ERA history import/export start #6173

ak88 commented Oct 9, 2023

MarekM25 Oct 10, 2023

MarekM25 Oct 10, 2023

ak88 Nov 6, 2023

LukaszRozmej Nov 7, 2023

LukaszRozmej commented Oct 12, 2023

ak88 Nov 6, 2023

LukaszRozmej left a comment

LukaszRozmej Nov 7, 2023

LukaszRozmej Nov 7, 2023

ak88 Nov 7, 2023 •

edited

LukaszRozmej Nov 7, 2023

LukaszRozmej Nov 7, 2023

ak88 Nov 7, 2023

smartprogrammer93 commented Jan 16, 2024

ERA history import/export start #6173

ERA history import/export start #6173

Conversation

ak88 commented Oct 9, 2023

Changes

Types of changes

What types of changes does your code introduce?

Testing

Requires testing

If yes, did you write tests?

Notes on testing

Documentation

Requires documentation update

Requires explanation in Release Notes

Remarks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LukaszRozmej commented Oct 12, 2023

Choose a reason for hiding this comment

LukaszRozmej left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ak88 Nov 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smartprogrammer93 commented Jan 16, 2024

ak88 Nov 7, 2023 •

edited