New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snapshot import #4291

Merged
merged 18 commits into from Sep 14, 2017

Conversation

Projects
None yet
4 participants
@gumb0
Member

gumb0 commented Jul 31, 2017

Importing Parity's warp snapshot

The way to try it:

  1. Download snapshot using this branch #4227 or using Parity (it puts it some dir like Parity/Ethereum/chains/ethereum/db/906a34e69aec8c0d/snapshot)
  2. ./eth --import-snapshot snapshot_dir -v 9
    (also setting database path with -d might be useful)
  3. After snapshot import is complete regular sync start automatically from the snapshot block

@gumb0 gumb0 changed the base branch from insert-without-parent to develop Aug 1, 2017

@chfast

This comment has been minimized.

Show comment
Hide comment
@chfast

chfast Aug 1, 2017

Collaborator

In general snappy external project looks ok, I will try to fix it tomorrow on linux.

However, we have snappy already because leveldb uses it. So on Linux/macOS we might try to using system level snappy (because we are using system level leveldb). I'm not sure if the Windows prebuilt dep has the snappy in it (check the script building it) and if it easy accessible (public API).

Collaborator

chfast commented Aug 1, 2017

In general snappy external project looks ok, I will try to fix it tomorrow on linux.

However, we have snappy already because leveldb uses it. So on Linux/macOS we might try to using system level snappy (because we are using system level leveldb). I'm not sure if the Windows prebuilt dep has the snappy in it (check the script building it) and if it easy accessible (public API).

@gumb0

This comment has been minimized.

Show comment
Hide comment
@gumb0

gumb0 Aug 3, 2017

Member

@chfast all builds are failing now

Member

gumb0 commented Aug 3, 2017

@chfast all builds are failing now

@gumb0

This comment has been minimized.

Show comment
Hide comment
@gumb0

gumb0 Aug 3, 2017

Member

@chfast Where's the script building windows leveldb?

Member

gumb0 commented Aug 3, 2017

@chfast Where's the script building windows leveldb?

@chfast

This comment has been minimized.

Show comment
Hide comment
@chfast

chfast Aug 3, 2017

Collaborator

https://github.com/ethereum/cpp-dependencies
3 .bat scripts with _leveldb suffix.

Collaborator

chfast commented Aug 3, 2017

https://github.com/ethereum/cpp-dependencies
3 .bat scripts with _leveldb suffix.

@codecov-io

This comment has been minimized.

Show comment
Hide comment
@codecov-io

codecov-io Aug 17, 2017

Codecov Report

Merging #4291 into develop will increase coverage by 0.5%.
The diff coverage is 93.15%.

Impacted file tree graph

@@            Coverage Diff             @@
##           develop    #4291     +/-   ##
==========================================
+ Coverage    69.65%   70.16%   +0.5%     
==========================================
  Files          308      314      +6     
  Lines        23345    23711    +366     
==========================================
+ Hits         16262    16637    +375     
+ Misses        7083     7074      -9
Impacted Files Coverage Δ
libethereum/Client.h 68.42% <ø> (ø) ⬆️
libdevcore/TrieDB.h 94.05% <ø> (ø) ⬆️
libethereum/BlockChainImporter.h 100% <100%> (ø)
...est/unittests/libethereum/SnapshotImporterTest.cpp 100% <100%> (ø)
libethereum/SnapshotStorage.h 28.57% <28.57%> (ø)
libethereum/SnapshotImporter.h 33.33% <33.33%> (ø)
libethereum/StateImporter.h 50% <50%> (ø)
libethereum/SnapshotImporter.cpp 88.37% <88.37%> (ø)
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1db05b...ac71a1f. Read the comment docs.

codecov-io commented Aug 17, 2017

Codecov Report

Merging #4291 into develop will increase coverage by 0.5%.
The diff coverage is 93.15%.

Impacted file tree graph

@@            Coverage Diff             @@
##           develop    #4291     +/-   ##
==========================================
+ Coverage    69.65%   70.16%   +0.5%     
==========================================
  Files          308      314      +6     
  Lines        23345    23711    +366     
==========================================
+ Hits         16262    16637    +375     
+ Misses        7083     7074      -9
Impacted Files Coverage Δ
libethereum/Client.h 68.42% <ø> (ø) ⬆️
libdevcore/TrieDB.h 94.05% <ø> (ø) ⬆️
libethereum/BlockChainImporter.h 100% <100%> (ø)
...est/unittests/libethereum/SnapshotImporterTest.cpp 100% <100%> (ø)
libethereum/SnapshotStorage.h 28.57% <28.57%> (ø)
libethereum/SnapshotImporter.h 33.33% <33.33%> (ø)
libethereum/StateImporter.h 50% <50%> (ø)
libethereum/SnapshotImporter.cpp 88.37% <88.37%> (ø)
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1db05b...ac71a1f. Read the comment docs.

@gumb0 gumb0 changed the title from WIP - Snapshot import to Snapshot import Sep 5, 2017

@gumb0 gumb0 added the needs review label Sep 5, 2017

@gumb0

This comment has been minimized.

Show comment
Hide comment
@gumb0

gumb0 Sep 5, 2017

Member

I think this can be reviewed now. I will work on adding some unit tests.

Member

gumb0 commented Sep 5, 2017

I think this can be reviewed now. I will work on adding some unit tests.

@gumb0 gumb0 requested review from chfast and chriseth Sep 12, 2017

@chfast chfast removed their assignment Sep 13, 2017

Show outdated Hide outdated libethereum/BlockChainImporter.h
Show outdated Hide outdated libethereum/BlockChainImporter.cpp
Show outdated Hide outdated libethereum/SnapshotStorage.cpp
Show outdated Hide outdated libethereum/SnapshotStorage.cpp
struct SnapshotImportLog: public LogChannel
{
static char const* name() { return "SNAP"; }
static int const verbosity = 9;

This comment has been minimized.

@chfast

chfast Sep 13, 2017

Collaborator

Were you able to use it for debugging? I find our logging system without filters quite useless.

@chfast

chfast Sep 13, 2017

Collaborator

Were you able to use it for debugging? I find our logging system without filters quite useless.

This comment has been minimized.

@gumb0

gumb0 Sep 13, 2017

Member

Yeah. On Windows it somehow looks more clear, at least in release build and I can see snapshot importing progress. On Linux there's more spam from somewhere (probably BlockChain)

@gumb0

gumb0 Sep 13, 2017

Member

Yeah. On Windows it somehow looks more clear, at least in release build and I can see snapshot importing progress. On Linux there's more spam from somewhere (probably BlockChain)

@chfast

chfast approved these changes Sep 13, 2017

Awesome!

@gumb0 gumb0 merged commit 3c49a02 into develop Sep 14, 2017

1 of 3 checks passed

ci/circleci No test commands were found
Details
continuous-integration/appveyor/pr AppVeyor build failed
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@chriseth chriseth removed the needs review label Sep 14, 2017

@gumb0 gumb0 deleted the snapshot-import branch Sep 14, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment