Warp-only sync with warp-barrier [blocknumber] flag. #8228

tomusdrw · 2018-03-26T17:16:15Z

Enables warp-only sync in case --warp-after <block-number> is provided.

This allows the user to ignore old snapshots floating around and also makes sure that you are actually warp syncing.
After that block is reached we fall back to regular sync.

Resolves #8097

tomusdrw · 2018-03-26T17:18:05Z

sync/src/chain.rs

@@ -658,6 +668,10 @@ impl ChainSync {
 			return Ok(());
 		}

+		if self.sync_start_time.is_none() {


Moved that after the initial checks, otherwise in case we are not using warp-only sync, we might timeout a bit too early (I actually got a lot of peers on different chains from discovery), without even giving a chance to start warping.

folsen · 2018-03-27T08:35:52Z

LGTM, but I think we'll need some guidance on how to choose the block number. If people go on etherscan and pick the highest, they'll have to wait a long time until they find a snapshot for that, if they go back 100 blocks (1000 blocks?), they're more likely to find a snapshot quickly and can sync from there. Just a one-liner recommendation would be good.

5chdn · 2018-03-27T10:26:44Z

Can we just set a default to 5 million? This would partially address Fred's comment as users do not really need to care about this feature unless they really want to fine-tune parity.

jamesray1 · 2018-03-27T10:43:13Z

The problem with setting it to 5 million is that it will then gradually get longer to sync from there, so you'd have to manually update the default every so often. If you go back 5000 blocks that should be enough to guarantee that nearly all peers (except for those that don't take snapshots) would have taken a snapshot.

andresilva

LGTM!

tomusdrw · 2018-03-27T11:48:43Z

@5chdn I'm more in favour of (optionally) storing network start date and average block time in the spec file and compute default value from this, at least it's always green.

tomusdrw · 2018-03-28T10:12:48Z

Voting for a better CLI name:

👍 --warp-at-least-to/--warp-to <bn>
😄 --warp-until <bn>
🎉 --warp-beyond <bn>
😕 --warp-after <bn>
❤️ --warp-barrier <bn>

CC @bjornwgnr @Tbaut @5chdn @debris

5chdn · 2018-03-28T10:18:16Z

--warp-barrier ❤️

http://memory-alpha.wikia.com/wiki/Warp_barrier

--warp-threshold 👎

andresilva · 2018-03-28T10:20:38Z

I vote for --warp-to or --warp-until (I know these don't convey the meaning that the warp might actually be to after the given block).

jamesray1 · 2018-03-28T10:35:12Z

After that block is reached we fall back to regular sync.

This suggests that warp-at-least-to is most appropriate. I dislike relying on voting only as it encourages mob rule, I prefer rough consensus. If someone has a better reason to go with another option, I'm all ears.

Tbaut · 2018-03-28T10:58:06Z

👍 --warp-at-least-to/--warp-to <bn>
would be my favorit but is too long.. so I voted for
❤️ --warp-barrier <bn>

What about
--min-wrap-block ?

5chdn · 2018-03-28T11:40:07Z

@5chdn I'm more in favour of (optionally) storing network start date and average block time in the spec file and compute default value from this, at least it's always green.

Of course, this must be a chain-dependent default, so chain-spec is an idea.

jamesray1 · 2018-03-28T13:29:39Z

To convey the meaning more shortly than --warp-at-least-to there's --warp-to-min, or --warp2min if you want to be really lazy.

jamesray1 · 2018-04-03T11:38:15Z

I got errors when building from source:

   Compiling tokio-rustls v0.4.0
   Compiling node-health v0.1.0 (file:///home/james/parity/dapps/node-health)
error[E0015]: calls in constants are limited to struct and enum constructors
  --> dapps/node-health/src/health.rs:29:27
   |
29 | const TIMEOUT: Duration = Duration::from_secs(5);
   |                           ^^^^^^^^^^^^^^^^^^^^^^
   |
note: a limited form of compile-time function evaluation is available on a nightly compiler via `const fn`
  --> dapps/node-health/src/health.rs:29:27
   |
29 | const TIMEOUT: Duration = Duration::from_secs(5);
   |                           ^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: Could not compile `node-health`.
warning: build failed, waiting for other jobs to finish...
error: build failed

Tbaut · 2018-04-03T11:43:06Z

I compiled master this morning without problem, did you do rustup update then cargo clean && cargo build --release ?

andresilva · 2018-04-03T11:44:12Z

@jamesray1 Update your rust compiler to 1.25, since those methods on Duration were made const.

jamesray1 · 2018-04-03T12:04:26Z

OK, updated on my desktop, running cargo clean && cargo build --release now.

jamesray1 · 2018-04-03T12:05:08Z

On my laptop:

error: Could not compile `tiny-keccak`.
warning: build failed, waiting for other jobs to finish...
error: build failed
james@james-Latitude-E6410:~/parity$ rustup update
rustup: command not found
james@james-Latitude-E6410:~/parity$ curl https://sh.rustup.rs -sSf | sh
info: downloading installer
error: it looks like you have an existing installation of Rust at:
error: /usr/bin
error: rustup cannot be installed alongside Rust. Please uninstall first
error: if this is what you want, restart the installation with `-y'
error: cannot install while Rust is installed

Sorry this is kind of basic, but is there a better way to update than uninstalling Rust then installing Rustup?

5chdn · 2018-04-03T12:05:57Z

Which rust version?

jamesray1 · 2018-04-03T12:08:25Z

rustc 1.22.1

jamesray1 · 2018-04-03T12:10:31Z

I'll try curl -sf -L https://static.rust-lang.org/rustup.sh | sh as per https://www.reddit.com/r/rust/comments/3gppup/how_do_i_update_rust_to_the_latest_version_on/.

5chdn · 2018-04-03T12:30:51Z

just rustup update 1.25 is latest

jamesray1 · 2018-04-03T12:45:04Z

OK so it built on the desktop: Finished release [optimized] target(s) in 1847.93 secs.

jamesray1 · 2018-04-03T13:27:29Z

On the laptop, the first try included:
Internal error occurred: Failed to find tool. Is c++ installed?

So I ran sudo apt-get install build-essential.

Then running cargo clean && cargo build --release again gave:

error: failed to run custom build command for `hidapi v0.3.1 (https://github.com/paritytech/hidapi-rs#e77ea09c)`
process didn't exit successfully: `/home/james/parity/target/release/build/hidapi-585e15f77feb3102/build-script-build` (exit code: 101)
--- stdout
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CC_x86_64-unknown-linux-gnu = None
CC_x86_64_unknown_linux_gnu = None
HOST_CC = None
CC = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CFLAGS_x86_64-unknown-linux-gnu = None
CFLAGS_x86_64_unknown_linux_gnu = None
HOST_CFLAGS = None
CFLAGS = None
DEBUG = Some("false")
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "etc/hidapi/hidapi" "-Wall" "-Wextra" "-o" "/home/james/parity/target/release/build/hidapi-4f244163ae0b978b/out/etc/hidapi/linux/hid.o" "-c" "etc/hidapi/linux/hid.c"
cargo:warning=etc/hidapi/linux/hid.c:44:10: fatal error: libudev.h: No such file or directory
cargo:warning= #include <libudev.h>
cargo:warning=          ^~~~~~~~~~~
cargo:warning=compilation terminated.
exit code: 1

--- stderr
thread 'main' panicked at '

Internal error occurred: Command "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "etc/hidapi/hidapi" "-Wall" "-Wextra" "-o" "/home/james/parity/target/release/build/hidapi-4f244163ae0b978b/out/etc/hidapi/linux/hid.o" "-c" "etc/hidapi/linux/hid.c" with args "cc" did not execute successfully (status code exit code: 1).

', /home/james/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.4/src/lib.rs:1984:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

RUST_BACKTRACE=1 cargo clean && cargo build --release
<snip compiles>
error: failed to run custom build command for `hidapi v0.3.1 (https://github.com/paritytech/hidapi-rs#e77ea09c)`
process didn't exit successfully: `/home/james/parity/target/release/build/hidapi-585e15f77feb3102/build-script-build` (exit code: 101)
--- stdout
TARGET = Some("x86_64-unknown-linux-gnu")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CC_x86_64-unknown-linux-gnu = None
CC_x86_64_unknown_linux_gnu = None
HOST_CC = None
CC = None
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-unknown-linux-gnu")
CFLAGS_x86_64-unknown-linux-gnu = None
CFLAGS_x86_64_unknown_linux_gnu = None
HOST_CFLAGS = None
CFLAGS = None
DEBUG = Some("false")
running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "etc/hidapi/hidapi" "-Wall" "-Wextra" "-o" "/home/james/parity/target/release/build/hidapi-4f244163ae0b978b/out/etc/hidapi/linux/hid.o" "-c" "etc/hidapi/linux/hid.c"
cargo:warning=etc/hidapi/linux/hid.c:44:10: fatal error: libudev.h: No such file or directory

cargo:warning= #include <libudev.h>
cargo:warning=          ^~~~~~~~~~~
cargo:warning=compilation terminated.
exit code: 1

--- stderr
thread 'main' panicked at '

Internal error occurred: Command "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "etc/hidapi/hidapi" "-Wall" "-Wextra" "-o" "/home/james/parity/target/release/build/hidapi-4f244163ae0b978b/out/etc/hidapi/linux/hid.o" "-c" "etc/hidapi/linux/hid.c" with args "cc" did not execute successfully (status code exit code: 1).

', /home/james/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.4/src/lib.rs:1984:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

andresilva · 2018-04-03T13:31:29Z

I think you're missing some dependencies, it's documented here: https://wiki.parity.io/Setup#dependencies.

apt-get install build-essential openssl libssl-dev libudev-dev

jamesray1 · 2018-04-03T13:35:19Z

@5chdn as explained above rustup update gives command not found. Also after running curl -sf -L https://static.rust-lang.org/rustup.sh | sh the version is still the same: 1.22.1.

jamesray1 · 2018-04-03T13:37:43Z

I'll try uninstalling Rust and reinstalling.

jamesray1 · 2018-04-03T13:54:06Z

I ran sudo /usr/local/lib/rustlib/uninstall.sh, and removing rustc, rustdoc and rust-gdb in /usr/bin but still I get:

james@james-Latitude-E6410:~$ curl https://sh.rustup.rs -sSf | sh
info: downloading installer
error: it looks like you have an existing installation of Rust at:
error: /usr/bin
error: rustup cannot be installed alongside Rust. Please uninstall first
error: if this is what you want, restart the installation with `-y'
error: cannot install while Rust is installed

Interestingly:

james@james-Latitude-E6410:~$ rustc -V
The program 'rustc' is currently not installed. You can install it by typing:
sudo apt install rustc

5chdn · 2018-04-03T13:56:28Z

Please keep the Parity issue tracker on topic.

jamesray1 · 2018-04-03T23:07:24Z

Fixed: removed /usr/cargo/bin.

Overnight on the desktop, running ./target/release/parity --warp-barrier 5362706 (parity was on the old version, I've fixed that now) didn't work, it synced 1165 snapshots and has gone to #4263546, from #4255872.

Tbaut · 2018-04-04T07:16:07Z

5362706 is a bit optimistic. Could you share the logs (toward the end of the snapshot sync and the beginning of the blocks sync)?

5chdn · 2018-04-04T09:54:08Z

1155 snapshots take you directly to 5_370_000, just reset your db and try again

jamesray1 · 2018-04-05T01:26:42Z

Moving this to Gitter: https://gitter.im/paritytech/parity?at=5ac57b411130fe3d36a26b0a.

how do I reset the database? I searched on the wiki and looked at parity --help. Is there a way to reset it without deleting it?

Actually for the latest command I didn't set --log-file.

https://gist.github.com/jamesray1/74fa10b3f9f78f2c3c9ace21ada317bb

jamesray1 · 2018-04-05T01:27:58Z

Actually for the latest command I didn't set --log-file.

5chdn · 2018-04-05T08:29:00Z

reset = delete

roninkaizen · 2018-04-09T08:49:57Z

thx to afri about the menioning of rust version 1.25
worked once for me-
a detailled mention within the wiki would be nice as well @Tbaut

* Warp-only sync with warp-after [blocknumber] flag. * Fix tests. * Fix configuration tests. * Rename to warp barrier.

* Warp-only sync with warp-barrier [blocknumber] flag. (#8228) * Warp-only sync with warp-after [blocknumber] flag. * Fix tests. * Fix configuration tests. * Rename to warp barrier. * Allow unsafe js eval on Parity Wallet. (#8204) * Update musicoin spec in line with gmc v2.6.2 (#8242) * Supress TemporaryInvalid verification failures. (#8256) * Include suicided accounts in state diff (#8297) * Include suicided accounts in state diff * Shorten form match -> if let * Test suicide trace diff in State * replace_home for password_files, reserved_peers and log_file (#8324) * replace_home for password_files, reserved_peers and log_file * typo: arg_log_file is Option * Enable UI by default, but only display info page. * Fix test. * Fix naming and remove old todo. * Change "wallet" with "browser UI"

tomusdrw added A0-pleasereview 🤓 Pull request needs code review. M4-core ⛓ Core client code / Rust. labels Mar 26, 2018

tomusdrw commented Mar 26, 2018

View reviewed changes

5chdn added this to the 1.11 milestone Mar 27, 2018

5chdn added the B7-releasenotes 📜 Changes should be mentioned in the release notes of the next minor version release. label Mar 27, 2018

5chdn mentioned this pull request Mar 27, 2018

PR draft: faster sync #8097

Closed

andresilva approved these changes Mar 27, 2018

View reviewed changes

tomusdrw changed the title ~~Warp-only sync with warp-after [blocknumber] flag.~~ Warp-only sync with warp-barrier [blocknumber] flag. Mar 28, 2018

tomusdrw added 4 commits March 28, 2018 18:00

Warp-only sync with warp-after [blocknumber] flag.

489cc91

Fix tests.

e835393

Fix configuration tests.

cd3f8db

Rename to warp barrier.

678f18c

tomusdrw force-pushed the td-warp-only branch from a718906 to 678f18c Compare March 28, 2018 16:00

5chdn merged commit 899c1a4 into master Mar 29, 2018

5chdn deleted the td-warp-only branch March 29, 2018 09:20

5chdn added the B0-patch label Mar 29, 2018

Tbaut mentioned this pull request Apr 3, 2018

Snapshot sync doesn't work in most cases #8290

Closed

tomusdrw added a commit that referenced this pull request Apr 10, 2018

Warp-only sync with warp-barrier [blocknumber] flag. (#8228)

71ffb44

* Warp-only sync with warp-after [blocknumber] flag. * Fix tests. * Fix configuration tests. * Rename to warp barrier.

tomusdrw mentioned this pull request Apr 10, 2018

[beta] Backports #8346

Merged

5chdn added A8-looksgood 🦄 Pull request is reviewed well. and removed A0-pleasereview 🤓 Pull request needs code review. labels May 16, 2018

Warp-only sync with warp-barrier [blocknumber] flag. #8228

Warp-only sync with warp-barrier [blocknumber] flag. #8228

Conversation

tomusdrw commented Mar 26, 2018

tomusdrw Mar 26, 2018

Choose a reason for hiding this comment

folsen commented Mar 27, 2018

5chdn commented Mar 27, 2018

jamesray1 commented Mar 27, 2018

andresilva left a comment

Choose a reason for hiding this comment

tomusdrw commented Mar 27, 2018

tomusdrw commented Mar 28, 2018 • edited

5chdn commented Mar 28, 2018 • edited

andresilva commented Mar 28, 2018 • edited

jamesray1 commented Mar 28, 2018 • edited

Tbaut commented Mar 28, 2018

5chdn commented Mar 28, 2018

jamesray1 commented Mar 28, 2018 • edited

jamesray1 commented Apr 3, 2018

Tbaut commented Apr 3, 2018

andresilva commented Apr 3, 2018

jamesray1 commented Apr 3, 2018

jamesray1 commented Apr 3, 2018 • edited

5chdn commented Apr 3, 2018

jamesray1 commented Apr 3, 2018

jamesray1 commented Apr 3, 2018

5chdn commented Apr 3, 2018

jamesray1 commented Apr 3, 2018

jamesray1 commented Apr 3, 2018 • edited

andresilva commented Apr 3, 2018

jamesray1 commented Apr 3, 2018

jamesray1 commented Apr 3, 2018

jamesray1 commented Apr 3, 2018 • edited

5chdn commented Apr 3, 2018

jamesray1 commented Apr 3, 2018 • edited

Tbaut commented Apr 4, 2018

5chdn commented Apr 4, 2018

jamesray1 commented Apr 5, 2018 • edited

jamesray1 commented Apr 5, 2018

5chdn commented Apr 5, 2018

roninkaizen commented Apr 9, 2018

tomusdrw commented Mar 28, 2018 •

edited

5chdn commented Mar 28, 2018 •

edited

andresilva commented Mar 28, 2018 •

edited

jamesray1 commented Mar 28, 2018 •

edited

jamesray1 commented Mar 28, 2018 •

edited

jamesray1 commented Apr 3, 2018 •

edited

jamesray1 commented Apr 3, 2018 •

edited

jamesray1 commented Apr 3, 2018 •

edited

jamesray1 commented Apr 3, 2018 •

edited

jamesray1 commented Apr 5, 2018 •

edited