New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catch up filter headers and block headers in parallel #98

Merged
merged 7 commits into from Oct 3, 2018

Conversation

Projects
None yet
4 participants
@halseth
Contributor

halseth commented Sep 17, 2018

Catch up filter headers and block headers in parallel.

Also includes a fix for rollback of filter headers, and a proper fix to what was attempted in #95.

@coveralls

This comment has been minimized.

coveralls commented Sep 17, 2018

Coverage Status

Coverage decreased (-1.02%) to 69.478% when pulling ee90510 on halseth:parallel-header-catchup into 29105cb on lightninglabs:master.

Show resolved Hide resolved blockmanager.go Outdated
Show resolved Hide resolved blockmanager.go Outdated

@halseth halseth referenced this pull request Sep 20, 2018

Closed

Update dependencies #1947

5 of 5 tasks complete

@halseth halseth force-pushed the halseth:parallel-header-catchup branch 3 times, most recently from 90bea83 to fa55fcf Sep 20, 2018

@halseth halseth changed the title from Proof-of-concept: Catch up filter headers and block headers in parallel to Catch up filter headers and block headers in parallel Sep 20, 2018

b.newHeadersSignal.L.Lock()
for !b.IsCurrent() {
for !(b.IsCurrent() || b.filterHeaderTip+wire.CFCheckptInterval <= b.headerTip) {

This comment has been minimized.

@cfromknecht

cfromknecht Sep 21, 2018

Contributor

already discussed irl, but it might be worth reversing the order (again) to avoid db hits amap

This comment has been minimized.

@halseth

halseth Sep 21, 2018

Contributor

reversed.

@halseth halseth force-pushed the halseth:parallel-header-catchup branch from fa55fcf to c08b16e Sep 21, 2018

@halseth

This comment has been minimized.

Contributor

halseth commented Sep 21, 2018

Now waits 100 CF intervalse before starting catch up.

@halseth halseth force-pushed the halseth:parallel-header-catchup branch 3 times, most recently from 6aa7b26 to 973863b Sep 24, 2018

@halseth

This comment has been minimized.

Contributor

halseth commented Sep 24, 2018

Reverted back to waiting only one interval. Instead the latest commit will fetch the filter checkpoints up to the latest block checkpoint, avoiding doing this fetch every time the block headers advance. This means that we will reuse these checkpoints until blockeaders > chain_checkpoint, at which point we will fetch new checkpoints.

@@ -225,7 +226,14 @@ func newBlockManager(s *ChainService) (*blockManager, error) {
if err != nil {
return nil, err
}
bm.filterHeaderTipHash = header.BlockHash()
// We must also ensure the the filter header tip hash is set to the

This comment has been minimized.

@Roasbeef
Show resolved Hide resolved blockmanager.go Outdated
Show resolved Hide resolved blockmanager.go
Show resolved Hide resolved blockmanager.go Outdated
Show resolved Hide resolved blockmanager.go
@Roasbeef

This comment has been minimized.

Member

Roasbeef commented Sep 26, 2018

Tested this locally with multiple syncs from both a remote and local node, didn't run into any major issues. Ensured that it was able to proceed after forced restarts, and seemed to handle the started no problem. However, I think there're a few places where a quit signal isn't being properly threaded through, as I noticed after some attempts, I was unable to kill the process.

@halseth

This comment has been minimized.

Contributor

halseth commented Sep 26, 2018

When do you experience you are unable to kill the process?

Haven't been able to reproduce it myself, but a hunch I had was that one of the loops where we are waiting for the newHeadersSignal would stall. However, in the block manager stop method we will repeatedly signal until all goroutines are shut down, so it should be handling that case.

This PR shouldn't change the quit behaviour in any way, but a goroutine dump in the unable-to-kill case should expose the culprit.

@Roasbeef

LGTM 🥑

Ready to go in after a rebase!

@halseth halseth force-pushed the halseth:parallel-header-catchup branch from 3bd50c0 to 1c339ed Sep 30, 2018

halseth added some commits Sep 30, 2018

blockmanager: remove filter goroutines.
Since we are only handling one filter type, we can simplify the code by
making it clearly synchronous.
blockmanager: fetch filter checkpoints up to chain checkpoints
This commit makes the fetching of filter checkpoints go up to the hash
of the latest chain checkpoints in case the block headers haven't caught
up that far.

This helps in terms of avoiding doing a lot of filter checkpoint fetches
up to small heights while the block headers are catching up.
blockmanager: start cfheader sync immediately when lagging a cp inter…
…val behind

This commit makes the cfhandler start fetching checkpointed filter
headers imemdiately when they are lagging at least a CF checkpoint
interval behind the blockheaders.

@halseth halseth force-pushed the halseth:parallel-header-catchup branch from 1c339ed to 3fb5213 Sep 30, 2018

@halseth

This comment has been minimized.

Contributor

halseth commented Oct 1, 2018

Rebased.

@cfromknecht

This comment has been minimized.

Contributor

cfromknecht commented Oct 1, 2018

@halseth perhaps the source of the inability to quit while syncing is a result of using Sleep? Might suggest selecting on time.After and quit to ensure we can exit

@halseth

This comment has been minimized.

Contributor

halseth commented Oct 1, 2018

I think it should successfully quit after the sleep timeout is over, but I agree that's cleaner. Will add :)

@Roasbeef

LGTM 🕺

@Roasbeef Roasbeef merged commit 838f7ba into lightninglabs:master Oct 3, 2018

1 of 2 checks passed

coverage/coveralls Coverage decreased (-1.02%) to 69.478%
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment