Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move bootstrapping queue out of common #2856

Merged
merged 99 commits into from
Apr 2, 2024
Merged
Show file tree
Hide file tree
Changes from 83 commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
c5b7aa9
Implement interval tree for syncing
StephenButtolph Feb 20, 2024
63efd3b
Persist intervals
StephenButtolph Feb 20, 2024
2454c46
rename
StephenButtolph Feb 20, 2024
0efa5eb
Add block execution
StephenButtolph Feb 21, 2024
6a65a2a
Add length + TODOs
StephenButtolph Feb 21, 2024
f2234bd
nit
StephenButtolph Feb 21, 2024
1f63238
error rather than panic
StephenButtolph Feb 21, 2024
6947876
Refactor database passing
StephenButtolph Feb 21, 2024
28207f4
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 19, 2024
be64296
Add interval tests
StephenButtolph Mar 19, 2024
884f39c
Fix too frequent iterator releases and batch writes
StephenButtolph Mar 19, 2024
e4e92cc
nit + support cancellation
StephenButtolph Mar 19, 2024
72314d1
Add invariant comment + tests
StephenButtolph Mar 19, 2024
e494033
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 19, 2024
a64b863
wip
StephenButtolph Mar 19, 2024
782a02c
wip
StephenButtolph Mar 20, 2024
8851470
Merge branch 'master' into interval-tree-syncing
StephenButtolph Mar 20, 2024
2b07179
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 20, 2024
5358587
support cancellation
StephenButtolph Mar 20, 2024
9d24b20
save
StephenButtolph Mar 20, 2024
0de6142
remove test code
StephenButtolph Mar 20, 2024
e658404
keep acceptor callback
StephenButtolph Mar 20, 2024
03f60c6
lint
StephenButtolph Mar 20, 2024
30ac2fb
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 20, 2024
f81281a
fix test
StephenButtolph Mar 20, 2024
410db8c
Merge branch 'interval-tree-syncing-integration' of github.com:ava-la…
StephenButtolph Mar 20, 2024
984849c
remove incorrect db cleanup
StephenButtolph Mar 20, 2024
f4a724a
nits
StephenButtolph Mar 20, 2024
f01b6a3
allow specifying the log level
StephenButtolph Mar 20, 2024
9d7383f
merged
StephenButtolph Mar 20, 2024
0021f3e
cleanup metrics
StephenButtolph Mar 20, 2024
f1933d6
lint
StephenButtolph Mar 20, 2024
dd4a57f
add error information
StephenButtolph Mar 20, 2024
58eeb52
move queue out of common
StephenButtolph Mar 20, 2024
f50eb51
fix
StephenButtolph Mar 20, 2024
d5da1e9
upstream
StephenButtolph Mar 20, 2024
7373a89
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 20, 2024
25e2801
cleanup
StephenButtolph Mar 20, 2024
5ca206b
reduce copied code
StephenButtolph Mar 20, 2024
cd90ef1
Merge branch 'interval-tree-syncing-integration' of github.com:ava-la…
StephenButtolph Mar 20, 2024
d9a7f8a
nit cleanup
StephenButtolph Mar 20, 2024
bd2b2d2
nit cleanup
StephenButtolph Mar 20, 2024
f0c5c10
Merge branch 'master' into interval-tree-syncing
StephenButtolph Mar 20, 2024
68ea448
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 20, 2024
b6d6ba9
Merge branch 'master' into interval-tree-syncing
StephenButtolph Mar 21, 2024
281dc8d
merged
StephenButtolph Mar 21, 2024
2c85c58
Merge branch 'interval-tree-syncing-integration' into move-job-queue
StephenButtolph Mar 21, 2024
06fd656
Merge branch 'master' into interval-tree-syncing
StephenButtolph Mar 21, 2024
672c23f
merged
StephenButtolph Mar 21, 2024
175ab8e
nit
StephenButtolph Mar 21, 2024
c89d24d
Remove redundent interface
StephenButtolph Mar 21, 2024
249819c
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 21, 2024
a736e6c
nit
StephenButtolph Mar 21, 2024
faa5329
nit
StephenButtolph Mar 21, 2024
bc3fedf
nit
StephenButtolph Mar 25, 2024
8134e16
nit
StephenButtolph Mar 25, 2024
9991e39
Move GetMissingBlockIDs and Execute out of the interval package
StephenButtolph Mar 25, 2024
2877502
Merge branch 'master' into interval-tree-syncing
StephenButtolph Mar 25, 2024
50754ad
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 25, 2024
a7e2458
wip
StephenButtolph Mar 25, 2024
f1dd0a6
nit
StephenButtolph Mar 26, 2024
bf29d86
remove storage
StephenButtolph Mar 26, 2024
0626129
merged
StephenButtolph Mar 26, 2024
5a8fadc
nit
StephenButtolph Mar 26, 2024
69ed897
nit
StephenButtolph Mar 26, 2024
9815047
nit
StephenButtolph Mar 26, 2024
5bf4899
nit
StephenButtolph Mar 26, 2024
9da3bc1
nit
StephenButtolph Mar 26, 2024
93aac21
Merge branch 'interval-tree-syncing' into interval-tree-syncing-integ…
StephenButtolph Mar 26, 2024
bd5687a
nit
StephenButtolph Mar 26, 2024
b6e3ee1
merged
StephenButtolph Mar 26, 2024
c376df2
add process test
StephenButtolph Mar 26, 2024
53bdb78
nit
StephenButtolph Mar 26, 2024
7310832
nit
StephenButtolph Mar 26, 2024
7de5499
wip
StephenButtolph Mar 26, 2024
4f88827
wip
StephenButtolph Mar 26, 2024
21d1fd1
remove duplicate code
StephenButtolph Mar 26, 2024
dd98c10
nit
StephenButtolph Mar 26, 2024
0f5c1fc
Merge branch 'interval-tree-syncing-integration' into move-job-queue
StephenButtolph Mar 26, 2024
81f4b67
nit fix log check
StephenButtolph Mar 26, 2024
ac06824
Merge branch 'interval-tree-syncing-integration' into move-job-queue
StephenButtolph Mar 26, 2024
03a22b8
nit
StephenButtolph Mar 26, 2024
f2f3186
Merge branch 'interval-tree-syncing-integration' into move-job-queue
StephenButtolph Mar 26, 2024
d753cb5
nit
StephenButtolph Mar 26, 2024
aa8d248
Remove haltableContext
StephenButtolph Mar 26, 2024
5d67d98
merged
StephenButtolph Mar 26, 2024
ef09bd2
nit
StephenButtolph Mar 26, 2024
fe23a59
Merge branch 'master' into interval-tree-syncing-integration
StephenButtolph Mar 27, 2024
db1c8af
Cleanup getMissingBlockIDs tests
StephenButtolph Mar 27, 2024
8d6f491
refactor process
StephenButtolph Mar 27, 2024
699fae3
nit
StephenButtolph Mar 27, 2024
e5e8dff
cleanup
StephenButtolph Mar 27, 2024
0c2b01a
fix test
StephenButtolph Mar 28, 2024
8ce7bfb
Remove fetch eta metric
StephenButtolph Mar 28, 2024
3b32eec
cleanup execution test
StephenButtolph Mar 28, 2024
673318e
Merge branch 'master' into interval-tree-syncing-integration
StephenButtolph Mar 28, 2024
f4b6127
nits
StephenButtolph Mar 29, 2024
4e07f30
Merge branch 'interval-tree-syncing-integration' into move-job-queue
StephenButtolph Apr 2, 2024
f45c3f0
Merge branch 'master' into move-job-queue
StephenButtolph Apr 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
19 changes: 5 additions & 14 deletions chains/manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,10 @@ import (
"github.com/ava-labs/avalanchego/network"
"github.com/ava-labs/avalanchego/proto/pb/p2p"
"github.com/ava-labs/avalanchego/snow"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/bootstrap/queue"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/state"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/vertex"
"github.com/ava-labs/avalanchego/snow/engine/common"
"github.com/ava-labs/avalanchego/snow/engine/common/queue"
"github.com/ava-labs/avalanchego/snow/engine/common/tracker"
"github.com/ava-labs/avalanchego/snow/engine/snowman/block"
"github.com/ava-labs/avalanchego/snow/engine/snowman/syncer"
Expand Down Expand Up @@ -85,10 +85,10 @@ var (
VertexDBPrefix = []byte("vertex")
VertexBootstrappingDBPrefix = []byte("vertex_bs")
TxBootstrappingDBPrefix = []byte("tx_bs")
BlockBootstrappingDBPrefix = []byte("block_bs")
BlockBootstrappingDBPrefix = []byte("interval_block_bs")

// Bootstrapping prefixes for ChainVMs
ChainBootstrappingDBPrefix = []byte("bs")
ChainBootstrappingDBPrefix = []byte("interval_bs")

errUnknownVMType = errors.New("the vm should have type avalanche.DAGVM or snowman.ChainVM")
errCreatePlatformVM = errors.New("attempted to create a chain running the PlatformVM")
Expand Down Expand Up @@ -579,10 +579,6 @@ func (m *manager) createAvalancheChain(
if err != nil {
return nil, err
}
blockBlocker, err := queue.NewWithMissing(blockBootstrappingDB, "block", ctx.Registerer)
if err != nil {
return nil, err
}

// Passes messages from the avalanche engines to the network
avalancheMessageSender, err := sender.New(
Expand Down Expand Up @@ -837,7 +833,7 @@ func (m *manager) createAvalancheChain(
BootstrapTracker: sb,
Timer: h,
AncestorsMaxContainersReceived: m.BootstrapAncestorsMaxContainersReceived,
Blocked: blockBlocker,
DB: blockBootstrappingDB,
VM: vmWrappingProposerVM,
}
var snowmanBootstrapper common.BootstrapableEngine
Expand Down Expand Up @@ -952,11 +948,6 @@ func (m *manager) createSnowmanChain(
vmDB := prefixdb.New(VMDBPrefix, prefixDB)
bootstrappingDB := prefixdb.New(ChainBootstrappingDBPrefix, prefixDB)

blocked, err := queue.NewWithMissing(bootstrappingDB, "block", ctx.Registerer)
if err != nil {
return nil, err
}

// Passes messages from the consensus engine to the network
messageSender, err := sender.New(
ctx,
Expand Down Expand Up @@ -1175,7 +1166,7 @@ func (m *manager) createSnowmanChain(
BootstrapTracker: sb,
Timer: h,
AncestorsMaxContainersReceived: m.BootstrapAncestorsMaxContainersReceived,
Blocked: blocked,
DB: bootstrappingDB,
VM: vm,
Bootstrapped: bootstrapFunc,
}
Expand Down
2 changes: 1 addition & 1 deletion snow/engine/avalanche/bootstrap/bootstrapper_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ import (
"github.com/ava-labs/avalanchego/snow/choices"
"github.com/ava-labs/avalanchego/snow/consensus/avalanche"
"github.com/ava-labs/avalanchego/snow/consensus/snowstorm"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/bootstrap/queue"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/getter"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/vertex"
"github.com/ava-labs/avalanchego/snow/engine/common"
"github.com/ava-labs/avalanchego/snow/engine/common/queue"
"github.com/ava-labs/avalanchego/snow/engine/common/tracker"
"github.com/ava-labs/avalanchego/snow/snowtest"
"github.com/ava-labs/avalanchego/snow/validators"
Expand Down
2 changes: 1 addition & 1 deletion snow/engine/avalanche/bootstrap/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ package bootstrap
import (
"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/snow"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/bootstrap/queue"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/vertex"
"github.com/ava-labs/avalanchego/snow/engine/common"
"github.com/ava-labs/avalanchego/snow/engine/common/queue"
"github.com/ava-labs/avalanchego/snow/engine/common/tracker"
"github.com/ava-labs/avalanchego/snow/validators"
)
Expand Down
2 changes: 1 addition & 1 deletion snow/engine/avalanche/bootstrap/tx_job.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ import (
"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/snow/choices"
"github.com/ava-labs/avalanchego/snow/consensus/snowstorm"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/bootstrap/queue"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/vertex"
"github.com/ava-labs/avalanchego/snow/engine/common/queue"
"github.com/ava-labs/avalanchego/utils/logging"
"github.com/ava-labs/avalanchego/utils/set"
)
Expand Down
2 changes: 1 addition & 1 deletion snow/engine/avalanche/bootstrap/vertex_job.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ import (
"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/snow/choices"
"github.com/ava-labs/avalanchego/snow/consensus/avalanche"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/bootstrap/queue"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/vertex"
"github.com/ava-labs/avalanchego/snow/engine/common/queue"
"github.com/ava-labs/avalanchego/utils/logging"
"github.com/ava-labs/avalanchego/utils/set"
)
Expand Down
53 changes: 53 additions & 0 deletions snow/engine/snowman/bootstrap/acceptor.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
// Copyright (C) 2019-2024, Ava Labs, Inc. All rights reserved.
// See the file LICENSE for licensing terms.

package bootstrap

import (
"context"

"github.com/prometheus/client_golang/prometheus"

"github.com/ava-labs/avalanchego/snow"
"github.com/ava-labs/avalanchego/snow/consensus/snowman"
"github.com/ava-labs/avalanchego/snow/engine/snowman/block"
)

var (
_ block.Parser = (*parseAcceptor)(nil)
_ snowman.Block = (*blockAcceptor)(nil)
)

type parseAcceptor struct {
parser block.Parser
ctx *snow.ConsensusContext
numAccepted prometheus.Counter
}

func (p *parseAcceptor) ParseBlock(ctx context.Context, bytes []byte) (snowman.Block, error) {
blk, err := p.parser.ParseBlock(ctx, bytes)
if err != nil {
return nil, err
}
return &blockAcceptor{
Block: blk,
ctx: p.ctx,
numAccepted: p.numAccepted,
}, nil
}

type blockAcceptor struct {
snowman.Block

ctx *snow.ConsensusContext
numAccepted prometheus.Counter
}

func (b *blockAcceptor) Accept(ctx context.Context) error {
if err := b.ctx.BlockAcceptor.Accept(b.ctx, b.ID(), b.Bytes()); err != nil {
return err
}
err := b.Block.Accept(ctx)
b.numAccepted.Inc()
return err
}
113 changes: 0 additions & 113 deletions snow/engine/snowman/bootstrap/block_job.go

This file was deleted.