Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pebble as valid value for --db-type. #2244

Merged
merged 85 commits into from
Oct 31, 2023
Merged
Show file tree
Hide file tree
Changes from 82 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
d24ee4c
initial commit
danlaine Sep 11, 2023
d578d00
Merge remote-tracking branch 'upstream/dev' into pebble
danlaine Sep 11, 2023
db2eeeb
update pebble dep
danlaine Sep 11, 2023
0983042
Merge remote-tracking branch 'upstream/dev' into pebble
danlaine Sep 12, 2023
528cc2d
appease linter
danlaine Sep 12, 2023
1bcfa65
nit
danlaine Sep 12, 2023
73a934f
nit
danlaine Sep 12, 2023
1e094b4
nit
danlaine Sep 12, 2023
5f39300
Merge branch 'dev' into pebble
danlaine Sep 12, 2023
d937e6c
Merge branch 'dev' into pebble
danlaine Sep 15, 2023
721275e
nit
danlaine Sep 15, 2023
5a8d855
Merge branch 'dev' into pebble
danlaine Sep 18, 2023
9f7665b
fix tests
danlaine Sep 18, 2023
1db44c4
nit
danlaine Sep 18, 2023
7e4cf61
Merge branch 'dev' into pebble
danlaine Sep 19, 2023
4051e88
nit
danlaine Sep 19, 2023
4bc0630
Merge branch 'dev' into pebble
danlaine Sep 22, 2023
662975f
Merge branch 'dev' into pebble
danlaine Sep 25, 2023
cd0a8e2
appease linter
danlaine Sep 25, 2023
261e710
Merge branch 'dev' into pebble
dhrubabasu Sep 27, 2023
59a8615
Merge remote-tracking branch 'upstream/dev' into pebble
danlaine Oct 2, 2023
a710f54
remove TODO
danlaine Oct 2, 2023
f21c81d
remove unneeded var
danlaine Oct 2, 2023
cc7a15e
unexport error
danlaine Oct 2, 2023
f9192ed
atomic bool --> bool
danlaine Oct 2, 2023
155edc9
merged
StephenButtolph Oct 24, 2023
72a249d
match pebble version
StephenButtolph Oct 24, 2023
4b08b25
fix test:
danlaine Oct 25, 2023
b8fa698
change pebble opening log; edit comment
danlaine Oct 25, 2023
a1b7fba
pebbledb.New nit
danlaine Oct 25, 2023
6b93c90
Lock --> RLock in Put, Delete
danlaine Oct 25, 2023
610e7f8
iterator nits
danlaine Oct 25, 2023
fc91cd1
Compact Lock --> RLock
danlaine Oct 25, 2023
5aff591
prefixBounds --> keyRange and prefixToUpperBound
danlaine Oct 25, 2023
49499fa
Merge remote-tracking branch 'upstream/dev' into pebble
danlaine Oct 25, 2023
bba3048
remove unneeded iterator creator code
danlaine Oct 25, 2023
3ffc6a8
remove unneeded check in iter.Next
danlaine Oct 25, 2023
a8ff7b8
remove unused var
danlaine Oct 25, 2023
c6bacc7
change switch order in iter.Next to not overwrite iter error
danlaine Oct 25, 2023
b886790
remove unneeded lock grab and db closed check in iter.Next
danlaine Oct 25, 2023
f771665
prevent Compact panic
danlaine Oct 25, 2023
3412216
fix default MaxConcurrentCompactions; fix compaction iterator leak; a…
danlaine Oct 25, 2023
f0e3f2e
cleanup and comments
danlaine Oct 25, 2023
c4d1a87
add Key and Value invariant on database.Iterator
danlaine Oct 25, 2023
d379e77
spelling is hard
danlaine Oct 25, 2023
8eea9eb
make pebble.New take in config bytes instead of config
danlaine Oct 25, 2023
c350116
Merge remote-tracking branch 'upstream/dev' into pebble
danlaine Oct 25, 2023
b64de08
os.MkDirTemp --> t.TempDir
danlaine Oct 25, 2023
b1768c1
remove unneeded file removal
danlaine Oct 25, 2023
afd88e3
Close db
StephenButtolph Oct 25, 2023
b360819
add flags for pebble; only unmarshal config bytes if given
danlaine Oct 25, 2023
6f89805
Merge remote-tracking branch 'upstream/dev' into pebble-2
danlaine Oct 25, 2023
9b4343f
Merge branch 'dev' into pebble-2
danlaine Oct 26, 2023
417785f
comments
danlaine Oct 26, 2023
da99730
Merge branch 'pebble-2' of github.com:ava-labs/avalanchego into pebble-2
danlaine Oct 26, 2023
fbfdc6d
Merge branch 'dev' into pebble-2
danlaine Oct 26, 2023
70d3420
Merge branch 'dev' into pebble-2
danlaine Oct 27, 2023
f41856a
Merge branch 'dev' into pebble-2
danlaine Oct 27, 2023
b49340b
nit move name declaration
danlaine Oct 30, 2023
c18a208
Merge branch 'dev' into pebble-2
danlaine Oct 30, 2023
3dca091
remove db manager
danlaine Oct 30, 2023
a131092
fix initDatabase
danlaine Oct 30, 2023
ce33816
fix initDatabase
danlaine Oct 30, 2023
0b66354
revert go.mod and go.sum changes
danlaine Oct 30, 2023
f729156
remove DBServer proto type
danlaine Oct 30, 2023
d0ff782
revert scripts changes
danlaine Oct 30, 2023
4ea6944
revert byte change
danlaine Oct 30, 2023
3e6841f
nit
danlaine Oct 30, 2023
af1bc23
change healthy reporting
danlaine Oct 30, 2023
0891312
Merge branch 'dev' into remove-db-manager
danlaine Oct 30, 2023
e6b970a
update coreth dep
danlaine Oct 31, 2023
8ee8821
Merge branch 'remove-db-manager' into pebble-no-db-manager
danlaine Oct 31, 2023
ba9fac7
update level/pebble creation
danlaine Oct 31, 2023
a0b4ef8
remove useless vm.connCloser.Close call
danlaine Oct 31, 2023
4fd170d
fix camelCase
danlaine Oct 31, 2023
81bb71e
nits
danlaine Oct 31, 2023
3987bf5
fix database prefixing
danlaine Oct 31, 2023
a8f1fe1
Merge remote-tracking branch 'upstream/dev' into remove-db-manager
danlaine Oct 31, 2023
3a71f38
Merge branch 'dev' into pebble-2
danlaine Oct 31, 2023
63efa76
make prefix var
danlaine Oct 31, 2023
887c094
Merge branch 'remove-db-manager' into pebble-no-db-manager
danlaine Oct 31, 2023
10a9a56
Merge remote-tracking branch 'upstream/pebble-2' into pebble-no-db-ma…
danlaine Oct 31, 2023
8feabe9
Reduce diff + complexity
StephenButtolph Oct 31, 2023
de99f6e
include comment
StephenButtolph Oct 31, 2023
f562920
merged
StephenButtolph Oct 31, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 3 additions & 5 deletions api/keystore/keystore.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ import (
"github.com/ava-labs/avalanchego/chains/atomic"
"github.com/ava-labs/avalanchego/database"
"github.com/ava-labs/avalanchego/database/encdb"
"github.com/ava-labs/avalanchego/database/manager"
"github.com/ava-labs/avalanchego/database/prefixdb"
"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/utils/json"
Expand Down Expand Up @@ -105,13 +104,12 @@ type keystore struct {
bcDB database.Database
}

func New(log logging.Logger, dbManager manager.Manager) Keystore {
currentDB := dbManager.Current()
func New(log logging.Logger, db database.Database) Keystore {
return &keystore{
log: log,
usernameToPassword: make(map[string]*password.Hash),
userDB: prefixdb.New(usersPrefix, currentDB.Database),
bcDB: prefixdb.New(bcsPrefix, currentDB.Database),
userDB: prefixdb.New(usersPrefix, db),
bcDB: prefixdb.New(bcsPrefix, db),
}
}

Expand Down
17 changes: 0 additions & 17 deletions api/keystore/service.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,8 @@ import (
"go.uber.org/zap"

"github.com/ava-labs/avalanchego/api"
"github.com/ava-labs/avalanchego/database/manager"
"github.com/ava-labs/avalanchego/database/memdb"
"github.com/ava-labs/avalanchego/utils/formatting"
"github.com/ava-labs/avalanchego/utils/logging"
"github.com/ava-labs/avalanchego/version"
)

type service struct {
Expand Down Expand Up @@ -115,17 +112,3 @@ func (s *service) ExportUser(_ *http.Request, args *ExportUserArgs, reply *Expor
reply.Encoding = args.Encoding
return nil
}

// CreateTestKeystore returns a new keystore that can be utilized for testing
func CreateTestKeystore() (Keystore, error) {
dbManager, err := manager.NewManagerFromDBs([]*manager.VersionedDatabase{
{
Database: memdb.New(),
Version: version.Semantic1_0_0,
},
})
if err != nil {
return nil, err
}
return New(logging.NoLog{}, dbManager), nil
}
36 changes: 14 additions & 22 deletions api/keystore/service_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@ import (
"github.com/stretchr/testify/require"

"github.com/ava-labs/avalanchego/api"
"github.com/ava-labs/avalanchego/database/memdb"
"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/utils/formatting"
"github.com/ava-labs/avalanchego/utils/logging"
"github.com/ava-labs/avalanchego/utils/password"
)

Expand All @@ -23,8 +25,7 @@ var strongPassword = "N_+=_jJ;^(<;{4,:*m6CET}'&N;83FYK.wtNpwp-Jt" // #nosec G101
func TestServiceListNoUsers(t *testing.T) {
require := require.New(t)

ks, err := CreateTestKeystore()
require.NoError(err)
ks := New(logging.NoLog{}, memdb.New())
s := service{ks: ks.(*keystore)}

reply := ListUsersReply{}
Expand All @@ -35,8 +36,7 @@ func TestServiceListNoUsers(t *testing.T) {
func TestServiceCreateUser(t *testing.T) {
require := require.New(t)

ks, err := CreateTestKeystore()
require.NoError(err)
ks := New(logging.NoLog{}, memdb.New())
s := service{ks: ks.(*keystore)}

{
Expand Down Expand Up @@ -66,8 +66,7 @@ func genStr(n int) string {
func TestServiceCreateUserArgsCheck(t *testing.T) {
require := require.New(t)

ks, err := CreateTestKeystore()
require.NoError(err)
ks := New(logging.NoLog{}, memdb.New())
s := service{ks: ks.(*keystore)}

{
Expand Down Expand Up @@ -100,8 +99,7 @@ func TestServiceCreateUserArgsCheck(t *testing.T) {
func TestServiceCreateUserWeakPassword(t *testing.T) {
require := require.New(t)

ks, err := CreateTestKeystore()
require.NoError(err)
ks := New(logging.NoLog{}, memdb.New())
s := service{ks: ks.(*keystore)}

{
Expand All @@ -117,8 +115,7 @@ func TestServiceCreateUserWeakPassword(t *testing.T) {
func TestServiceCreateDuplicate(t *testing.T) {
require := require.New(t)

ks, err := CreateTestKeystore()
require.NoError(err)
ks := New(logging.NoLog{}, memdb.New())
s := service{ks: ks.(*keystore)}

{
Expand All @@ -140,12 +137,11 @@ func TestServiceCreateDuplicate(t *testing.T) {
func TestServiceCreateUserNoName(t *testing.T) {
require := require.New(t)

ks, err := CreateTestKeystore()
require.NoError(err)
ks := New(logging.NoLog{}, memdb.New())
s := service{ks: ks.(*keystore)}

reply := api.EmptyReply{}
err = s.CreateUser(nil, &api.UserPass{
err := s.CreateUser(nil, &api.UserPass{
Password: strongPassword,
}, &reply)
require.ErrorIs(err, errEmptyUsername)
Expand All @@ -154,8 +150,7 @@ func TestServiceCreateUserNoName(t *testing.T) {
func TestServiceUseBlockchainDB(t *testing.T) {
require := require.New(t)

ks, err := CreateTestKeystore()
require.NoError(err)
ks := New(logging.NoLog{}, memdb.New())
s := service{ks: ks.(*keystore)}

{
Expand Down Expand Up @@ -185,8 +180,7 @@ func TestServiceExportImport(t *testing.T) {

encodings := []formatting.Encoding{formatting.Hex}
for _, encoding := range encodings {
ks, err := CreateTestKeystore()
require.NoError(err)
ks := New(logging.NoLog{}, memdb.New())
s := service{ks: ks.(*keystore)}

{
Expand All @@ -212,8 +206,7 @@ func TestServiceExportImport(t *testing.T) {
exportReply := ExportUserReply{}
require.NoError(s.ExportUser(nil, &exportArgs, &exportReply))

newKS, err := CreateTestKeystore()
require.NoError(err)
newKS := New(logging.NoLog{}, memdb.New())
newS := service{ks: newKS.(*keystore)}

{
Expand Down Expand Up @@ -324,16 +317,15 @@ func TestServiceDeleteUser(t *testing.T) {
t.Run(tt.desc, func(t *testing.T) {
require := require.New(t)

ksIntf, err := CreateTestKeystore()
require.NoError(err)
ksIntf := New(logging.NoLog{}, memdb.New())
ks := ksIntf.(*keystore)
s := service{ks: ks}

if tt.setup != nil {
require.NoError(tt.setup(ks))
}
got := &api.EmptyReply{}
err = s.DeleteUser(nil, tt.request, got)
err := s.DeleteUser(nil, tt.request, got)
require.ErrorIs(err, tt.expectedErr)
if tt.expectedErr != nil {
return
Expand Down
9 changes: 4 additions & 5 deletions chains/linearizable_vm.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,12 @@ import (
"context"

"github.com/ava-labs/avalanchego/api/metrics"
"github.com/ava-labs/avalanchego/database"
"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/snow"
"github.com/ava-labs/avalanchego/snow/engine/avalanche/vertex"
"github.com/ava-labs/avalanchego/snow/engine/common"
"github.com/ava-labs/avalanchego/snow/engine/snowman/block"

dbManager "github.com/ava-labs/avalanchego/database/manager"
)

var (
Expand All @@ -32,7 +31,7 @@ type initializeOnLinearizeVM struct {

registerer metrics.OptionalGatherer
ctx *snow.Context
dbManager dbManager.Manager
db database.Database
genesisBytes []byte
upgradeBytes []byte
configBytes []byte
Expand All @@ -47,7 +46,7 @@ func (vm *initializeOnLinearizeVM) Linearize(ctx context.Context, stopVertexID i
return vm.vmToInitialize.Initialize(
ctx,
vm.ctx,
vm.dbManager,
vm.db,
vm.genesisBytes,
vm.upgradeBytes,
vm.configBytes,
Expand All @@ -74,7 +73,7 @@ func NewLinearizeOnInitializeVM(vm vertex.LinearizableVMWithEngine) *linearizeOn
func (vm *linearizeOnInitializeVM) Initialize(
ctx context.Context,
_ *snow.Context,
_ dbManager.Manager,
_ database.Database,
_ []byte,
_ []byte,
_ []byte,
Expand Down
37 changes: 17 additions & 20 deletions chains/manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ import (
"github.com/ava-labs/avalanchego/api/metrics"
"github.com/ava-labs/avalanchego/api/server"
"github.com/ava-labs/avalanchego/chains/atomic"
"github.com/ava-labs/avalanchego/database"
"github.com/ava-labs/avalanchego/database/meterdb"
"github.com/ava-labs/avalanchego/database/prefixdb"
"github.com/ava-labs/avalanchego/ids"
"github.com/ava-labs/avalanchego/message"
Expand Down Expand Up @@ -57,7 +59,6 @@ import (
"github.com/ava-labs/avalanchego/vms/proposervm"
"github.com/ava-labs/avalanchego/vms/tracedvm"

dbManager "github.com/ava-labs/avalanchego/database/manager"
timetracker "github.com/ava-labs/avalanchego/snow/networking/tracker"

aveng "github.com/ava-labs/avalanchego/snow/engine/avalanche"
Expand Down Expand Up @@ -180,7 +181,7 @@ type ManagerConfig struct {
BlockAcceptorGroup snow.AcceptorGroup
TxAcceptorGroup snow.AcceptorGroup
VertexAcceptorGroup snow.AcceptorGroup
DBManager dbManager.Manager
DB database.Database
MsgCreator message.OutboundMsgBuilder // message creator, shared with network
Router router.Router // Routes incoming messages to the appropriate chain
Net network.Network // Sends consensus messages to other validators
Expand Down Expand Up @@ -596,18 +597,16 @@ func (m *manager) createAvalancheChain(
State: snow.Initializing,
})

meterDBManager, err := m.DBManager.NewMeterDBManager("db", ctx.Registerer)
meterDB, err := meterdb.New("db", ctx.Registerer, m.DB)
if err != nil {
return nil, err
}
prefixDBManager := meterDBManager.NewPrefixDBManager(ctx.ChainID[:])
vmDBManager := prefixDBManager.NewPrefixDBManager(vmDBPrefix)

db := prefixDBManager.Current()
vertexDB := prefixdb.New(vertexDBPrefix, db.Database)
vertexBootstrappingDB := prefixdb.New(vertexBootstrappingDBPrefix, db.Database)
txBootstrappingDB := prefixdb.New(txBootstrappingDBPrefix, db.Database)
blockBootstrappingDB := prefixdb.New(blockBootstrappingDBPrefix, db.Database)
prefixDB := prefixdb.New(ctx.ChainID[:], meterDB)
vmDB := prefixdb.New(vmDBPrefix, prefixDB)
vertexDB := prefixdb.New(vertexDBPrefix, prefixDB)
vertexBootstrappingDB := prefixdb.New(vertexBootstrappingDBPrefix, prefixDB)
txBootstrappingDB := prefixdb.New(txBootstrappingDBPrefix, prefixDB)
blockBootstrappingDB := prefixdb.New(blockBootstrappingDBPrefix, prefixDB)

vtxBlocker, err := queue.NewWithMissing(vertexBootstrappingDB, "vtx", ctx.AvalancheRegisterer)
if err != nil {
Expand Down Expand Up @@ -730,7 +729,7 @@ func (m *manager) createAvalancheChain(
err = dagVM.Initialize(
context.TODO(),
ctx.Context,
vmDBManager,
vmDB,
genesisData,
chainConfig.Upgrade,
chainConfig.Config,
Expand Down Expand Up @@ -796,7 +795,7 @@ func (m *manager) createAvalancheChain(

registerer: snowmanRegisterer,
ctx: ctx.Context,
dbManager: vmDBManager,
db: vmDB,
genesisBytes: genesisData,
upgradeBytes: chainConfig.Upgrade,
configBytes: chainConfig.Config,
Expand Down Expand Up @@ -1004,15 +1003,13 @@ func (m *manager) createSnowmanChain(
State: snow.Initializing,
})

meterDBManager, err := m.DBManager.NewMeterDBManager("db", ctx.Registerer)
meterDB, err := meterdb.New("db", ctx.Registerer, m.DB)
if err != nil {
return nil, err
}
prefixDBManager := meterDBManager.NewPrefixDBManager(ctx.ChainID[:])
vmDBManager := prefixDBManager.NewPrefixDBManager(vmDBPrefix)

db := prefixDBManager.Current()
bootstrappingDB := prefixdb.New(bootstrappingDB, db.Database)
prefixDB := prefixdb.New(ctx.ChainID[:], meterDB)
vmDB := prefixdb.New(vmDBPrefix, prefixDB)
bootstrappingDB := prefixdb.New(bootstrappingDB, prefixDB)

blocked, err := queue.NewWithMissing(bootstrappingDB, "block", ctx.Registerer)
if err != nil {
Expand Down Expand Up @@ -1145,7 +1142,7 @@ func (m *manager) createSnowmanChain(
if err := vm.Initialize(
context.TODO(),
ctx.Context,
vmDBManager,
vmDB,
genesisData,
chainConfig.Upgrade,
chainConfig.Config,
Expand Down
3 changes: 2 additions & 1 deletion config/flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (

"github.com/ava-labs/avalanchego/database/leveldb"
"github.com/ava-labs/avalanchego/database/memdb"
"github.com/ava-labs/avalanchego/database/pebble"
"github.com/ava-labs/avalanchego/genesis"
"github.com/ava-labs/avalanchego/snow/consensus/snowball"
"github.com/ava-labs/avalanchego/trace"
Expand Down Expand Up @@ -103,7 +104,7 @@ func addNodeFlags(fs *pflag.FlagSet) {
fs.Uint64(AddSubnetDelegatorFeeKey, genesis.LocalParams.AddSubnetDelegatorFee, "Transaction fee, in nAVAX, for transactions that add new subnet delegators")

// Database
fs.String(DBTypeKey, leveldb.Name, fmt.Sprintf("Database type to use. Should be one of {%s, %s}", leveldb.Name, memdb.Name))
fs.String(DBTypeKey, leveldb.Name, fmt.Sprintf("Database type to use. Must be one of {%s, %s, %s}", leveldb.Name, memdb.Name, pebble.Name))
fs.String(DBPathKey, defaultDBDir, "Path to database directory")
fs.String(DBConfigFileKey, "", fmt.Sprintf("Path to database config file. Ignored if %s is specified", DBConfigContentKey))
fs.String(DBConfigContentKey, "", "Specifies base64 encoded database config content")
Expand Down