Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade to badger/v3 #78

Merged
merged 1 commit into from
Jul 2, 2021
Merged

Upgrade to badger/v3 #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/edge.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
- name: Install Go
uses: actions/setup-go@v1
with:
go-version: 1.14
go-version: 1.16
id: go
- name: Login to DockerHub Registry
uses: docker/login-action@v1
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/latest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
- name: Install Go
uses: actions/setup-go@v1
with:
go-version: 1.14
go-version: 1.16
id: go
- name: Login to DockerHub Registry
uses: docker/login-action@v1
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
- name: Install Go
uses: actions/setup-go@v1
with:
go-version: 1.14
go-version: 1.16
id: go
- name: Login to DockerHub Registry
uses: docker/login-action@v1
Expand Down
22 changes: 10 additions & 12 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,19 @@ env:
GO111MODULE: "on"

jobs:

test:
name: Run Tests
runs-on: ubuntu-latest
steps:
- name: Install Go
uses: actions/setup-go@v1
with:
go-version: 1.16
id: go

- name: Install Go
uses: actions/setup-go@v1
with:
go-version: 1.14
id: go

- name: Check out code into the Go module directory
uses: actions/checkout@v1
- name: Check out code into the Go module directory
uses: actions/checkout@v1

- name: Run tests
run: |
go test -race ./...
- name: Run tests
run: |
go test -race ./...
1 change: 1 addition & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ ARG GOARCH="amd64"
ENV GO111MODULE=${GO111MODULE}
ENV GOOS=${GOOS}
ENV GOARCH=${GOARCH}

# GOPATH => /go
RUN mkdir -p /go/src/talaria
COPY . src/talaria
Expand Down
5 changes: 2 additions & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/kelindar/talaria

go 1.14
go 1.16

require (
cloud.google.com/go v0.57.0 // indirect
Expand All @@ -17,8 +17,7 @@ require (
github.com/armon/go-metrics v0.3.3 // indirect
github.com/aws/aws-sdk-go v1.30.25
github.com/crphang/orc v0.0.6
github.com/dgraph-io/badger/v2 v2.0.1-rc1.0.20200626160443-3042e3776798
github.com/dgraph-io/ristretto v0.0.2 // indirect
github.com/dgraph-io/badger/v3 v3.2103.0
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
github.com/dnaeon/go-vcr v1.0.1 // indirect
github.com/emitter-io/address v1.0.0
Expand Down
74 changes: 10 additions & 64 deletions go.sum
View file
Open in desktop

Large diffs are not rendered by default.

26 changes: 16 additions & 10 deletions internal/storage/disk/disk.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (
"sync/atomic"
"time"

"github.com/dgraph-io/badger/v2"
"github.com/dgraph-io/badger/v3"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will it auto upgrade a v2 database on opening?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, haven't tried. I would also like @qiaowei-g to review the options. There's many changes in badger and apparently they are using manual memory allocation to use a bit less memory and free it better.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on the description in badger-#1555, v2 data cannot be read by v3, so we will have to handle the deployment carefully to ensure no data loss.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, I'll mark it as 1.3 release so we don't mix things up.

"github.com/grab/async"
"github.com/kelindar/talaria/internal/config"
"github.com/kelindar/talaria/internal/encoding/key"
Expand Down Expand Up @@ -72,11 +72,22 @@ func (s *Storage) Open(dir string, options config.Badger) error {

opts := badger.DefaultOptions(dir)

//WithNumLevelZeroTables().
//WithNumLevelZeroTablesStall()

switch options.Default {
case config.BadgerStorage:
opts = opts.WithMaxLevels(64 << 15).WithValueLogMaxEntries(5000).WithLevelOneSize(1 << 16).WithLevelSizeMultiplier(3).WithMaxLevels(25).WithSyncWrites(false)
opts = opts.WithMaxLevels(64 << 15).
WithValueLogMaxEntries(5000).
WithBaseLevelSize(1 << 16).
WithLevelSizeMultiplier(3).
WithMaxLevels(25).
WithSyncWrites(false)
case config.BadgerIngestion:
opts = opts.WithLevelOneSize(204800000).WithMaxLevels(3).WithSyncWrites(false)
opts = opts.
WithBaseLevelSize(204800000).
WithMaxLevels(3).
WithSyncWrites(false)
}

// Create the options
Expand All @@ -87,7 +98,7 @@ func (s *Storage) Open(dir string, options config.Badger) error {
// max size of lsm tree in bytes after which data is propagated to disk.
// The default is 64 MB.
if options.MaxTableSize != nil {
opts = opts.WithMaxTableSize(*options.MaxTableSize)
opts = opts.WithBaseTableSize(*options.MaxTableSize)
}

// ValueLogMaxEntries sets the maximum number of entries a value log file can hold approximately.
Expand All @@ -101,7 +112,7 @@ func (s *Storage) Open(dir string, options config.Badger) error {
// LevelOneSize sets the maximum total size for Level 1.
// The default value of LevelOneSize is 20MB.
if options.LevelOneSize != nil {
opts = opts.WithLevelOneSize(*options.LevelOneSize)
opts = opts.WithBaseLevelSize(*options.LevelOneSize)
}

// Maximum number of levels of compaction allowed in the LSM.
Expand All @@ -118,11 +129,6 @@ func (s *Storage) Open(dir string, options config.Badger) error {
opts = opts.WithLevelSizeMultiplier(*options.LevelSizeMultiplier)
}

// Truncate indicates whether value log files should be truncated to delete corrupt data, if any.
// This option is ignored when ReadOnly is true.
// The default value of Truncate is false.
opts = opts.WithTruncate(true)

opts = opts.WithLogger(&logger{s.monitor})
s.monitor.Info("opening badger with options %+v", opts)

Expand Down
1 change: 0 additions & 1 deletion internal/storage/disk/disk_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ import (
"time"

"github.com/kelindar/talaria/internal/config"

"github.com/kelindar/talaria/internal/encoding/key"
"github.com/kelindar/talaria/internal/monitor"
"github.com/stretchr/testify/assert"
Expand Down