Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(Dgraph): Use Badger with new WAL format. #6643

Merged
merged 31 commits into from Oct 7, 2020
Merged

feat(Dgraph): Use Badger with new WAL format. #6643

merged 31 commits into from Oct 7, 2020

Conversation

jarifibrahim
Copy link
Contributor

@jarifibrahim jarifibrahim commented Oct 4, 2020

This PR moves Dgraph to the new version of Badger that includes the new WAL. Some changes were
needed to make this work in Dgraph.

  • Make sure that cache sizes are not zero when needed in alpha, bulk loader, and live loader.
  • Remove flags that are no longer needed since the loading modes were changed to only
    support mmap.
  • Remove deprecated options when opening DBs.
  • Update how cache metrics are reported.
  • Various other changes required to make the build compile.

This change is Reviewable

Docs Preview: Dgraph Preview

@github-actions github-actions bot added area/bulk-loader Issues related to bulk loading. area/live-loader Issues related to live loading. labels Oct 4, 2020
@jarifibrahim jarifibrahim changed the title [Do not merge] Use Badger with new WAL format Use Badger with new WAL format Oct 7, 2020
Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Let's get it merged asap.

Reviewed 6 of 15 files at r1, 3 of 9 files at r2, 5 of 7 files at r3, 7 of 8 files at r5, 1 of 1 files at r6, 2 of 2 files at r7.
Reviewable status: 23 of 26 files reviewed, 3 unresolved discussions (waiting on @jarifibrahim and @manishrjain)


go.sum, line 166 at r7 (raw file):

github.com/google/codesearch v1.0.0 h1:z4h5JoHkUS+GqxqPDrldC3Y0Qq0vHAGgaDEW5pWU/ys=
github.com/google/codesearch v1.0.0/go.mod h1:qCnXDFnak/trCmLaE50kgPte3AX9jSeruZexWEOivi0=
github.com/google/flatbuffers v1.12.0 h1:/PtAHvnBY4Kqnx/xCQ3OIV9uYcSFGScBsWI3Oogeh6w=

go mod tidy


test.sh, line 244 at r7 (raw file):

if [[ :${TEST_SET}: == *:systest:* ]]; then
    # TODO: Fix this test. The fix consists of updating the test script to

This test isn't that useful. So, disabling it for now.


dgraph/cmd/bulk/run.go, line 114 at r7 (raw file):

		"The compression level for Badger. A higher value uses more resources.")
	flag.Int64("badger.cache_mb", 64, "Total size of cache (in MB) per shard in reducer.")
	flag.String("badger.cache_percentage", "100,0",

maybe 70,30

@martinmr martinmr changed the title Use Badger with new WAL format feat(Dgraph): Use Badger with new WAL format. Oct 7, 2020
@martinmr martinmr marked this pull request as ready for review October 7, 2020 15:35
@martinmr martinmr merged commit b1b47dc into master Oct 7, 2020
@martinmr martinmr deleted the martinmr/wal branch October 7, 2020 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/bulk-loader Issues related to bulk loading. area/live-loader Issues related to live loading.
Development

Successfully merging this pull request may close these issues.

None yet

4 participants