-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow Dgraph to run badger in memory #8877
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are our plans for this PR? Do we want to add a separate option in Dgraph for in memory badger?
dgraph/cmd/alpha/run.go
Outdated
@@ -683,7 +681,7 @@ func run() { | |||
|
|||
raft := z.NewSuperFlag(Alpha.Conf.GetString("raft")).MergeAndCheckDefault(worker.RaftDefaults) | |||
x.WorkerConfig = x.WorkerOptions{ | |||
TmpDir: Alpha.Conf.GetString("tmp"), | |||
TmpDir: "./t", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need this?
dgraph/cmd/alpha/run.go
Outdated
@@ -700,7 +698,9 @@ func run() { | |||
Badger: bopts, | |||
} | |||
x.WorkerConfig.Parse(Alpha.Conf) | |||
fmt.Println("WORKER CONFIG ============", x.WorkerConfig) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will this print correctly? Could you copy an example here?
raftwal/log.go
Outdated
@@ -114,7 +114,7 @@ func openLogFile(dir string, fid int64) (*logFile, error) { | |||
Dir: dir, | |||
EncryptionKey: encryptionKey, | |||
EncryptionKeyRotationDuration: 10 * 24 * time.Hour, | |||
InMemory: false, | |||
InMemory: true, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we are doing this permanent?
worker/config.go
Outdated
@@ -84,8 +84,6 @@ func (opt *Options) validate() { | |||
x.Check(err) | |||
td, err := filepath.Abs(x.WorkerConfig.TmpDir) | |||
x.Check(err) | |||
x.AssertTruef(pd != wd, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should add a special condition for in memory.
worker/server_state.go
Outdated
s.WALstore, err = raftwal.InitEncrypted(Config.WALDir, x.WorkerConfig.EncryptionKey) | ||
x.Check(err) | ||
} | ||
{ | ||
// Postings directory | ||
// All the writes to posting store should be synchronous. We use batched writers | ||
// for posting lists, so the cost of sync writes is amortized. | ||
x.Check(os.MkdirAll(Config.PostingDir, 0700)) | ||
fmt.Println("Here", Config.PostingDir) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should add a test to ensure that we don't break it in future.
@@ -118,19 +118,25 @@ func (s *ServerState) initStorage() { | |||
|
|||
{ | |||
// Write Ahead Log directory | |||
x.Checkf(os.MkdirAll(Config.WALDir, 0700), "Error while creating WAL dir.") | |||
if Config.WALDir != "" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@harshil-goel just trying to understand the changes.
Is it the case that when badger is run in-memory, WALDir and PostingDir will be empty? And raftwal.InitEncrypted() handles this case properly?
This PR has been stale for 60 days and will be closed automatically in 7 days. Comment to keep it open. |
No description provided.