Spikes in read transaction time with in memory mode badger.

[hqhs]

What version of Go are you using (go version)?

$ go version
go version go1.14

What operating system are you using?

amd64, dockerized

What version of Badger are you using?

v2.0.3

Does this issue reproduce with the latest master?

Probably, yes, I can't test it in my current setup.

Steps to Reproduce the issue

I have badger DB working as in-memory key-value store. Entry size is 3kb on avarage, there's approx. 400k entries with 150k read requests/minute, but they change frequently: 15k/minute.

What Badger options were set?

opts := badger.DefaultOptions("").
	WithInMemory(true)

What did you expect to see?

I expected to see stable response time for any read operation, but currently there's strange spikes in response time:

I've spent some time with documentation and sources, but failed to reproduce issue locally and to find desired combination to get rid of them. Setting WithSyncWrites(false), for example, fixes issue, but memory usage keeps growing until OOM. Tweaking MaxTableSize or NumCompactors affects response time, but doesn't fix issue completely. Interestingly, read perfomance during spike degrades than the writing load increases, as I understood, this is somehow related to periodic compaction of entries, but I have no idea how to approach the problem.

The desired scenario for me is stable read time under 100ms, I don't care about write perfomance since it's always happens
asynchronously. Also does not care pretty much about higher memory or cpu usage, as long as it's stable and predictable.

Which configuration should I use for my use case?

Thanks you for awesome work! Sorry for any grammar mistakes, I'm not native speaker.

[jarifibrahim]

Setting WithSyncWrites(false), for example, fixes issue, but memory usage keeps growing until OOM.

This option should have no effect since you're running in in-memory mode. This is valid only for normal mode.
There was a bug because of which we were creating SST files in in-memory mode and it was fixed in #1374. Maybe your data is being read from the SST which causes the read latency.

as I understood, this is somehow related to periodic compaction of entries, but I have no idea how to approach the problem.

Compactions are background tasks and they shouldn't affect the reads. Badger has snapshot isolation so your reads shouldn't be blocked by anything (except the last write that hasn't finished yet).

@hqhs Can you try out the master version of badger? It has some bugs but it has the in-memory bug fix as well. If you can run your application on master version of badger and check if the latency improves, that will be useful.

[hqhs]

Okey, I'll try it tomorrow and report the results.

[hqhs]

Issue is repodusable on latest master:

	github.com/dgraph-io/badger/v2 v2.0.1-rc1.0.20200630163423-09dfa663bbc3

What are out next steps? What badger metrics could help to investigate it? I'm thinking about turning on block/mutex profiling in isolated environment using golang's pprof.

[hqhs]

Also I'm seeing 2x spikes in CPU usage during moments of increased response time.

[jarifibrahim]

Can you collect a CPU profile? The CPU profile should show what's using all your CPU https://golang.org/pkg/net/http/pprof/ .

Please collect a CPU profile when the CPU usage is high.

[minhaj-shakeel]

Github issues have been deprecated.
This issue has been moved to discuss. You can follow the conversation there and also subscribe to updates by changing your notification preferences.