cache the entire bolt backend into memory before restoring index

[xiang90]

etcd has an in-mem secondary index for the entire db. Reading the entire boltdb without moving the db file into page cache is slow. It is OK for etcd to scan the entire file first and let os move all file into page cache. It increase the throughput of read from 6K/second to 2M/second with very little overhead (pre read ingthe whole db has a throughput around 300MB/second on SSD)

@benbjohnson Do you have any suggestion? If boltdb just supports warm-up, it would be great. But I still wonder why doing a seq read on a seq written db without warm-up is so slow.

[benbjohnson]

@xiang90 Even though the keys are written sequentially, bolt may write them randomly since it reuses old pages when available. One easy fix, if you're targeting Linux 2.6.23+, is to add an DB.MmapFlags parameter that would allow you to set MAP_POPULATE which should do the readahead. Would that work?

[xiang90]

@benbjohnson

Even though the keys are written sequentially, bolt may write them randomly since

Actually, this is a newly created db with all seq writes.

One easy fix, if you're targeting Linux 2.6.23+, is to add an DB.MmapFlags parameter that would allow you to set MAP_POPULATE which should do the readahead. Would that work?

I will give it a try.

[xiang90]

@benbjohnson

Thanks btw. I will let you know the result soon!

[benbjohnson]

@xiang90

this is a newly created db with all seq writes

Even with sequential writes, bolt will still have pages it will reuse because of the freelist being written and partially filled data pages. Those will get filled on the next transaction and the old half filled page will be available for reuse.

I will give it a try.

You'll need to add the MmapFlags as a field on bolt.Options since it's needed during bolt.Open(). Then copy that over to a DB.MmapFlags so it can be OR'd on each mmap() call. Let me know if you need any help with it.

https://github.com/boltdb/bolt/blob/master/db.go#L662-L675

[xiang90]

Fixed by #3865