Stop using mmap in the store-gateway

[pracucci]

We want to get rid of mmap usage in the store-gateway, in order to definitely solve the problem of disk I/O hanging the process (see this and this for reference). This issue tracks the work to stop using mmap in the store-gateway.

---

Timeline

All dates subject to change, no refunds 😛

Nov 21

• Take ownership of the code and initial cleanup.
• Fix v1 index.Symbol lookup bug - ce19084
• Add tests for encoding.Decbuf - 4f74ccb to 9a60b74

Nov 28

• Add tests for indexheader.StreamBinaryReader - bda4548
• Change encoding.Reader interface to return errors - 863cbe0, 31de2b3, bfe4819
• Add tests for encoding.FileReader - 4e79107
• Fix any remaining encoding.FileReader issues - 685926f
• Add Seek() to encoding.Decbuf (use case 1, use case 2) - 90d25b5, d757b12
• Create feature flag to control use of indexheader.StreamBinaryReader 15b7a58
• Make indexheader.StreamBinaryReader goroutine safe (each instance of encoding.Decbuf needs a new reader) - fd66a99
• Switch uses of encoding.BufReader to encoding.FileReader
• Stream data from disk when computing CRC32 checksums rather than reading the entire segment into memory - dc46d17
• store-gateway integration test to use indexheader.StreamBinaryReader - 8bb4fdf
• Cleanup of dead code, error handling, lints.

Dec 5+

• Merge to main behind feature flag. Add alternate implementation of index-header reader that does not use mmap #3639
• Performance optimization (iterative).
  • Pooling of file handles - Pool file handles used by StreamBinaryReader components #3703
  • Make pooling of bufio.Reader not specific to each index-header - Store gateway mmap removal: use a shared pool of buffers across all Decbuf instances #3691
  • Apply []string optimization to readOffsetTable link 1 link 2 Not possible with mmap-less implementation

[pracucci]

We had a brief in-person conversation yesterday with Steve. The TL;DR is:

• The mmap-ed file is opened currently in newFileBinaryReader()
• The mmap-ed []byte usage entrypoints are:
  • The newFileBinaryReader() itself to pre-populate the in-memory data structures used to map symbols and postings offsets (in both cases we keep 1 entry every 32)
  • BinaryReader.LabelValues()
  • BinaryReader.PostingsOffset() (calling BinaryReader.postingsOffset())

Looking at the actual usage of the mmap-ed []byte, we could change the implementation of the following underlying components to work with file I/O syscalls instead of reading a mmap-ed []byte:

• newFileBinaryReader(): use syscalls without ever reading the whole index-header in memory
• encoding.Decbuf: write a new version of it which works with io.ReadSeeker (or similar interface)
• index.NewSymbols: write a new verson of it which works with io.ReadSeeker (or similar interface)

[56quarters]

Work on this will live in the following branch until it is ready to be merged to main: https://github.com/grafana/mimir/tree/56quarters/indexheader-mmap-removal

Based on the initial work done by @stevesg here d8e5035

[charleskorn]

Draft PR for this is now up at #3639