[feat] - buffered file reader

[ahrav]

Description:

This PR introduces a new readers package with a bufferedFileReader implementation for efficient random access reading, seeking, and closing operations. The bufferedFileReader combines the functionality of BufferedFileWriter for buffered writing and an io.ReadSeekCloser for random access reading and seeking. It also provides a Close method to release the buffer pool.

BUT Why??

The main reasons for introducing this new reader implementation are two-fold:

1. Memory Management: This will provide a way to create a custom reader that can accept an io.Reader with any amount of data without worrying about holding all the data in memory. This addresses the current issue with how we handle the result of “git cat-file” where we hold the entire contents in memory. It will also alleviate memory concerns in other scenarios where the reader we pass to handlers.HandleFile holds a lot of data.

2. Archiver Library Compatibility: The archiver library we use depends on the underlying reader implementing the seekReaderAt (io.ReaderAt & io.Seeker) interface. This new bufferedFileReader struct allows us to use it in all those places while leveraging our existing BufferedFileWriter when writing data.

Ok... but, why not use the existing DiskbufferReader??

While we have an existing diskbufferReader implementation which is a valid solution, we decided to introduce the new bufferedFileReader for the following reasons:

1. Efficiency for Small Files: The majority of files we need to process are under 10MB in size. Creating a temporary file for every single file, as done by the diskbufferReader, can be inefficient for small files. The bufferedFileReader avoids this overhead by storing small files in memory using the BufferedFileWriter.

2. Observability and Metrics: The BufferedFileWriter, which the bufferedFileReader leverages, provides observability through existing instrumented metrics. This allows us to monitor and track the performance of the new reader implementation more effectively.

3. Buffer Pool Optimization: The BufferedFileWriter is optimized to use an existing buffer pool to reduce memory allocation. This optimization can improve performance.

Key features:

• Utilizes BufferedFileWriter for buffered writing and data storage.
• Supports random access reading and seeking using an io.ReadSeekCloser.
• Implements io.Reader, io.Seeker, and io.ReaderAt interfaces.
• Close method releases the buffer back to the pool.
• Designed for efficient handling of large data with minimal memory usage.

Checklist:

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

[rosecodym]

Since we control all the relevant implementations, is it correct to say that we always expect this to succeed? And if so, should ReadCloser() be changed to return a *io.ReadSeekCloser so that we don't have to do this at runtime?

[ahrav]

I considered raising this question in the pr: If we modify the return type to io.ReadSeekCloser, we will also need to revise the ReadCloser method in the contentReader interface within gitparse.go. This adjustment left me somewhat uncertain, but I'm happy to make the change.

I also tend to get a little squimish around interface assertions. 😅

[rosecodym]

the test failure should be fixed on main