[Perf Improvement] Random reads should re-use previous reader even if existing reader can serve the request

[ashmeenkaur]

Description

Random reads were not working as intended; they did not reuse the existing reader, even when existing reader had the data to serve the request.

This issue occurred because we were using the Read method (instead of ReadFull). According to the documentation, this method does not wait for the entire data to be available, preventing the reader's start offset from being moved by the required number of bytes.

Fix: We should use io.ReadFull() instead.

Before this change:

time="06/02/2024 08:34:55.221154" severity=TRACE message="fuse_debug: Op 0x0000016e        connection.go:415] <- ReadFile (inode 2, PID 367041, handle 27, offset 41943040, 16384 bytes)"
time="06/02/2024 08:34:55.221364" severity=TRACE message="gcs: Req             0x2c: -> Read(\"1\", [31457280, 32505856)) (330.506132ms): OK"
time="06/02/2024 08:34:55.221400" severity=TRACE message="gcs: Req             0x2d: <- Read(\"1\", [41943040, 42991616))"
time="06/02/2024 08:34:55.576079" severity=TRACE message="fuse_debug: Op 0x0000016e        connection.go:497] -> OK ()"
time="06/02/2024 08:34:55.576375" severity=TRACE message="fuse_debug: Op 0x00000170        connection.go:415] <- ReadFile (inode 2, PID 367041, handle 27, offset 41959424, 32768 bytes)"
time="06/02/2024 08:34:55.751697" severity=TRACE message="fuse_debug: Op 0x00000170        connection.go:497] -> OK ()"
time="06/02/2024 08:34:55.751938" severity=TRACE message="fuse_debug: Op 0x00000172        connection.go:415] <- ReadFile (inode 2, PID 367041, handle 27, offset 42012672, 20480 bytes)"
time="06/02/2024 08:34:55.752120" severity=TRACE message="gcs: Req             0x2d: -> Read(\"1\", [41943040, 42991616)) (530.710831ms): OK"
time="06/02/2024 08:34:55.752174" severity=TRACE message="gcs: Req             0x2e: <- Read(\"1\", [42012672, 43061248))"
time="06/02/2024 08:34:56.002057" severity=TRACE message="fuse_debug: Op 0x00000172        connection.go:497] -> OK ()"

After this change:

time="06/02/2024 09:58:56.900148" severity=TRACE message="fuse_debug: Op 0x0000015a        connection.go:415] <- ReadFile (inode 2, PID 421837, handle 26, offset 41943040, 16384 bytes)"
time="06/02/2024 09:58:56.900362" severity=TRACE message="gcs: Req             0x29: -> Read(\"1\", [31457280, 32505856)) (256.173234ms): OK"
time="06/02/2024 09:58:56.900407" severity=TRACE message="gcs: Req             0x2a: <- Read(\"1\", [41943040, 42991616))"
time="06/02/2024 09:58:57.124098" severity=TRACE message="fuse_debug: Op 0x0000015a        connection.go:497] -> OK ()"
time="06/02/2024 09:58:57.124378" severity=TRACE message="fuse_debug: Op 0x0000015c        connection.go:415] <- ReadFile (inode 2, PID 421837, handle 26, offset 41959424, 32768 bytes)"
time="06/02/2024 09:58:57.284816" severity=TRACE message="fuse_debug: Op 0x0000015c        connection.go:497] -> OK ()"
time="06/02/2024 09:58:57.285058" severity=TRACE message="fuse_debug: Op 0x0000015e        connection.go:415] <- ReadFile (inode 2, PID 421837, handle 26, offset 42012672, 20480 bytes)"
time="06/02/2024 09:58:57.445157" severity=TRACE message="fuse_debug: Op 0x0000015e        connection.go:497] -> OK ()"

Link to the issue in case of a bug fix.

Internal bug b/322770463

Testing details

1. Manual - Manually verified from logs
2. Unit tests - added
3. Integration tests - Ran via KOKORO
4. Perf tests -