Rework the way file handle reads are performed

[jacobsa]

I'm currently working on sequential read performance in regions of large objects not cached locally. The throughput isn't great, at least in part because the default chunk size of 16 MiB used by gcsfuse is not large enough to amortize out the overhead of starting a read within a large object. (Cf. Google-internal thread go/vkhor).

Making the chunk size bigger is not a great solution, because it further exacerbates the problem of vastly over-reading for small random reads. In the end the user is going to be reading randomly or sequentially. In the former case we want to read basically exactly what they ask for, and in the latter case we probably want to start a read from their offset to infinity, and just use it until they stop reading. This is also a lot simpler.

So let's throw out the complicated caching layer with leases, etc. Instead:

• File handles maintain state in the form of the last read opened, the offset at which it is positioned, and how many bytes remain. If it can be reused, they reuse it.
• The first time a read is made through a handle where there is no HTTP response in progress or the response is positioned in the wrong place, the file upgrades the read to 1-2 MiB (a sane size for GCS, compare to 128 KiB reads from Linux and 1 MiB reads from OS X).
• If that read completes and the next read is at its end offset, the next request is for the read offset to infinity.

Two properties are interesting here:

• GCS never gets a request smaller than 1-2 MiB for a larger object. (This is already true today.)
• For a large sequential read, we make at most two requests. The first may be slowish, but the second will give us the full possible single-stream throughput.

If it turns out we need caching, we can later cache outside of this machinery by quantizing reads and caching what comes back.