Retrieval Optimization To Singularity

[hannahhoward]

Currently, retrievial through singularity is implemented through a read seeker abstraction. Since the io.ReadSeeker abstraction does not know about the over range parameters of the incoming HTTP request to motion, each call to the io.Reader.Read method is implemented by making a seperate range request into Singularity for the exact parameters of that individual read operation.

Under the hood, the implementation of HTTP.ServeContent that we employ uses io.Copy, which in turn executes read operations in the neigborhood of 32K, and extremely small request. Worse, inside Singularity, we duplicate this issue, turning read requests into 32k Filecoin retrievals, which is obviously extremely inefficient.

The proposed optimization here is as follows:

• modify Store.Get to take an optional list of configuration parameters
• make the first one some kind of "Ranges" struct
• parse the HTTP Range header -- this code to do this is surprisingly not easily available but there are examples in the HTTP ServeContent implementation (https://cs.opensource.google/go/go/+/refs/tags/go1.21.1:src/net/http/fs.go;l=887) and then I found a good example here (https://github.com/gotd/contrib/blob/v0.19.0/http_range/range.go).
  • one caveat, you need to get the size of the overall blob first to parse ranges correctly. So I'd recommend the Ranges struct look like:

type Ranges struct {
   Header string
}

type Range {
  Start uint64
  End uint64
}

func (r Range) Parse(size uint64) ([]Range, error) {
...
}

• Now you can pass your ranges to your read seeker, and change it's implementation as follows:
  • on first read, check if your current offset lies within one of the ranges
    • if so, make a pipe reader
    • start a go routine and execute a range request to singularity for the current position until the end of the current range -- pass it the pipe writer for a destination (sidebar: it's super annoying that the Swagger generated API takes a writer instead of returning a reader)
    • now save the pipe reader on the struct and use its read method to serve the current read operation
    • for subsequent read operations if an existing reader is already present, read from it (remove it from the struct if you hit EOF)
    • otherwise follow the operations above to execute another larger range request
    • wipe the saved reader whenever there's a Seek and cancel the HTTP request in progress.

Sidebar: all this works but it seems like we're in the guts a bit, and I wonder if ReedSeeker + HTTP.ServeContent is the right abstraction any more.

[hannahhoward]

To achieve the actual benefits of this you need to also implement data-preservation-programs/singularity#366, which is nearly identical in nature, and there will likely be a lot of code you can share.

[xinaxu]

@gammazero currently fixing a critical issue
data-preservation-programs/singularity#388

The retrieval optimization work is on going

[gammazero]

Reopened - finishing optimization to minimize separate requests to singularity within the same range.