proposal: net/http: optimize http.Dir when files are not modified and add GODEBUG to preserve old behavior

[qiulaidongfeng]

Proposal: net/http: optimize http.Dir when files are not modified and add GODEBUG to preserve old behavior

Author(s): @qiualidongfeng

Last updated: 2023-11-19

Abstract

If there are no errors, http.Dir calls os.Open once every time Open is called. If the file has not changed, performance can be improved by reducing system calls by caching *os.File and []byte read from *os.File.

Background

Background as described above.
Proposals to optimize http.Dir.

Proposal

http.Dir.Open When the file modification time does not change, use the cached *os.FIle and []byte.
Add the GODEBUG environment variable httpdircache, go1.23 defaults to 1, enabling the behavior after the proposal implementation. Setting the value to 0 restores the previous behavior.

Rationale

1, directly change the behavior, when the file modification time is unchanged but the content changes, backward incompatible.

2, Third-party libraries, such as https://github.com/qiulaidongfeng/cachefs , This is a prototype written in a third-party library before sending CL to the standard library. The advantage is that it can be used less than go1.23, and the disadvantage is that the code needs to be manually modified to use.

Compatibility

100% backward compatibility with godebug=httpdircache=0.

Implementation

I will send CL to implement the proposal, which is too close to go1.22 code freezing, so this proposal is for go1.23.

[gopherbot]

Change https://go.dev/cl/543636 mentions this issue: net/http: optimize http.Dir when files are not modified and add GODEUG to preserve old behavior

[seankhliao]

This significantly changes the memory propfile of a server using http.Dir, it becomes much easier to trigger an OOM DoS by requesting a large number of files that all get cached.

[qiulaidongfeng]

This significantly changes the memory propfile of a server using http.Dir, it becomes much easier to trigger an OOM DoS by requesting a large number of files that all get cached.

one A path should cache only one copy of *os.File and []byte into sync.Map. If this uses too much memory, I can modify the proposal and CL to cache only small files.

[rittneje]

As @seankhliao noted, caching can consume potentially unbounded memory. In addition, clients may want to be able to configure the cache policy, like how many files are cached, which files are cached, how long they are cached for, etc. It does not make sense for the standard library to impose one solution for the entire binary, nor does it make sense for the existing un-cached behavior to essentially be treated as deprecated, both of which are implied by the use of GODEBUG.

This should be a separate http.FileSystem implementation that clients can opt into. (And at that point it does not need to even be part of the standard library.)