New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/sys/unix: add ParseDirent variant that doesn't allocate? #48161
Comments
Yes, please. I wrote almost the same unsafe code several years ago for the same purpose (counting a process's FDs without a bajillion allocations). |
We have similar logic too: https://github.com/beyondstorage/go-service-fs/blob/master/readdir_unix.go |
Turns out the inefficient way is (un)surprisingly common: |
Seems fine in principle, but what's the API that you suggest? |
Not exactly sure, but I'm sure it'll be slightly horrifying which is why I had to prepare y'all with this bug first. |
How about? // ParseNextDirent parses the next directory entry,
// returning the directory name and the number of consumed bytes.
// It returns 0 if the input appears truncated and -1 if it is invalid.
func ParseNextDirent(b []byte) (name string, consumed int) {
// NOTE: This function is inlineable so that the compiler can prove
// based on how the caller uses the returned name argument
// whether it needs to allocate or not.
s, n := parseNextDirent(b)
return string(s), n
}
func parseNextDirent(b []byte) (name []byte, consumed int) For consistency with the existing |
The menory layout of type Dirent struct {
Ino uint64 // 64-bit inode number
Off int64 // 64-bit offset to next structure
Reclen uint16 // Size of this dirent
Type uint8 // File type
Name [256]int8 // Filename (null-terminated)
_ [5]byte // Zero padding byte
} So I think return |
@dsnet, I don't want any strings in the API. We already have the high-level nice API if you want strings. And yes, we'd need the type and uint64 inode number. |
I recently wrote a function:
... to return how many file descriptors are open by the process. For Linux, the naive implementation is:
... but that allocates a bunch. Enough that it showed up disturbingly high in profiles. (worse: the more FDs open, the worse it is)
With some unsafe and Linux-specific stuff, you can get it down to zero allocations (e.g. tailscale/tailscale#2785).
Any objections to a
unix.ParseDirent
variant that parses dirents without allocating?Then https://pkg.go.dev/golang.org/x/tools/internal/fastwalk could also use it.
/cc @ianlancetaylor @tklauser @crawshaw @dsnet
The text was updated successfully, but these errors were encountered: