os: ReadFile doesn't work on some /proc-like files

[bradfitz]

Go 1.24.0, plan9/386

Certain (many) plan9 files get generated on demand and don't have a known length ahead of time. The same situation exists with e.g. Linux /proc or /sys files.

os.ReadFile fails to read some of them.

For example, my /net/iproute on plan9 has length 0, according to ls -l:

term% ls -l /net/iproute
--rw-rw-r-- I 0 network gelnda 0 Mar. 1 16:45 /net/iproute

But I can cat it, and wc says it has 1215 bytes:

term% wc /net/iproute
     23     138    1215 /net/iproute

os.ReadFile, on the other hand, returns (511 bytes, nil) truncating it without any error.

What's happening is that os.ReadFile stats it, sees it has 0 length, assumes it's a special /proc-like file, starts with a minimum 512 byte buffer, and then issues a 512 byte read. The plan9 kernel returns with (511, nil). os.ReadFile then has 1 byte remaining between its capacity and length so it issues a one byte read to the kernel and the read returns (0, io.EOF).

Apparently that /proc-y special file doesn't like small buffers. It looks like the buffer size has to be at least large enough to fit one whole serialized route?

Perhaps os.ReadFile needs to say that at least for files that reported 0 length stat size at the beginning, we never issue a read system call with a buffer smaller than 512 bytes?

/cc @rsc @ianlancetaylor

[gopherbot]

Change https://go.dev/cl/654315 mentions this issue: os: guarantee min buffer size for ReadFile reads on /proc-like files

[gabyhelp]

Related Code Changes

• os: guarantee min buffer size for ReadFile reads on /proc-like files

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

Your CL says "Linux /proc and /sys filesystems can exhibit the same problems."

Out of curiosity, do you know of Linux files that have this problem (of a read with a too-small buffer returning 0 despite there being more to read)?

Most procfs files use Linux's seq_file internal helper, which buffers output so that the read buffer size is disconnected from the size of items within the file. So those at least I think should work OK.

[prattmic]

Ah, looks like https://go.dev/cl/266364 tests (TestReadFileProc) /proc/sys/fs/pipe-max-size.

Indeed, it looks like these sysctl files require a buffer large enough to hold the integer value.

[bradfitz]

@prattmic no recent examples come to mind. I just have vague memories of hitting this in the past, but it was possibly with a FUSE filesystem and not a /proc or /sys file from Linux itself.

[bradfitz]

@prattmic, I read all /proc and /sys files byte at a time on a Debian box with Linux 5.10.0 and while there were tons of small files that misbehaved with 1 byte reads (single integers followed by newline), ....

/proc/sys/net/netfilter/nf_log_all_netns: "0\n" (2) => "0" (1)
/proc/sys/net/nf_conntrack_max: "262144\n" (7) => "2" (1)
/proc/sys/net/unix/max_dgram_qlen: "512\n" (4) => "5" (1)
/proc/sys/user/max_cgroup_namespaces: "257157\n" (7) => "2" (1)
/proc/sys/user/max_fanotify_groups: "128\n" (4) => "1" (1)
...

... I couldn't find one that was over 512 bytes (when read in big chunks) that misbehaved when read in 1 byte chunks.

[bradfitz]

Now I'm curious to try the same experiment on FreeBSD, etc.

[bradfitz]

I just tested FreeBSD. FreeBSD has the same problem for its /proc/N/dbregs file.