runtime: checkfds breaks on AIX and Solaris

[rolandshoemaker]

http://go.dev/cl/509020 broke the aix-ppc64 and solaris-amd64-oraclerel builders.

For AIX it appears that the secureFDs was previously not working as intended, as something at another level (presumably in the kernel) was already reopening FDs when the binary was SUID, meaning our calls to fcntl(F_GETFD) always returned the expected value with no error. Doing this when not in SUID returns -1 with an error of 0, which breaks our assumption that EBADF will be returned (also violates the IBM documentation, possibly suggesting we are not properly getting the error code).

For Solaris, I've not been able to acquire a buildlet to test yet, but the Oracle docs suggest nothing particularly obvious. Possible there is a similar thing happening as AIX?

Build logs:

• solaris-amd64-oraclerel: https://build.golang.org/log/5aa59f1efad3052841bc7d30603c2f29115d0daa
• aix-ppc64: https://build.golang.org/log/2c918acbd63b7da3ee0306178d3122f6e1e7f2bb

cc @ianlancetaylor @heschi

[bcmills]

Based on the platforms affected, I suspect a bug in syscall/exec_libc.go.

[rolandshoemaker]

Solaris appears to have the same behavior (returning -1, 0) as AIX.

@bcmills is probably right, although I don't immediately see anything obvious.

[ianlancetaylor]

Argh, the AIX gomote does not support ssh.

[bcmills]

Ah, here we go. https://pubs.opengroup.org/onlinepubs/9699919799/functions/exec.html:

If file descriptor 0, 1, or 2 would otherwise be closed after a successful call to one of the exec family of functions, implementations may open an unspecified file for the file descriptor in the new process image.

aix and solaris probably do that.

[bcmills]

(That does violate the POSIX requirement that “[t]he return value shall not be negative” for F_GETFD, but maybe this is an old bug that was overlooked because it happens so rarely.)

[ianlancetaylor]

But fcntl is returning -1. That definitely suggests that it is failing. https://www.ibm.com/docs/en/i/7.4?topic=ssw_ibm_i_74/apis/fcntl.html . But for some reason we are seeing an errno value of 0.

[bcmills]

Right. fcntl(…, F_GETFD, …) gets the flags associated with the file descriptor, but perhaps that fails because nothing explicitly associated any flags with it to begin with (it was opened implicitly by execve).

And then maybe it forgets to set errno on that path because the FD number is actually valid despite the lack of flags.

Perhaps we could confirm (or refute) that by trying some other operation on the descriptor?

[ianlancetaylor]

Hmmm, calling read on the descriptor does return 0.

[ianlancetaylor]

But calling open of /dev/null also returns 0. That is, it succeeds. I think we are indeed failing to return the correct errno value.

[ianlancetaylor]

Oh, the problem is that checkfds is called before minit. The call to minit is required to fetch the correct errno value.

That is also the problem on Solaris.

[gopherbot]

Change https://go.dev/cl/513215 mentions this issue: runtime2: don't check fcntl errno in checkfds on AIX and Solaris