os/exec: improved API and default behavior for the PWD env variable #50599
Description
Background
On POSIX platforms, the PWD environment variable represents (emphasis mine) “an absolute pathname of the current working directory.”
The specific value of PWD is important if the current working directory can be reached through multiple absolute paths, as is the case for subdirectories of /tmp on macOS. It affects the behavior of os.Getwd (see #8400), which can in turn affect both the error and non-error output of programs, including behaviors like paths in debug metadata (which can invalidate build caches if skewed) and parent-directory lookup (important for, among other things, identifying the current Go workspace and module). A PWD in /tmp arises naturally in any Go test that executes a subprocesses within a directory obtained by calling (*testing.T).TempDir, which is (or ought to be!) common for tests of command-line binaries.
Failing to set PWD appropriately when setting the Dir has led to a number of subtle bugs within the Go project, such as:
- x/build: "unexpected stale targets" on darwin builders #33598 (“unexpected stale targets” on macOS)
- cmd/go: 'go generate' fails to update PWD variable #43862 (unexpected output in
go:generate)
Moreover, setting PWD correctly is not trivial: PWD is required to be absolute, but the Dir field may be relative, leading to bugs like #46832 when the two do not match.
Proposal
I propose two ergonomic improvements to the exec.Cmd API:
-
If the
Envfield of anexec.Cmdis nil and itsDirfield is non-empty on a platform that usesPWD, the subprocess'sPWDvariable should be implicitly set to an appropriate absolute path (computed fromos.Getwd()ifDiris relative) when the command is started.- If the
Dirfield is empty, thePWDvariable (or lack thereof) would continue to be inherited from the Go process, as it is today. - If the
Envfield is non-nil, itsPWDvariable (or lack thereof) would continue to be used, as it is today.
- If the
-
A new method,
(*Cmd).Environ, analogous toos.Environ(), should return a copy of the environment that would be used by the command in its current configuration: either a copy of the current value of theEnvfield (if non-nil), oros.Environ()augmented with thePWDand/orSYSTEMROOTvariables that would otherwise be used.
Part (1) alone would set a more appropriate PWD for existing commands that set Dir and do not explicitly modify the environment — that is, commands for which the author almost certainly hasn't considered the PWD interaction at all.
Part (2) would make it easier to set up a command that runs in a particular directory with a small number of overridden environment variables — the common case when environment variables other than PWD are changed at all.
Compatibility
This proposal would set an updated PWD variable for some commands that today receive a stale one (or none at all) inherited from the parent Go process.
It would have no effect on commands that have an empty Dir field or a non-nil Env field.
It may change the observable behavior of commands that have a nil Env but are sensitive to the value of PWD, including commands that enumerate all variables in the environment. For commands that are sensitive to the value of the PWD variable, it should nearly always be a strict improvement: the semantics of the PWD variable are explicitly described in the POSIX standard, and this change would bring the behavior in line with that standard on POSIX-like platforms.
The only case I can conceive of in which a user explicitly wants to inherit a missing or mismatched PWD is when testing the behavior of a command-line binary in a non-POSIX-compliant environment. However, for that rare case it would be straightforward for the user to explicitly set the Env field to a slice with the desired PWD variable, including no such variable at all.
Limitations
This proposal would not fix the PWD variable for commands that explicitly set Env with an incorrect or missing PWD today (such as from an explicit call to os.Environ()). That is a deliberate choice to bias toward maintaining compatibility, but may limit the scope of improvements.
Alternatives considered
- We could set the
PWDvariable more or less aggressively:
a. We could omit thePWDfrom the command's environment if the Go process itself has noPWDvariable, even if the platform normally usesPWD.
b. We could setPWD(on platforms that use it) wheneverDiris non-empty andEnvdoes not include an explicit entry for it, analogous to the conditions under which we setSYSTEMROOTon Windows today. That would fix commands that set explicit variables but accidentally omitPWD, but would remove the ability to deliberately leavePWDcompletely unset when using theDirfield. (At best, it would be set to the empty string.)
c. We could setPWD(on platforms that use it) wheneverDiris non-empty andEnvincludes aPWDthat is not an absolute path that resolves toDir. That would enforce compliance with the POSIX standard for the variable, but may be expensive to detect, and would remove the ability to deliberately use a noncompliantPWD.
In my opinion, options (a) or (b) would be reasonable but (c) would be too invasive.
- Instead of the proposed
Environmethod, we could add aSetDirmethod that both sets theDirfield and appends an appropriate value to theEnvfield, populating theEnvfield usingos.Environ()if it was previously nil.
I worry that appending to the existing Env field might be a bit too subtle: it wouldn't be intuitively obvious to me whether SetDir appends to Env (potentially overwriting entries in aliased slices), or to Env[:len(Env):len(Env)] (producing a possibly-unexpected new allocation). That could be addressed with documentation, but I think the Environ method is easier to describe, especially by analogy to os.Environ.