proposal: os,path/filepath: make os.Readlink and filepath.EvalSymlinks on Windows (mostly) only evaluate symlinks

[bcmills]

Background

Today (as of Go 1.21.3), the documentation for path/filepath.EvalSymlinks (including on Windows!) states:

EvalSymlinks returns the path name after the evaluation of any symbolic links. If path is relative the result will be relative to the current directory, unless one of the components is an absolute symbolic link. EvalSymlinks calls Clean on the result.

Notably, EvalSymlinks is not documented to evaluate non-symlinks, just as one would not label a jar as “cinnamon” if it actually contains a mix of many ingredients.

Today, the documentation for os.Readlink states:

“Readlink returns the destination of the named symbolic link.”

#57766 notwithstanding, that documentation seems unambiguously intended to match the behavior of the POSIX readlink function.

Unfortunately, today the implementation of os.Readlink on Windows tries to do much more than what it says on the tin. Specifically, it tries to evaluate the named path to a “canonical” path using GetFinalPathNameByHandle, not just evaluate its symlink components to their targets.

• path/filepath: fix EvalSymLinks documentation #40180

Because it tries to do so much more, its implementation has some fairly serious bugs — and those bugs also come through in filepath.EvalSymlinks:

• path/filepath: EvalSymlinks of a directory which is mapped from volumeID (instead of drive letter) does not work #39786
• path/filepath: endless loop in EvalSymlinks because it resolves a link to the same path it asked for #40176

In addition, filepath.EvalSymlinks on Windows currently canonicalizes the path elements in case-insensitive paths to use the casing found in the filesystem. I believe that most existing callers of filepath.EvalSymlinks on Windows care primarily about evaluating symlinks and getting canonical casing for the path elements, and not at all about other kinds of canonicalization (because those other kinds are where most of the bugs are).

In the past, we have proposed to do ...something(?) (I'm not clear exactly what) with filepath.EvalSymlinks, and to add some new function — perhaps called filepath.Resolve — to evaluate “canonical” paths, providing the existing (aggressive, Windows-specific) behavior of the filepath.EvalSymlinks function as well as the Unix behavior of filepath.EvalSymlinks (which, to reiterate, only evaluates symbolic links).

• proposal: path/filepath: add Resolve, replacing EvalSymlinks #42201

#42201 was rejected due to a lack of consensus. The sticking point seems to have been the definition of “canonical”, which I agree is not well-defined — and I argue is not well-defined on Unix either, where we could imagine such things as platform-specific paths for mounted GUID volumes, hard-link targets, open files in /proc filesystems, and so on.

This proposal is more limited in scope, aided by the deeper GODEBUG support added for #56986.

Proposal

I propose that we add a GODEBUG setting (perhaps winsymlink?) that changes the behavior of os.Readlink and os.Lstat, with a default that varies with the Go version.

At old Go releases (winsymlink=0):

• os.Lstat would continue to consider both symlinks and mount points to be ModeSymlink.
• os.Readlink would continue to try to resolve both symlinks and mount points by calling windows.GetFinalPathNameByHandle and interpreting the result.
• Since the existing behavior of os.Readlink has little to do with the function's documented behavior, I don't believe it is meaningful to even try to define what constitutes a “bug” in that implementation, let alone try to fix it. I propose that we mostly leave it as-is, bugs and all.

At new Go releases (winsymlink=1):

• os.Lstat would only report ModeSymlink for IO_REPARSE_TAG_SYMLINK. All other reparse tags — including mount points — would be reported as either ModeIrregular or regular files (for DEDUP reparse points in particular) per proposal: os: treat all non-symlink reparse points as ModeIrregular on Windows #61893 and os: NTFS deduped file changed from regular to irregular #63429.
• os.Readlink would only evaluate the target of a symbolic link (that is, a reparse point with tag IO_REPARSE_TAG_SYMLINK), and return a value that is analogous to the value returned by POSIX readlink (as os.Readlink already does on POSIX platforms).
  • If the symlink refers to an absolute path, the absolute path should be returned as an Win32 path (not an NT path) when that is feasible (see proposal: os,path/filepath: make os.Readlink and filepath.EvalSymlinks on Windows (mostly) only evaluate symlinks #63703 (comment)).
  • If the symlink refers to a relative path, the returned path fragment should be relative to the directory containing the symlink.
• As a result, filepath.EvalSymlinks would only do two transformations:
  1. Evaluate symlinks (— “what it says on the tin”).
  2. Canonicalize the casing of path elements (to maintain compatibility where it is likely to matter).
     • In addition, this behavior should be added to the doc comment for EvalSymlinks.

Since filepath.walkSymlinks is written in a platform-independent way, I believe that fixing os.Readlink may suffice to fix filepath.EvalSymlinks to the above specification. However, if we discover any other bugs in filepath.EvalSymlinks, we should fix them to be in line with that.

In addition, syscall.Readlink should be deprecated on Windows, and golang.org/x/sys/windows.Readlink should be deprecated: the Win32 API itself does not define a readlink system call, and platform-agnostic abstractions belong in os, not syscall.

• The behavior of these Readlink functions may be left unchanged, or may be changed to also follow the winsymlink GODEBUG setting.

Compatibility

This proposal would provide backward-compatibility using a GODEBUG setting whose default varies with the Go version in use.

Where it changes the behavior of functions, it would change them to better align with the existing documentation for those functions — a kind of change explicitly allowed by the Go 1 compatibility guidelines.

This proposal does not attempt to define a notion of “canonical path” on Windows. Programs would remain free to define their own notion of “canonical” using lower-level syscalls like GetFinalPathNameByHandle if desired.

(attn @golang/windows; CC @networkimprov)

[qmuntal]

This proposal is interesting. I haven't realized till now that GetFinalPathNameByHandle is the root of most devil in filepath.EvalSymlinks.

I've done a quick prototype of your proposal (see CL 537295) and added a bunch of tests to make sure they fix #39786 and #40176. Everything seems to work fine with just one modification to the proposal: os.Readlink evaluate the target of a symbolic link AND replaces any NT prefix (\??\) with its Win32 counterpart. This means the following:

• Volumes with a device letter assigned get the NT prefix stripped. E.g. \??\C:\foo\bar => C:\foo\bar
• UNC NT path prefix is replaces with the UNC path prefix. E.g. \??\UNC\foo\bar => \\foo\bar
• Volumes without a drive letter (i.e. mount points) get the NT prefix replaced with the Win32 extended path prefix. E.g. \??\Volume{abc}\foo\bar => \\?\Volume{abc}\foo\bar.

Notice the first two bullets are already implemented by os.Readlink. This is necessary so the resulting path can interoperate with other Win32 APIs, which normally don't support the NT path prefix.

Bonus point: the new behavior seems to be an alternative fix for the bug that was originally fixed by making os.Readlink use GetFinalPathNameByHandle in CL 164201.

[bcmills]

[edit: I no longer think this; see later comments]

I think I would still rather have Readlink itself only read links, which I believe would also keep #30463 fixed (because as far as I can tell it doesn't involve any actual symlinks). On Unix platforms, readlink only reads one level of link — it doesn't do anything with earlier elements in the path.

But, given that I'm proposing for EvalSymlinks to keep its case-conversion behavior, I think it would be reasonable for EvalSymlinks to also convert the NT prefix to its Win32 counterpart. That would still make EvalSymlinks do more than just evaluate symlinks, but at least it confines the extra transformations to within one function.

Especially given #57766, I suspect that the vast majority of existing callers are using filepath.EvalSymlinks rather than os.Readlink directly, so moving the transformation into EvalSymlinks probably won't affect many (if any) real programs.

[bcmills]

From reading #30463, I believe that it was filed in response to containerd/continuity#113, which I believe is better addressed by https://go.dev/cl/463177.

(The current code in github.com/containerd/continuity appears to incorrectly assume that os.Readlink returns an absolute or relative version of the requested path, which is not what it actually does on Unix; see #57766. For that, I have filed containerd/continuity#232.)

Given that continuity.context.Walk is already (still) buggy, I don't think we should worry too much about maintaining the behavior established for it in #30463.

I think this proposal would address the underlying problem described in #29746 (comment) in a different way: namely, by not trying to resolve the C:\Data mount point as a symlink in the first place.

[qmuntal]

I think I would still rather have Readlink itself only read links, which I believe would also keep #30463 fixed (because as far as I can tell it doesn't involve any actual symlinks). On Unix platforms, readlink only reads one level of link — it doesn't do anything with earlier elements in the path.

Good enough 👍.

About changing os.Readlink to just read the link, what if we partially revert CL 164201 and implement the fix in syscall.Readlink instead (gated by godebug, if necessary). This way we will have just one definition of what reading a link does, and users calling syscall.Readlink will also benefit from having consitiency across OSes.

[bcmills]

Actually, on further reading I think you are correct that os.Readlink itself should return a win32 path: its argument is generally a win32 path, so its return value should be as well. (The exact format of the reparse data buffer should be an implementation detail.)

I think we should deprecate syscall.Readlink entirely on Windows. (As far as I can tell, the Win32 API does not actually provide a readlink system call at all. The syscall package is supposed to provide low-level system calls, not platform-independent wrappers — those belong in os, which is what I am proposing to fix.)

[bcmills]

Concretely: if a path with an NT prefix is passed to os.Readlink as its argument, it shouldn't make any particular effort to strip that prefix.

But if the argument refers to a symlink, and the symlink's substitute path includes an NT prefix, then os.Readlink should return it as a Win32 path instead.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Can someone briefly summarize the suggested changes here?

I may have misunderstood some comments above, but while I'm writing: We should not deprecate syscall.Readlink just because there is no Windows readlink system call. Strictly speaking there are no Windows system calls at all (that we are allowed to use), and the DLL calls that do exist definitely do not line up with the syscall API. Instead package syscall shoehorns Windows as best it can into a Unix-shaped boot. Even Open and Close are not Windows system calls or DLL calls. The fact that syscall.Readlink already exists and mostly matches os.Readlink suggests that it should remain that way, although it would be very good to deduplicate all that code into syscall (perhaps leaving fixLongPath in the os side).

[bcmills]

The changes are:

• os.Lstat and os.Readlink stop treating mount points as symlinks, and instead report them as irregular files.
• os.Stat continues to resolve the underlying directory for a mount point (as it does today).
• os.Readlink stops using GetFinalPathNameByHandleW to try to resolve volumes to drive letters or shares, and instead converts the paths purely lexically to Win32 paths.
• syscall.Readlink and windows.Readlink either get updated to match the new os.Readlink behavior sans fixLongPath, or become deprecated only on Windows. (I don't have a preference as to which, but I'm not sure how to plumb a GODEBUG setting that far down in the standard library.)
  • syscall.Readlink does not currently match os.Readlink on Windows because it doesn't transform NT volume paths to Win32 paths.
  • syscall.Readlink does not currently match syscall.Readlink on POSIX platforms because POSIX does not treat mount points as symlinks.
  • syscall.Readlink should remain non-deprecated on POSIX platforms either way.

[qmuntal]

os.Lstat and os.Readlink stop treating mount points as symlinks, and instead report them as irregular files.

Although we end up not flaging mount points as symlinks, IMO it is still usefull that os.Readlink supports mount points (IO_REPARSE_TAG_MOUNT_POINT). Doing a quick web search, I can see many os.Readlink calls explicitly using it for mount points, and other languages do also support them in their "read link" equivalent:

• dotnet: https://github.com/dotnet/runtime/blob/18ee31dc1e265750b792ec1b1ee57794e4a87b04/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Windows.cs#L637
• boost: https://github.com/boostorg/filesystem/blob/53eabaeabbf85fda2915a03612323df77fbaf23d/src/operations.cpp#L4138C7-L4138C7
• cpyhton: https://github.com/zooba/cpython/blob/941e81255e9e862838945c7e49d82d64e5bc9e64/Modules/posixmodule.c#L7859C52-L7859C52

About the nt to win32 path mapping, cpython has this gem: https://github.com/boostorg/filesystem/blob/53eabaeabbf85fda2915a03612323df77fbaf23d/src/operations.cpp#L2200.
Rust also does nt to win32 path mapping, but a little simpler:
https://github.com/rust-lang/rust/blob/5103173af1948430886534268d8f18b7e8386234/library/std/src/sys/windows/fs.rs#L542

[bcmills]

Hmm, so we would report mount points as ModeIrregular (instead of ModeSymlink), but allow os.Readlink to resolve them anyway if it is called explicitly?

That does appear to be consistent with what cpython does, at least:
https://github.com/zooba/cpython/blob/67444902a0f10419a557d0a2d3b8675c31b075a9/Python/fileutils.c#L1098-L1104

But Boost does uniformly treat them as symlinks:
https://github.com/boostorg/filesystem/blob/53eabaeabbf85fda2915a03612323df77fbaf23d/src/windows_tools.hpp#L74-L89
(OTOH, I have to wonder if Boost also stumbles over mount points that refer to directories. The assumption that mount point == directory junction seems incorrect, given the evidence in #29746 (comment). 😅)

Doing a quick web search, I can see many os.Readlink calls explicitly using it for mount points

I'm having trouble finding any calls to os.Readlink that are specifically looking to resolve mount points. Can you give some examples (or a query to find them)?

[qmuntal]

I'm having trouble finding any calls to os.Readlink that are specifically looking to resolve mount points. Can you give some examples (or a query to find them)?

You can use this query: https://github.com/search?q=junction+os.readlink+language%3AGo&type=code&l=Go. Some selected examples: 1, 2, 3. I acknowledge that these are not popular projects and the value of the os.Readlink call is dubious, but my gut feeling is that there is value on keeping mount point support in os.Readlink (with the necessary fixes).

[TBBle]

From the discussion of the current Python state, particularly python/cpython#82015 (comment), it's noted that when using a remote filesystem, symlinks and mount points are importantly different, as symlinks are resolved on the client (i.e. like POSIX) but the mount-point has to be resolved server-side when transitioning it, i.e. symlink-resolving code that reacts to seeing ModeSymlink during a walk and tries to canonicalise it may do the wrong thing when seeing a mount point e.g. a drive-letter reference would be referring to a drive on the server, but resolved to a drive on the client.

os.ReadLink being able to resolve any kind of name-surrogate is kind-of nice, but since it failed for volume mounts (the one place I'd tried to use it), I ended up using the Win32 API calls when I needed it anyway. At one point I had code that differentiated a mount point from symlink by checking that os.ReadLink returned its input. I would have changed that into winio.DecodeReparsePoint before merging, but we actually replaced the use of both symlinks and mount points with Bind Filter in later evolution of that PR, removing this distinction. Bindfilter may also be using reparse points (ProjFS does); but almost certainly not either of the tags we're talking about here, so it should just look like a directory/file unless calling low-level Windows-specific repase-point-supporting APIs.

So I have no strong feelings on os.ReadLink: it seems to me that in the Python equivalent of this discussion, it was kept able to handle both symlinks and mount/junctions simply because it already could and there wasn't a strong need seen to change that. So having it either able to read a mount point into its win32 path or fail with "That's not a symlink" seems fine to me, and the latter is consistent with Lstat not claiming it's a symlink either. (I don't think failing for volume mounts but succeeding for drive-letter file or directory junctions or similar is great... but since that's the current status quo, it's defensible.)

[bcmills]

A few observations from sampling the first page of the query results in #63703 (comment).

---

There are lots of clones of https://github.com/rclone/rclone/. It only appears to call os.Readlink for links it has translated (but erroneously assumes that links are absolute).

It only translates links for actual symlinks, and actually goes out of its way not to translate mount points that way:
https://github.com/rclone/rclone/blob/0548e61910856bfd719e232f89448213e35cb4e6/backend/local/local.go#L563-L568

---

There are a couple of clones of https://github.com/BishopFox/sliver. It claims: “There is an edge case where if you are trying to download a junction on Windows, you will get access denied”, and it appears to be trying to work around that. It isn't at all clear to me what causes that error or how the attempted workaround works, but I have a suspicion that it may be the same sort of case as in #29746 (comment).

That workaround was added recently by @RafBishopFox, so perhaps they can shed some light on the problem?

---

There are a couple of clones of https://github.com/spicetify/spicetify-cli, but from what I can tell the calls to os.Readlink are in Linux-specific code, and the code that deals with Windows junctions appears to be unrelated.

Its remaining calls to filepath.EvalsSymlinks are in GetExecutableDir, and in that case it seems like they need to evaluate a file-symlink to the executable rather than a directory-junction. (The program uses GetExecutableDir to locate another subdirectory, and reading that subdirectory will succeed regardless of whether the returned directory is a mount point or its underlying target.)

---

https://github.com/buildkite/agent uses os.Readlink on some Linux-specific paths (in devfs and procfs). Its only portable use of os.Readlink is for a symlink created by os.Symlink:
https://github.com/buildkite/agent/blob/a4f1da625523e62d394f2a4a42d437c1a2c3e575/clicommand/agent_start.go#L1278-L1301

---

https://github.com/cloudfoundry/groot-windows does use os.Readlink, but as far as I can tell only on an actual symlink to a regular file. For junctions, it uses its own getSymlinkDest helper that uses syscall.CreateFile to open and convert the reparse buffer itself.

---

There are a couple of clones of https://github.com/vanadium-archive/go.jiri, but from what I can tell none of them have been updated since 2016 — and they appear to only use actual symlinks (created by os.Symlink) anyway. (And it looks like a false-positive in the GitHub search: it uses the word “conjunction”, not “junction”. 😵‍💫)

---

https://github.com/tranvictor/jarvis uses os.Readlink in various file-reading utilities, but appears to do so incorrectly: it assumes that the path is absolute (compare #57766), and it looks like the subsequent calls would work correctly on link paths anyway (so the calls to Readlink are actually count-productive 😅).

---

https://github.com/petethacker/ll uses os.Readlink, but appears to only use its result for logging. It's not clear to me what the tool is intended to do (its README is quite sparse and https://pkg.go.dev/github.com/petethacker/ll shows no package documentaion).

---

https://github.com/nyaosorg/go-windows-junction uses os.Readlink for logging in tests and examples. That would indeed break if os.Readlink were changed to no longer resolve mount points. This is the first unambiguous use of os.Readlink for mount points I've seen!

---

https://github.com/onozaty/go-sandbox also uses os.Readlink to explicitly read a junction target. From what I can tell, the code in that repo is essentially entirely examples, so it isn't clear to me whether its usage represents testing out a real use-case or just experimenting with the existing semantics.

[bcmills]

Since the vast majority of calls to os.Readlink seem to be wrong anyway, I agree that it's probably reasonable to allow it to continue to read junction targets for the very few packages that do so intentionally and correctly (mainly https://github.com/nyaosorg/go-windows-junction).

But I do think it is important to arrange that we do not report junctions as symlinks and do not try to resolve them in filepath.EvalSymlinks. It seems clear to me that a lot of programs do use filepath.EvalSymlinks to try to resolve paths to something “canonical”, and those programs should generally resolve paths to a drive letter rather than a volume name that might not even be on the local machine.

[qmuntal]

Great summary @bcmills. To be clear, I'm completely onboarded on not reporting junctions as symlinks. I'm just doing some pushback to limit the blast radius of this change.

[RafBishopFox]

There are a couple of clones of https://github.com/BishopFox/sliver. It claims: “There is an edge case where if you are trying to download a junction on Windows, you will get access denied”, and it appears to be trying to work around that. It isn't at all clear to me what causes that error or how the attempted workaround works, but I have a suspicion that it may be the same sort of case as in #29746 (comment).

That workaround was added recently by @RafBishopFox, so perhaps they can shed some light on the problem?

I originally wrote that code sometime last year, so my recollection is a bit fuzzy. I was working on functionality in Sliver to create and send back an archive of a specified file or directory on a remote machine given filters (like *.docx to only bring back Word documents in a directory). In the case of a directory, I was getting access denied errors when trying to download files when the specified directory was a junction (reading the files from the directory pointed to by the junction did not result in an error). My workaround was to use os.Lstat to get information about the directory without resolving the junction. Then, if the resulting FileInfo object indicated that we were working with a symlink/junction, the code would resolve the junction and use the resolved path to read the files within the directory.

I cannot reproduce the behavior now, so maybe something else was going on with my test scenario. I remember it was an odd edge case, and for some reason, the junction was not being resolved automatically.

[bcmills]

@RafBishopFox, I wonder if the problem you were running into was one of the ones fixed by https://go.dev/cl/463177. 🤔

[RafBishopFox]

@RafBishopFox, I wonder if the problem you were running into was one of the ones fixed by https://go.dev/cl/463177. 🤔

Quite possibly! At the time, I think I chalked it up to a quirk with how I was trying to do things, and since the workaround was easy to implement, I did not spend a ton of time investigating it.

[rsc]

What is the current proposal and is there general agreement for it yet?