proposal: path/filepath: add FromFS to safely convert a slash-separated path into an operating system path

[neild]

The filepath.FromSlash function sounds like it converts a /-separated path into an operating system path. What it actually does is replace every / in its input with os.PathSeparator.

FromSlash can map an input path into semantically different paths on different operating systems. For example, FromSlash("a\b") returns the filename a\b on Unix and the file b in the directory a on Windows. FromSlash("C:/foo") returns a relative path on Unix and an absolute path on Windows.

#56694 involves failures in the standard library to safely convert a non-platform-specific /-separated path into a semantically equivalent operating system path. The fix (https://go.dev/cl/455716) introduces an internal function to perform this operation. We should have a public API for this.

The proposal:

// FromFS converts a slash-separated path into an operating system path.
//
// FromFS returns an error if the path cannot be represented by the operating system.
// For example, paths containing '\' and ':' characters are rejected on Windows.
func FromFS(path string) (string, error)

FromFS rejects empty paths ("") and, on Windows, reserved device names such as NUL.

FromFS and IsLocal (#56219) are similar in that both involve performing safety checks on a potentially-untrusted input path. They serve different roles, however:

• FromFS takes a /-separated path in the form operated on by the path package and safely converts it to a semantically-equivalent operating system path.
• IsLocal takes an operating system path and reports whether it refers to something surprising.

[bcmills]

A concrete example of where this could be useful:
https://cs.opensource.google/go/x/build/+/master:cmd/buildlet/buildlet.go;l=1633-1670;drc=e73b639045d6176f6de4efb526c3d55475c065c5

[rsc]

I'm starting to get confused about which of these new filepath functions should be used when.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[neild]

I'm starting to get confused about which of these new filepath functions should be used when.

I am very sympathetic to that confusion.

Counting IsLocal added in 1.20, we have three new functions:

• filepath.IsLocal checks to see if a OS path is hinky. A hinky path is one that refers to something outside the current directory, or something weird like COM1 on Windows. I've run across a number of sanitization functions along the lines of strings.TrimPrefix(filepath.Clean("/"+p), "/") to ensure that a path is neither absolute nor contains .. components; IsLocal is intended to be a more robust and portable version of accomplishing the same goal. It's a somewhat specialized function that most users will not need, but there are so many subtleties (particularly surrounding Windows device names) that I'm still convinced it was worthy of inclusion into the standard library.
• filepath.FromFS converts a non-OS-specific /-separated path as one might find in a URL and converts it into an OS-specific path. Unlike FromSlash, it reports an error if the path can't be represented locally. Everyone who uses FromSlash today should use FromFS instead.
• filepath.IsReserved isn't particularly useful on its own, in my opinion, but both IsLocal and FromFS need to be aware of Windows reserved device names. A function which answers the limited question of whether a name is reserved seems like a useful building block to expose, especially since getting this right is quite tricky.

You should use FromFS when converting a /-separated path into an OS path.

You should use IsLocal to verify that a path from an untrusted source doesn't refer to anything surprising.

You should use IsReserved when building a sanitization function along these lines, if the above aren't sufficient.

[martin-sucha]

It is unclear to me why the function is called FromFS.

The proposal says that it supports paths that path package supports. The path package supports absolute paths. So it seems that filepath.FromFS would support absolute paths. However, I'm just guessing here since the proposal does not specify the desired behavior.

Then there is fs.ValidPath that rejects absolute paths.
Based on the name, I would expect that FromFS returns an error for any input where fs.ValidPath returns false.

If FromFS is supposed to support absolute paths and since FromFS is intended to replace FromSlash, perhaps it should just be called FromSlashV2? If FromFS is not supposed to support absolute paths, we should document that explicitly.

[rsc]

I don't think it's a given that everyone who uses FromSlash today should use FromFS.

Today, FromSlash converts slashes to the canonical form for the host OS but preserves the meaning of the existing path. So for example on Windows today, FromSlash("c:/foo") gives you c:\foo, which is the canonical form of its input. Similarly on a Mac you get FromSlash("c:/foo") is c:/foo, and in both cases the result of os.Open(p) and os.Open(FromSlash(p)) are the same.

It sounds like FromFS would not do that. You should use FromFS when the input is meant to be a "portable" slash-separated path as opposed to a slash-separated path interpreted according to the local OS. Programs that accept a file name on the command line but want to convert to native conventions should probably keep using FromSlash. The compiler does this sometimes for arranging canonical outputs and then inverting them. It should keep using FromSlash and ToSlash.

Maybe something reading from a zip file should use FromFS, but why not just have it use IsLocal+FromSlash instead of FromFS?

Or is it just programs implementing an fs.FS that need to use FromFS?

I don't think the exact scope is clearly defined yet.

[rsc]

It seems like we are stuck on the name here. I noticed that internal/safepath.FromFS is called with paths beginning with / (like /foo) but those are not actually io/fs paths. I'm not sure if the proposed FromFS rejects those or not, but safepath.FromFS does not.

Also it's probably too indirect a meaning to use "FS" here.

There are fundamentally two kinds of FromSlash: ones that are canonicalizing the OS interpretation and ones that are converting from "portable" to "local OS". The current FromSlash does the former. We need a name for the latter.

Maybe the From prefix is tripping us up and we should name this operation with some verb that can be the function name.

[ianlancetaylor]

filepath.Localize ?