-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
os: safer file open functions #67002
Comments
is this essentially https://pkg.go.dev/github.com/google/safeopen with Beneath -> In ? that also has a ReadFile / WriteFile variant which I'd use more then the create version. |
The design of this proposal is influenced by github.com/google/safeopen, but differs in a few areas. (Sorry, I really should have mentioned safeopen as prior art.) Of the three parts of this proposal:
|
Yes, please. When I was working on safe file operations and it turned out to be hard to do correctly without OS support. Without |
What, if any, changes would be made to "io/fs"? Ideally, there is a mirror of these APIs in that package. |
If we wanted to extend this proposal to package fs
// An OpenFile is a directory file whose entries may be opened with the Open method.
type OpenFile interface {
File
// Open opens the named file in the directory.
//
// When Open returns an error, it should be of type *PathError
// with the Op field set to "openat", the Path field set to name,
// and the Err field describing the problem.
//
// Open should reject attempts to open names that do not
// satisfy ValidPath(name), returning a *PathError with Err set to
// ErrInvalid or ErrNotExist.
Open(name string) (File, error)
} A more interesting question is I don't think we can change The interaction between Perhaps we should add a version of |
Perhaps the new names should include an At suffix to make clear to casual readers of the API that these are not the usual open system calls. Either way, the three new methods should probably reference some shared section of documentation on the concept of the |
I presume you mean the new method names? The functions have an "In" suffix. We could also include the In suffix on the methods; I waffled on whether it belongs there or not:
I'm trying to avoid the suffix "At" to make it clear that none of these calls are precisely |
Fair enough. Should the |
Probably, for consistency. |
Are we missing RemoveIn? |
We should probably have // RemoveIn removes the named file or (empty) directory.
// It applies the same constraints on files as [OpenFileIn].
// It otherwise behaves like [Remove].
func RemoveIn(parent, name string) error
// Remove removes the named file or (empty) directory
// in the directory associated with the file f.
// It applies the same constraints on files as [File.OpenFile].
func (f *File) RemoveIn(name string) error Perhaps also |
This proposal has been added to the active column of the proposals project |
Maybe it would be better if the parent was an fs.FS? Seems more widely applicable, if somewhat more complex. |
Note that Windows does not provide (AFAIK) an func RemoveIn(parent, name string) error
f, err := os.OpenIn(parent, name)
if err != nil {
return err
}
return syscall.SetFileInformationByHandle(f.Fd(), syscall.FileDispositionInfo, ...)
} |
As a user, when should I use os.Open vs os.OpenIn? Should I continue to default to os.Open, and only use OpenIn when I am actively avoiding a security issue, or should my default be OpenIn now? |
The If we want to add support for
I think that's fine. This proposal requires varying degrees of implementation depending on platform already. (Linux has the very nice openat2 with RESOLVE_BENEATH, platforms without an equivalent are going to require us to do more work to produce equivalent behavior.) If it's not possible to emulate unlinkat on Windows, that might be a problem, but it sounds like it should be possible.
You should use I don't know how to give comprehensive guidance on when to use one vs. the other; the two functions behave differently and you should use the one that suits your specific purposes. If you're writing a command-line tool that accepts an input filename from the user, you probably want to use |
The FS OpenFile looks good, yes. Somehow I skipped that comment, sorry. |
I don't really understand this. What is a program supposed to do if
which is exactly the "hazardous" behaviour you say you're trying to avoid. If the underlying platform truly has no equivalent to I think this could be addressed perfectly well in the docs by saying that some platforms (linux, windows, etc) provide extra guarantees around renamed files, and that others (plan9 and js) do not. |
The question is whether this is a reasonable fallback or not. In the case of In the case of Perhaps it's okay to say that I note also that if you don't need the |
I would recommend adding guidance in the documentation of the not |
Updated proposal, with comments on various changes arising from above discussion and working on implementation. The package os
// OpenFileIn opens the named file in the named directory.
//
// If the file contains relative path components (..), no component may
// refer to a location outside the parent directory. The file may not be
// "", an absolute path, or (on Windows) a reserved device name such as "NUL".
// The file may refer to the directory itself (.).
//
// If any component of the named file references a symbolic link
// referencing a location out of the parent directory,
// OpenFileIn returns an error.
//
// OpenFileIn otherwise behaves like OpenFile.
func OpenFileIn(parent, name string, flag int, perm FileMode) (*File, error)
// CreateIn creates or truncates the named file in the named parent directory.
// It applies the same constraints on files as [OpenFileIn].
// It otherwise behaves like [Create].
func CreateIn(parent, name string) (*File, error)
// Open opens the named file in the named parent directory for reading.
// It applies the same constraints on files as [OpenFileIn].
// It otherwise behaves like [Open].
func OpenIn(parent, name string) (*File, error) The package os
// OpenFileIn opens the named file in the directory associated with the file f.
//
// If the file contains relative path components (..), no component may
// refer to a location outside the parent directory. The file may not be
// "", an absolute path, or (on Windows) a reserved device name such as "NUL".
//
// If any component of the named file references a symbolic link
// referencing a location out of the parent directory,
// OpenFileIn returns an error.
func (f *File) OpenFileIn(name string, flag int, perm FileMode) (*File, error)
// CreateIn creates or truncates the named file in
// the directory associated with the file f.
// It applies the same constraints on files as [File.OpenFile].
func (f *File) CreateIn(name string) (*File, error)
// OpenIn opens the named file in the directory associated with the file f for reading.
// It applies the same constraints on files as [File.OpenFile].
func (f *File) OpenIn(name string) (*File, error) To the above, we add Open question: Should we add package os
// MkdirIn creates a new directory in the named parent directory
// with the specified name and permission bits (before umask).
// It applies the same constraints on files as [OpenFileIn].
// It otherwise behaves like [Mkdir].
func MkdirIn(parent, name string, perm FileMode) error
// MkdirIn creates a new directory in the directory associated with the file f.
// It applies the same constraints on files as [File.OpenFile].
func (f *File) MkdirIn(name string, perm FileMode) error
// RemoveIn removes the named file or (empty) directory.
// It applies the same constraints on files as [OpenFileIn].
// It otherwise behaves like [Remove].
func RemoveIn(parent, name string) error
// RemoveIn removes the named file or (empty) directory
// in the directory associated with the file f.
// It applies the same constraints on files as [File.OpenFile].
func (f *File) RemoveIn(name string) error
// StatIn returns a FileInfo describing the named file in the named parent directory.
// It applies the same constraints on files as [OpenFileIn].
// It otherwise behaves like [Stat].
func StatIn(parent, name string) (FileInfo, error)
// StatIn returns a FileInfo describing the named file in the directory associated with the file f.
// It applies the same constraints on files as [File.OpenFile].
func (f *File) StatIn(name string) (FileInfo, error) We add Open question: For the moment, we do not add any new optional interfaces to There are many existing APIs, both in and out of the standard library, that operate on an Open question: It seems likely to me that we're going to want more variations on package os
// DirFSIn returns a filesystem for the tree of files rooted at the directory dir.
// The directory dir must not be "".
//
// Open calls will resolve symbolic links, but return an error if any link points outside the directory dir.
//
// The returned filesystem implements [io/fs.FS], [io/fs.StatFS], [io/fs.ReadFileFS], and [io/fs.ReadDirFS].
func DirFSIn(dir string) *FS
type FS struct{}
func (fs *FS) Open(name string) (File, error)
func (fs *FS) Stat(name string) (FileInfo, error)
func (fs *FS) ReadFile(name string) ([]byte, error)
func (fs *FS) ReadDir(name string) ([]fs.DirEntry, error) The Open question: Should we add const (
// O_NOFOLLOW_ANY, when included in the flags passed to [OpenFile], [OpenFileIn],
// or [File.OpenFile], disallows resolution of symbolic links anywhere in the
// named file.
//
// O_NOFOLLOW_ANY affects the handling of symbolic links in all components
// of the filename. (In contrast, the O_NOFOLLOW flag supported by many
// platforms only affects resolution of the last path component.)
//
// O_NOFOLLOW_ANY does not disallow symbolic links in the parent directory name
// parameter of [OpenFileIn].
//
// O_NOFOLLOW_ANY does not affect traversal of hard links, Windows junctions,
// or Plan 9 bind mounts.
//
// On platforms which support symbolic links but do not provide a way to
// disable symbolic link traversal (GOOS=js), open functions return an error
// if O_NOFOLLOW_ANY is provided.
O_NOFOLLOW_ANY int = (some value)
) Open question: How should we handle Consider the following directory tree:
On the Unix command line, if we If we open the current directory and The On Windows, things are confusing (and I'm still trying to understand what's going on under the hood): Using
It appears that The question is: What should
My current inclination is the first option above: Permit both symlinks and I can, however, see a good argument for disallowing Open question: How should we handle platforms without GOOS=js does not permit implementing GOOS=js and GOOS=plan9 do not permit implementing I've argued above for supporting |
This is rather extensive API. Perhaps a separate package from os would be better? Maybe os/in? |
On a very minor note, Plan 9 can be considered to implement O_NOFOLLOW_ANY because there are no symlinks on Plan 9 at all. |
More generally, I understand the motivation here, but the amount of new API is a bit daunting. I think we need to keep thinking about reducing the total amount of API. It seems like there needs to be some type representing the constrained file system. For this message, let's call it a Dir. It would be defined like:
All the top-level convenience things like os.OpenIn can be left out. Code can use OpenDir followed by the operation it wants. That at least feels like a more manageable amount of API. I have been thinking for a while and have not come up with a name like more than Dir. It's certainly not perfect, and OpenDir would need a doc comment explaining that it's not opendir(3), but it's not bad. |
The concern I had about |
@cyphar, can you point to an explanation of the |
No change in consensus, so accepted. 🎉 The proposal is: package os
// Root represents a directory.
//
// Methods on Root can only access files and directories within that directory.
// If any component of a file name passed to a method of Root references a location
// outside the root, the method returns an error.
// File names may reference the directory itself (.).
//
// File names may contain symbolic links, but symbolic links may not
// reference a location outside the root.
// Symbolic links must not be absolute.
//
// Methods on Root do not prohibit traversal of filesystem boundaries,
// Linux bind mounts, /proc special files, or access to Unix device files.
//
// Methods on Root are safe to be used from multiple goroutines simultaneously.
//
// On most platforms, creating a Root opens a file descriptor or handle referencing
// the directory. If the directory is moved, methods on Root reference the original
// directory.
//
// Root's behavior differs on some platforms:
//
// - When GOOS=windows, file names may not reference Windows reserved device names
// such as NUL and COM1.
// - When GOOS=js, Root is vulnerable to TOCTOU (time-of-check-time-of-use)
// attacks in symlink validation, and cannot ensure that operations will not
// escape the root.
// - When GOOS=plan9 or GOOS=js, Root does not track directories across renames.
// On these platforms, a Root references a directory name, not a file descriptor
type Root struct { ... }
func OpenRoot(dir string) (*Root, error)
func (*Root) FS() fs.FS
func (*Root) OpenFile
func (*Root) Create
func (*Root) Open
func (*Root) OpenRoot
func (*Root) Close
func (*Root) Mkdir
func (*Root) Remove
func (*Root) MkdirAll
func (*Root) RemoveAll
func (*Root) Chmod
func (*Root) Chown
func (*Root) Chtimes
func (*Root) Lchown
func (*Root) Lstat
func (*Root) Readlink
func (*Root) Rename
func (*Root) Stat
func (*Root) Symlink
func (*Root) Link
func (*Root) Truncate
func OpenInRoot(dir, name string) (*File, error) {
r, err := OpenRoot(dir)
if err != nil { return nil }
return r.Open(name)
} |
Change https://go.dev/cl/627076 mentions this issue: |
Change https://go.dev/cl/627475 mentions this issue: |
Change https://go.dev/cl/629518 mentions this issue: |
Change https://go.dev/cl/629555 mentions this issue: |
Change https://go.dev/cl/629519 mentions this issue: |
Change https://go.dev/cl/629698 mentions this issue: |
My current intent is to submit a subset of this proposal for 1.24, with the remainder following in 1.25. 1.24 will contain: type Root struct { ... }
func OpenRoot(dir string) (*Root, error)
func (*Root) OpenFile
func (*Root) Create
func (*Root) Open
func (*Root) OpenRoot
func (*Root) Close
func (*Root) Mkdir
func (*Root) Remove
func (*Root) Lstat
func (*Root) Stat
func OpenInRoot(dir, name string) (*File, error) That will leave the following functions for 1.25: func (*Root) MkdirAll
func (*Root) RemoveAll
func (*Root) Chmod
func (*Root) Chown
func (*Root) Chtimes
func (*Root) Lchown
func (*Root) Readlink
func (*Root) Rename
func (*Root) Symlink
func (*Root) Link
func (*Root) Truncate The implementation in 1.24 will support all our ports (with the caveats mentioned above for GOOS=js and GOOS=plan9), but not does not take advantage of platform-specific features such as Linux's RESOLVE_BENEATH and Darwin's O_NOFOLLOW_ANY which allow for a more efficient implementation. That will also be a task for 1.25. |
Add os.Root, a type which represents a directory and permits performing file operations within that directory. For #67002 Change-Id: I863f4f1bc320a89b1125ae4237761f3e9320a901 Reviewed-on: https://go-review.googlesource.com/c/go/+/612136 Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Quim Muntal <quimmuntal@gmail.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
For #67002 Change-Id: Ibbf44c0bf62f53695a7399ba0dae5b84d5efd374 Reviewed-on: https://go-review.googlesource.com/c/go/+/627076 Reviewed-by: Quim Muntal <quimmuntal@gmail.com> Reviewed-by: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
For #67002 Change-Id: I0903f45dbb4c44ea0280c340c96c5f3c3c0781be Reviewed-on: https://go-review.googlesource.com/c/go/+/627475 Reviewed-by: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Quim Muntal <quimmuntal@gmail.com>
For #67002 Change-Id: Ib687c92d645b9172677e5781a3e51ef1a0427c30 Reviewed-on: https://go-review.googlesource.com/c/go/+/629518 Reviewed-by: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
For #67002 Change-Id: If919ee8a5e3d90e91c7848330762e3254245fba1 Reviewed-on: https://go-review.googlesource.com/c/go/+/629555 Reviewed-by: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Please see the updated proposal in #67002 (comment)
Directory traversal vulnerabilities are a common class of vulnerability, in which an attacker tricks a program into opening a file that it did not intend. These attacks often take the form of providing a relative pathname such as
"../../../etc/passwd"
, which results in access outside an intended location. CVE-2024-3400 is a recent, real-world example of directory traversal leading to an actively exploited remote code execution vulnerability.A related, but less commonly exploited, class of vulnerability involves unintended symlink traversal, in which an attacker creates a symbolic link in the filesystem and manipulates the target into following it.
I propose adding several new functions to the os package to aid in safely opening files with untrusted filename components and defending against symlink traversal.
It is very common for programs to open a file in a known location using an untrusted filename. Programs can avoid directory traversal attacks by first validating the filename with a function like
filepath.IsLocal
. Defending against symlink traversal is harder.I propose adding functions to open a file in a location:
The
OpenFileIn
,OpenIn
, andCreateIn
family of functions safely open a file within a given location, defending against directory traversal, symlinks to unexpected locations, and unexpected access to Windows device files.All modern Unix systems that I know of provide an
openat
call, to open a file relative to an existing directory handle (FD). Windows provides an equivalent (NtCreateFile
withObjectAttributes
including aRootDirectory
). Of the supported Go ports, I believe only js and plan9 do not supportopenat
or an equivalent.I propose adding support for
openat
-like behavior toos.File
:Like the top-level
CreateIn
,OpenIn
, andOpenFileIn
, the methods defend against accessing files outside the given directory. This is unlike the default behavior ofopenat
, which permits absolute paths, relative paths outside the root, and symlink traversal outside the root. (It corresponds to Linux'sopenat2
with theRESOLVE_BENEATH
flag.)A property of
openat
is that it follows a file across renames: If you open a directory, rename the directory, and useopenat
on the still-open FD, access is relative to the directory's new location. We cannot support this behavior on platforms which don't haveopenat
or an equivalent (plan9 and js). We could fall back to operating purely on filenames, such thatf.OpenIn(x)
is equivalent toos.OpenIn(f.Name(), x)
. However, this seems potentially hazardous. I propose, therefore, thatFile.CreateIn
,File.OpenIn
, andFile.OpenFileIn
return anerrors.ErrUnsupported
error on these platforms.The above functions defend against symlink traversal that leads outside of the designated root directory. Some users may wish to defend against symlink traversal entirely. Many modern operating systems provide an easy way to disable symlink following: Linux has
RESOLVE_NO_SYMLINKS
, Darwin hasO_NOFOLLOW_ANY
, and some other platforms have equivalents.I propose adding support for disabling symlink traversal to the
os
package:O_NOFOLLOW_ANY
may be passed toOpenFile
,OpenFIleIn
, orFile.OpenFIle
to disable symlink traversal in any component of the file name. ForOpenFileIn
, symlinks would still be permitted in the directory component.On platforms which do not support the equivalent of
O_NOFOLLOW_ANY
/RESOLVE_NO_SYMLINKS
natively, theos
package will use successiveopenat
calls withO_NOFOLLOW
to emulate it. On platforms with noopenat
(plan9 and js), open operations will return an error whenO_NOFOLLOW_ANY
is specified.The text was updated successfully, but these errors were encountered: