Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

std::fs::canonicalize returns UNC paths on Windows, and a lot of software doesn't support UNC paths #42869

Open
radix opened this issue Jun 23, 2017 · 38 comments
Labels
A-io Area: std::io, std::fs, std::net and std::path C-bug Category: This is a bug. O-windows Operating system: Windows T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.

Comments

@radix
Copy link
Contributor

radix commented Jun 23, 2017

Hi, I hope this is the right forum/format to register this problem, let me know if it's not.

Today I tried to use std::fs::canonicalize to make a path absolute so that I could execute it with std::process::Command. canonicalize returns so-called "UNC paths", which look like this: \\?\C:\foo\bar\... (sometimes the ? can be a hostname).

It turns out you can't pass a UNC path as the current directory when starting a process (i.e., Command::new(...).current_dir(unc_path)). In fact, a lot of other apps will blow up if you pass them a UNC path: for example, Microsoft's own cl.exe compiler doesn't support it: rust-lang/cc-rs#169

It feels to me that maybe returning UNC paths from canonicalize is the wrong choice, given that they don't work in so many places. It'd probably be better to return a simple "absolute path", which begins with the drive letter, instead of returning a UNC path, and instead provide a separate function specifically for generating UNC paths for people who need them.

Maybe if this is too much of an incompatible change, a new function for creating absolute paths should be added to std? I'd bet, however, that making the change to canonicalize itself would suddenly make more software suddenly start working rather than suddenly break.

@Mark-Simulacrum Mark-Simulacrum added O-windows Operating system: Windows T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. labels Jun 23, 2017
@retep998
Copy link
Member

retep998 commented Jun 23, 2017

canonicalize simply asks the kernel to canonicalize the path, which it does and happens to return the canonical path as a root local device path. Root local device paths are capable of representing some paths which normal absolute paths are incapable of representing accurately (such as components being named ".." or "." or having "/" in them), along with the fact that they're the only way to call many system functions with paths longer than MAX_PATH (aside from being on some recent version of Windows 10 and having a certain registry key enabled). As a result having libstd automatically strip the prefix would definitely break some situations. However I'm definitely in favor of having more support in libstd for converting between these different kinds of paths so the user can easily turn a root local device path into an absolute path. I'd also love to have an fs::normalize which merely normalizes a possibly relative path into an absolute path without hitting the filesystem on Windows.

radix added a commit to arpeggiorpg/arpeggiorpg that referenced this issue Jun 23, 2017
@retep998
Copy link
Member

In reference to your commit which referenced this PR, normalization is not the same as merely joining the path onto the current directory due to drive relative paths being relative to the current directory on the given drive. For example given a drive relative path of C:foo, and an env::current_dir() of D:\bar, normalizing C:foo will have to get the current directory for C:\ and could end up being normalized to something radically different such as C:\i\dont\even\foo.

@radix
Copy link
Contributor Author

radix commented Jun 24, 2017

thanks, @retep998 :) it's just a hacked-together build tool that probably will eventually be replaced with something else, and I didn't intend to notify this ticket about my commit. but I guess it goes to show that a good way to get an absolute path in std would be really helpful.

@nagisa
Copy link
Member

nagisa commented Jun 25, 2017

Command::current_dir should be fixed. I doubt we will change canonlicalize.

Note, the i-wrong tag is only for the Command::current_dir, not the canonicalize behaviour.

@nagisa nagisa added the I-wrong label Jun 25, 2017
@retep998
Copy link
Member

Quick testing on Windows 10.0.15063 indicates that both SetCurrentDirectoryW and CreateProcessW are okay with a current directory starting with \\?\. They are not okay with a current directory that exceeds MAX_PATH regardless of \\?\. CreateProcessW is okay with the path to the process itself starting with \\?\ regardless of whether the first parameter is used. CreateProcessW is only okay with the path to the process exceeding MAX_PATH if it starts with \\?\ and is specified as the first parameter which Rust does not currently use. I tested std::process::Command::current_dir and it works as expected, accepting paths starting with \\?\ but rejecting any paths exceeding MAX_PATH.

@kornelski
Copy link
Contributor

kornelski commented Sep 5, 2017

Technically, AFAIK it is safe to strip the prefix in common simple cases (absolute path with a drive letter, no reserved names, shorter than max_path), and leave it otherwise.

So I think there's no need to compromise on correctness as far as stdlib goes. The trade-off is between failing early and exposing other software that doesn't support UNC paths vs maximizing interoperability with non-UNC software.

In an ideal world, I would prefer the "fail early" approach, so that limitations are quickly found and removed. However, Windows/DOS path handling has exceptionally long and messy history and decades of Microsoft bending over backwards to let old software not upgrade its path handling. If Microsoft can't push developers towards UNC, and fails to enforce this even in their own products, I have no hope of Rust shifting the Windows ecosystem to UNC. It will rather just frustrate Rust users and make Rust seem less reliable on Windows.

So in this case I suggest trying to maximize interoperability instead, and canonicalize to regular paths whenever possible (using UNC only for paths that can't be handled otherwise).

Also, careful stripping of the prefix done in stdlib will be much safer than other crates stripping it unconditionally (because realistically whenever someone runs into this problem, they'll just strip it unconditionally)

@ofek
Copy link

ofek commented Oct 9, 2017

@kornelski I completely agree. The current behavior is unexpected in my opinion.

@danielpclark
Copy link

danielpclark commented Oct 10, 2017

I hope this is helpful…

According to Microsoft:

Note File I/O functions in the Windows API convert / to \ as part of converting the name to an NT-style name, except when using the \\?\ prefix as detailed in the following sections.

Source: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx

And the Ruby language uses forward slashes for File paths and that works on Windows.

@kornelski
Copy link
Contributor

kornelski commented Nov 22, 2017

I've looked at this problem in detail. There are a few rules which need to be checked to safely strip the UNC prefix. It can be implemented as a simple state machine.

I've implemented that using public APIs, but because OsStr is opaque it's not nearly as nice as stdlib's implementation could have been:

https://lib.rs/dunce

So I'm still hoping canonicalize would do it automatically, because if it's done only for legacy-compatible paths there's no downside: all paths work for UNC-aware programs, and all paths that can work for legacy programs work too.

@mykmelez
Copy link

mykmelez commented May 9, 2018

Another example of this issue that I encountered in alexcrichton/cargo-vendor#71:

url::URL.to_file_path() returns a non-UNC path (even if the URL was initialized with a UNC path). And std::path::Path.starts_with() doesn't normalize its arguments to UNC paths. So calling to_file_path() on a file: URL and then comparing it to the output of canonicalize() via starts_with() always returns false, even if the two paths represent the same resource:

extern crate url;

use std::path::Path;
use url::Url;

fn main() {
	// Path.canonicalize() returns a UNC path.
	let unc_path_buf = Path::new(r"C:\Windows\System").canonicalize().expect("path");
	let unc_path = unc_path_buf.as_path();

	// Meanwhile, Url.to_file_path() returns a non-UNC path,
	// even when initialized from a UNC path.
	let file_url = Url::from_file_path(unc_path).expect("url");
	let abs_path_buf = file_url.to_file_path().expect("path");
	let abs_path = abs_path_buf.as_path();

	// unc_path and abs_path refer to the same resource,
	// and they both "start with" themselves.
	assert!(unc_path.starts_with(unc_path));
	assert!(abs_path.starts_with(abs_path));

	// But they don't "start with" each other, so these fail.
	assert!(unc_path.starts_with(abs_path));
	assert!(abs_path.starts_with(unc_path));
}

Arguably, to_file_path() should return a UNC path, at least when initialized with one. And perhaps starts_with() should normalize its arguments (or perhaps clarify that it compares paths, not the resources to which they refer, and thus does no normalization). Also, the mitigation for consumers of this API is straightforward: canonicalize() all paths you compare if you do so to any of them. So maybe the current behavior is reasonable.

Nevertheless, it does feel like something of a footgun, so it's worth at least documenting how it differs from that of some other APIs on Windows.

@retep998
Copy link
Member

retep998 commented May 9, 2018

comparing it to the output of canonicalize() via starts_with() always returns false, even if the two paths represent the same resource:

Comparing canonical paths is a footgun in general because it is the wrong thing to do! Things like hard links and so on mean that such comparisons will never be entirely accurate. Please don't abuse canonicalization for this use case.

If you want to tell whether two paths point to the same file, compare their file IDs! That's what same-file does and it works great!

@kornelski
Copy link
Contributor

but starts_with is not for is-file-a-file comparison, but is-file-in-a-directory check. There are no hardlinks involved (and AFAIK apart from private implementation detail of macOS time machine, no OS supports directory hardlinks).

@retep998
Copy link
Member

retep998 commented May 9, 2018

There are more ways than just \\?\C:\ and C:\ to represent the same path, so unfortunately any sort of file in directory check is a hard problem. For example, paths can refer to drives via names other than drive letters.

@nagisa
Copy link
Member

nagisa commented May 9, 2018 via email

@ChrisDenton
Copy link
Contributor

Yes, but the rules are documented.

Could you provide a link? I know of people reverse engineering the current algorithm but I'm not aware of Microsoft docs that describe the complete lossy transformation.

@ChrisDenton
Copy link
Contributor

I'm aware of those links. The first one is the most relevant and has had more details added over its lifetime but still does not document the path parsing algorithm in its entirety. I still think the most robust solution is to attempt to round-trip the path and see if it changes.

n-dusan added a commit to openlawlibrary/stelae that referenced this issue Nov 28, 2023
Running `cargo test` on Windows gets flaky when calling `canonicalize()`
on Path. This is because on Windows, the disk pathing for an absolute
path starts with `//?/C:/foo/bar/`. See Rust GitHub issue and discussion
here [1]. To resolve, add a helper function that remove the `//?/` from
the canonical path.

[1] - rust-lang/rust#42869
n-dusan added a commit to openlawlibrary/stelae that referenced this issue Jan 1, 2024
Running `cargo test` on Windows gets flaky when calling `canonicalize()`
on Path. This is because on Windows, the disk pathing for an absolute
path starts with `//?/C:/foo/bar/`. See Rust GitHub issue and discussion
here [1]. To resolve, add a helper function that remove the `//?/` from
the canonical path.

[1] - rust-lang/rust#42869
n-dusan added a commit to openlawlibrary/stelae that referenced this issue Jan 1, 2024
Running `cargo test` on Windows gets flaky when calling `canonicalize()`
on Path. This is because on Windows, the disk pathing for an absolute
path starts with `//?/C:/foo/bar/`. See Rust GitHub issue and discussion
here [1]. To resolve, add a helper function that remove the `//?/` from
the canonical path.

[1] - rust-lang/rust#42869
@alanwilter
Copy link

It's more than 5 year now, do we have a reasonable solution?

@ChrisDenton
Copy link
Contributor

We now have std::path::absolute which will work better if you don't need to resolve symlinks. We do need to work towards stabilizing it however.

Removing the \\?\ prefix still requires a third party crate. We could add a windows-only function to Path that attempts to remove it. We would however need to guard against changing the meaning of the path (which can happen in certain edge cases).

@alanwilter
Copy link

Thanks for that but it's still extremely frustrating. No matter what I try I still get:

\\?\UNC\applications2...

In my csv output file.

Using:

let file_path = path::absolute(path)?.to_string_lossy().to_string();

The expected value is \\applications2\... (it's a Windows network drive).

I tried dunce as well.

@ChrisDenton
Copy link
Contributor

What's the input path? absolute shouldn't add \\?\UNC\ unless it's already in the input path (or in the current directory). I think dunce has issues with some paths, you could try omnipath instead.

@alanwilter
Copy link

It's like H:/opensight/ where H: is mapped to \\applications2\sousaa.

@ChrisDenton
Copy link
Contributor

Hm, using std::path::absolute should resolve that as H:\opensight\ because it doesn't look up where it's mapped to.

@alanwilter
Copy link

alanwilter commented Jan 8, 2024

You're right, it worked, it just that I used the wrong binary. Many thanks!

mergify bot pushed a commit to dfinity/sdk that referenced this issue Jan 12, 2024
…3502)

Call `dunce::canonicalize` rather than `Path::canonicalize`.  On Linux/Macos, this just calls `std::fs::canonicalize()`, so no change there.

PathBuf::canonicalize on Windows generally returns Windows NT UNC paths like `\\?\C:\Users\you\`.  Lots of programs, including Microsoft's, don't work with these kinds of paths. For example, `npm` will print out a message something like `The current working directory is a UNC path.  I'm going to use C:\Windows instead.`.

dunce:
> ... converts paths to legacy format whenever possible, but leaves UNC paths as-is when they can’t be unambiguously expressed in a simpler way. This allows legacy programs to access all paths they can possibly access, and UNC-aware programs to access all paths.

rust-lang/rust#42869 (comment)

Part of https://dfinity.atlassian.net/browse/SDK-1343
max-heller added a commit to max-heller/mdbook-pandoc that referenced this issue Apr 6, 2024
…nonicalize()` (#84)

On Windows, `std::fs::canonicalize()` [produces a weird type of path
that cannot be used with
`Command::current_dir()`](rust-lang/rust#42869).
This switches to using the
[`normpath`](https://docs.rs/normpath/latest/normpath/trait.PathExt.html#tymethod.normalize)
crate, which should avoid these issues.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-io Area: std::io, std::fs, std::net and std::path C-bug Category: This is a bug. O-windows Operating system: Windows T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests