Clients of the new path need to be updated for non-utf8 paths

[lilyball]

The new Path module (coming soon) represents POSIX paths as vectors (#7225) of non-NUL bytes instead of as ~str. This is because Linux allows for non-unicode paths.

Unfortunately, there are a lot of clients of Path, including related functionality like Process arguments and environment variables, that still use &str. These need to be updated to handle &[u8] so they can represent non-unicode Paths.

All of the known locations where this is an issue are marked up by // FIXME lines referencing this issue. Note that in the case of Process args and environments, these comments are at the usage locations rather than the implementation.

Sub-tasks that have been filed for various components:

• glob("*") does not support matching non-utf8 filenames #11916 glob("*") does not support matching non-utf8 filenames
• Process::new and dynamic_library::open_external should take &[u8]s, not Paths or ~strs #11650 Process::new and dynamic_library::open_external should take &[u8]s, not Paths or ~strs

[SimonSapin]

Path::display seems to still assume UTF-8. Is that something that needs to be fixed?

[alexcrichton]

When formatted with fmt::Default, the display object will replace all non-utf8 sequences with the "unicode replacement character" which I believe is \ufffd or something like that. The idea with it was to provide an easy way to print the string in a readable fashion, but not to make it usable to other functions (because it's not the original path).

[SimonSapin]

@alexcrichton , so display will have lots of replacement characters for byte-based non-UTF-8 filesystems. Is that considered good enough?

[alexcrichton]

It's considered good enough in the sense that Path::new(format!("{}", path.to_str())) != path. The display is not expected to be equal to the path itself, and is intended purely for display purposes when something expects a number of utf-8 bytes to show.

[SimonSapin]

Yes, not round-tripping in all cases for display is fine. But in cases the filesystem is known to have a different encoding (see Python’s sys.getfilesystemencoding()), shouldn’t we try and use that?

[alexcrichton]

Right now we don't have a great story of encoding/decoding strings in and out of UTF-8, but it would be pretty awesome once that story gets fleshed out to support re-encoding as utf-8 if we know the filesystem is different.

[lilyball]

I agree. Being able to support other encodings based on the known filesystem encoding would be great. But that can be added later, once it's actually possible to do so.

[SimonSapin]

@kballard Thanks, I think that answers what I wanted to know. Supporting various character encodings is a separate issue. (See my API proposal on the wiki and related discussion on rust-dev linked from there.) But once we have that, dealing with filesystems in various encodings is something that should be fixed.

[lilyball]

@SimonSapin Yeah, this wasn't actually planned initially, but due to how Display works, if we can figure out the filesystem encoding, Display could be updated to use the appropriate encoding. Alternatively, we could add another method .display_encoding(enc) that returns a Display object for the appropriate encoding, for the situation where Path can't figure out the filesystem encoding but the caller can.

[alexcrichton]

This has largely been accomplished since this was written both in terms of an implementation strategy as well as propagating throughout the libraries. The stabilization passes we've been doing will turn up any more related clients, and I've requested that the remaining issue, #11916, be transferred to the rust-lang/rust repo, so closing.

[Eh2406]

There is a FIXME related to this issue,
https://github.com/rust-lang/rust/blob/master/src/tools/compiletest/src/runtest.rs#L538
https://github.com/rust-lang/rust/blob/master/src/tools/compiletest/src/runtest.rs#L641
https://github.com/rust-lang/rust/blob/master/src/tools/compiletest/src/runtest.rs#L1497
Now that the issue is closed can the FIXME be fix, or made more specific?