Implement `fs_native_path` #108981

ChrisDenton · 2023-03-10T12:13:49Z

Implements the fs_native_path feature. See also rust-lang/libs-team#62.

Internal changes

Ultimately, I'd envision that Unix filesystem functions (for example) would simply take either an &CStr or maybe an internal &PosixPath type with relevant helper methods. However, as much as I'd love to transition immediately, I think it's more prudent to take it slower. If nothing else it'll be easier on reviewers. So this PR just moves the responsibility for creating concrete types into sys::fs while touching as little existing code as possible, which should then allow platforms to make changes at their own pace.

rustbot · 2023-03-10T12:13:56Z

r? @joshtriplett

(rustbot has picked a reviewer for you, use r? to override)

rustbot · 2023-03-10T12:13:59Z

Hey! It looks like you've submitted a new PR for the library teams!

If this PR contains changes to any rust-lang/rust public library APIs then please comment with @rustbot label +T-libs-api -T-libs to tag it appropriately. If this PR contains changes to any unstable APIs please edit the PR description to add a link to the relevant API Change Proposal or create one if you haven't already. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary.

Examples of T-libs-api changes:

Stabilizing library features
Introducing insta-stable changes such as new implementations of existing stable traits on existing stable types
Introducing new or changing existing unstable library APIs (excluding permanently unstable features / features without a tracking issue)
Changing public documentation in ways that create new stability guarantees
Changing observable runtime behavior of library APIs

The Miri subtree was changed

cc @rust-lang/miri

joboet

Sorry, didn't read the description properly

library/std/src/sys/unix/path/posix_path.rs

library/std/src/sys/unix/fs.rs

ChrisDenton · 2023-03-10T13:00:15Z

At the moment, Path is still used in sys for some things like api and File::open on the smaller platforms. IMHO sys should be slightly refactored so that it always takes NativePath

Yes, I mentioned as much. However, this is more than a slight refactoring in some cases. Like you said, Path is still used in places (as well as PathBuf). I would prefer to delegate as much platform-specific changes to other PRs as possible so that they can get proper attention instead of having one giant PR that changes everything all at once.

ChrisDenton · 2023-03-10T15:10:51Z

I should do a crater run. There's at least a possibility that some weird inference thing may break this, which would be a blocker.

ChrisDenton · 2023-03-10T15:11:42Z

Oops, right

@bors try

bors · 2023-03-10T15:11:52Z

⌛ Trying commit 1b5d6cd355deacecbb236cbc553cfc988ec1ed62 with merge 1648938e809d7a7838c36282edd641ee0a4f137f...

bors · 2023-03-10T18:04:05Z

☀️ Try build successful - checks-actions
Build commit: 1648938e809d7a7838c36282edd641ee0a4f137f (1648938e809d7a7838c36282edd641ee0a4f137f)

ChrisDenton · 2023-03-10T18:05:19Z

@craterbot check

craterbot · 2023-03-10T18:05:28Z

👌 Experiment pr-108981 created and queued.
🤖 Automatically detected try build 1648938e809d7a7838c36282edd641ee0a4f137f
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2023-03-10T18:06:01Z

🚧 Experiment pr-108981 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2023-03-11T15:39:22Z

🎉 Experiment pr-108981 is completed!
📊 5 regressed and 3 fixed (258126 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the blacklist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

ChrisDenton · 2023-03-11T21:02:13Z

5 regressions and 3 fixes but none of them look relevant. So it seems like the public trait in this PR wouldn't break anything inference wise.

apiraino · 2023-04-25T14:21:33Z

Unless I'm mistaken , this looks a purely T-libs patch so I'll remove the T-compiler

@rustbot label -T-compiler

ChrisDenton · 2023-04-25T14:23:44Z

Oops yes, sorry. I think that was a left over auto-label.

SUPERCILEX · 2023-05-17T16:14:44Z

library/std/src/sys/unix/fs.rs

+    use crate::sys::path::PathLike;
+
+    pub(crate) fn remove_file<P: PathLike>(path: P) -> io::Result<()> {
+        path.with_path(fs::unlink)


So is the only remaining work after this PR to swap these to with_native_path? That's exciting!

Yeah, that's the idea! It may not be trivial in all cases though so I've left that for follow up PRs.

bors · 2023-12-14T13:25:34Z

☔ The latest upstream changes (presumably #118566) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2024-01-13T16:21:20Z

☔ The latest upstream changes (presumably #117285) made this pull request unmergeable. Please resolve the merge conflicts.

Amanieu · 2024-01-15T15:05:11Z

library/std/src/sys/windows/path.rs

+#[unstable(feature = "fs_native_path_internals", issue = "none")]
+#[repr(transparent)]
+#[derive(Debug)]
+pub struct WindowsPath(pub [u16]);


Why is this called WindowsPath instead of just NativePath (which is used for all other targets)?

Looking back at the history here it seems I was originally thinking of making an equivalent of CStr (called WStr at the time). As the idea here evolved and narrowed in scope this eventually became WindowsPath,

But yes, I should just call this NativePath too. If there's a need for a more general type then that can and should be a separate PR.

Amanieu · 2024-01-15T15:12:48Z

library/std/src/path.rs

+/// native paths more directly to system APIs.
+#[unstable(feature = "fs_native_path", issue = "108979")]
+pub trait AsPath: pathlike::Sealed {
+    // NOTE: This trait is purely for external documentation purposes.


This seems quite complex since it requires duplicating the PathLike trait for every OS despite it having the same signature everywhere. Could we do something like this instead:

Move the PathLike methods into this trait, but make them doc-hidden and unstable.

Remove PathLike.

Sure. This was written awhile ago and underwent a number of revisions both before and during the ACP so I'm still trying to page in what my intentions were. I agree that would be much simpler.

Amanieu · 2024-01-15T15:14:18Z

library/std/src/sys/solid/fs.rs

@@ -325,14 +378,17 @@ fn cstr(path: &Path) -> io::Result<CString> {

 impl File {
    pub fn open(path: &Path, opts: &OpenOptions) -> io::Result<File> {


There seems to be some inconsistency between the different targets in offering different combinations of open, open_c and open_native. Could this all be cleaned up to have a single open method that takes a NativePath?

I think my intention was to avoid touching as much code as possible and some parts are using Path quite a bit so it's a bit more involved to change them to use NativePath or adding in the conversions. But I can take another look.

ChrisDenton · 2024-02-28T05:07:32Z

@Amanieu Given the recent restructuring and other changes in std, rebasing simplified things a lot. In addition I applied your suggest to simplify the trait.

Amanieu · 2024-02-29T00:52:06Z

library/std/src/os/windows/ffi.rs

+#[unstable(feature = "fs_native_path", issue = "108979")]
+impl NativePathExt for NativePath {
+    fn from_wide(wide: &[u16]) -> &NativePath {
+        assert_eq!(crate::sys::unrolled_find_u16s(0, wide), Some(wide.len().saturating_sub(1)));


Add a user-friendly panic message for when assert fails.

Amanieu · 2024-02-29T00:53:34Z

library/std/src/path.rs

+/// These types include [`Path`], [`OsStr`] and [`str`] as well as their owned
+/// counterparts [`PathBuf`], [`OsString`] and [`String`].
+///
+/// You can also implement you own [`AsRef<Path>`] for your own types and they'll


Suggested change

/// You can also implement you own [`AsRef<Path>`] for your own types and they'll

/// You can also implement [`AsRef<Path>`] for your own types and they'll

Reminder to address this typo.

Amanieu · 2024-03-06T02:41:14Z

library/std/src/sys/pal/solid/fs.rs

@@ -325,14 +377,17 @@ fn cstr(path: &Path) -> io::Result<CString> {

 impl File {
    pub fn open(path: &Path, opts: &OpenOptions) -> io::Result<File> {


This still still inconsistent: why is open kept in some targets but removed in other? I would expect everything to use open_native only and for open to be removed (or possible just change the signature of open to take native path).

Amanieu · 2024-03-06T02:41:34Z

library/std/src/sys/pal/windows/fs.rs

@@ -294,8 +349,12 @@ impl OpenOptions {
 }

 impl File {
-    pub fn open(path: &Path, opts: &OpenOptions) -> io::Result<File> {
+    pub(super) fn open(path: &Path, opts: &OpenOptions) -> io::Result<File> {


And replace `AsRef<Path>` with the `AsPath` trait.

bors · 2024-03-10T00:03:13Z

☔ The latest upstream changes (presumably #122256) made this pull request unmergeable. Please resolve the merge conflicts.

Amanieu · 2024-03-15T15:31:08Z

library/std/src/sys/pal/wasi/fs.rs

-                        p
-                    );
-                    return Err(io::Error::new(io::ErrorKind::Uncategorized, msg));
+fn open_parent(p: &CStr) -> io::Result<(ManuallyDrop<WasiFd>, PathBuf)> {


This should return a CString to avoid another conversion in open_at.

Amanieu · 2024-03-15T15:34:20Z

library/std/src/os/wasi/fs.rs

@@ -563,7 +563,9 @@ pub fn symlink<P: AsRef<Path>, U: AsRef<Path>>(
 /// This is a convenience API similar to `std::os::unix::fs::symlink` and
 /// `std::os::windows::fs::symlink_file` and `std::os::windows::fs::symlink_dir`.
 pub fn symlink_path<P: AsRef<Path>, U: AsRef<Path>>(old_path: P, new_path: U) -> io::Result<()> {


Should these methods be updated to use AsPath instead?

Amanieu · 2024-03-15T15:36:06Z

library/std/src/path.rs

+/// These types include [`Path`], [`OsStr`] and [`str`] as well as their owned
+/// counterparts [`PathBuf`], [`OsString`] and [`String`].
+///
+/// You can also implement you own [`AsRef<Path>`] for your own types and they'll


Reminder to address this typo.

Amanieu · 2024-03-15T15:37:51Z

library/std/src/sys/pal/wasi/fs.rs

    let (dir, file) = open_parent(p)?;
    dir.unlink_file(osstr2str(file.as_ref())?)
 }

-pub fn rename(old: &Path, new: &Path) -> io::Result<()> {


Why is only one changed to CStr but not both? In fact shouldn't all &Path in this file be changed?

Amanieu · 2024-03-15T15:39:51Z

library/std/src/sys/pal/windows/fs.rs

@@ -1132,16 +1186,16 @@ pub fn rmdir(p: &Path) -> io::Result<()> {
 }


Shouldn't all the methods above (rename, unlink, rmdir, etc) also be switched to NativePath?

rustbot assigned joshtriplett Mar 10, 2023

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Mar 10, 2023

ChrisDenton mentioned this pull request Mar 10, 2023

Tracking Issue for fs_native_path #108979

Open

3 tasks

joboet reviewed Mar 10, 2023

View reviewed changes

library/std/src/sys/unix/path/posix_path.rs Outdated Show resolved Hide resolved

library/std/src/sys/unix/fs.rs Outdated Show resolved Hide resolved

ChrisDenton force-pushed the aspath branch from 33f52c0 to eae801e Compare March 10, 2023 14:21

This comment was marked as outdated.

Sign in to view

craterbot added S-waiting-on-crater Status: Waiting on a crater run to be completed. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 10, 2023

craterbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-crater Status: Waiting on a crater run to be completed. labels Mar 11, 2023

rustbot removed the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Apr 25, 2023

ChrisDenton mentioned this pull request May 17, 2023

ACP: PathLike trait rust-lang/libs-team#62

Closed

SUPERCILEX reviewed May 17, 2023

View reviewed changes

ChrisDenton force-pushed the aspath branch from e989369 to 5d7945d Compare January 2, 2024 16:50

Amanieu reviewed Jan 15, 2024

View reviewed changes

Amanieu added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 21, 2024

ChrisDenton force-pushed the aspath branch from 5d7945d to f672708 Compare February 28, 2024 04:59

ChrisDenton added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Feb 28, 2024

This comment has been minimized.

Sign in to view

ChrisDenton force-pushed the aspath branch 3 times, most recently from cfc57ac to 308e304 Compare February 28, 2024 06:02

This comment has been minimized.

Sign in to view

ChrisDenton force-pushed the aspath branch from 308e304 to 1578ab6 Compare February 28, 2024 13:46

Amanieu reviewed Mar 6, 2024

View reviewed changes

ChrisDenton added 3 commits March 8, 2024 01:57

Add NativePath

59c426c

And replace `AsRef<Path>` with the `AsPath` trait.

Windows implementation of NativePath

a8b37df

Unix implementation of NativePath

abc0c63

ChrisDenton force-pushed the aspath branch from 1578ab6 to 03fb313 Compare March 8, 2024 02:00

This comment has been minimized.

Sign in to view

ChrisDenton force-pushed the aspath branch from 03fb313 to be3b69b Compare March 8, 2024 02:38

ChrisDenton added 3 commits March 8, 2024 05:49

Various OS implementations of NativePath

64ff8e2

WASI implementation of NativePath

bd3ca77

Bless NativePath

80261c4

ChrisDenton force-pushed the aspath branch from be3b69b to 80261c4 Compare March 8, 2024 05:54

rustbot added the O-wasi Operating system: Wasi, Webassembly System Interface label Mar 8, 2024

Amanieu reviewed Mar 15, 2024

View reviewed changes

		@@ -325,14 +378,17 @@ fn cstr(path: &Path) -> io::Result<CString> {

		impl File {
		pub fn open(path: &Path, opts: &OpenOptions) -> io::Result<File> {

	/// You can also implement you own [`AsRef<Path>`] for your own types and they'll
	/// You can also implement [`AsRef<Path>`] for your own types and they'll

		@@ -325,14 +377,17 @@ fn cstr(path: &Path) -> io::Result<CString> {

		impl File {
		pub fn open(path: &Path, opts: &OpenOptions) -> io::Result<File> {

		@@ -1132,16 +1186,16 @@ pub fn rmdir(p: &Path) -> io::Result<()> {
		}

Implement fs_native_path #108981

Are you sure you want to change the base?

Implement fs_native_path #108981

Conversation

ChrisDenton commented Mar 10, 2023 • edited Loading

Internal changes

rustbot commented Mar 10, 2023

rustbot commented Mar 10, 2023

joboet left a comment • edited Loading

Choose a reason for hiding this comment

ChrisDenton commented Mar 10, 2023

ChrisDenton commented Mar 10, 2023 • edited Loading

This comment was marked as outdated.

ChrisDenton commented Mar 10, 2023

bors commented Mar 10, 2023

bors commented Mar 10, 2023

ChrisDenton commented Mar 10, 2023

craterbot commented Mar 10, 2023

craterbot commented Mar 10, 2023

craterbot commented Mar 11, 2023

ChrisDenton commented Mar 11, 2023

apiraino commented Apr 25, 2023

ChrisDenton commented Apr 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented Dec 14, 2023

bors commented Jan 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrisDenton commented Feb 28, 2024

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

bors commented Mar 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Implement `fs_native_path` #108981

Implement `fs_native_path` #108981

ChrisDenton commented Mar 10, 2023 •

edited

Loading

joboet left a comment •

edited

Loading

ChrisDenton commented Mar 10, 2023 •

edited

Loading