Tracking Issue for std::os::fd

[sunfishcode]

Feature gate: #![feature(os_fd)]

This is a tracking issue for the std::os::fd module, which is currently defined on
both Unix and WASI , and contains OwnedFd, BorrowedFd, RawFd, AsFd,
AsRawFd, IntoRawFd, and FromRawFd.

Public API

The types and traits are the same as those already defined in std::os::unix::io
and std::os::wasi::io. This PR is just introducing a new alias for them to help
users write portable code.

// in std::os

#[cfg(any(unix, target_os = "wasi"))]
pub mod fd {
    pub type RawFd = ...;
    pub struct OwnedFd { ... }
    pub struct BorrowedFd<'a> { ... }

    pub trait AsFd { ... }
    pub trait AsRawFd { ... }
    pub trait IntoRawFd { ... }
    pub trait FromRawFd { ... }
}

Steps / History

• Implementation: Make std::os::fd public. #98368
• Final comment period (FCP)
• Stabilization PR

Unresolved Questions

• "fd" isn't literally an "os", yet this proposes having it as a public module under
  std::os. One can think of it as meaning "OS's that use fds". Is that weird?

[m-ou-se]

Is "OS's that use fds" anything other than just "Unix and Wasi"? How close is Wasi to Unix? Would we treat Wasi similar to Unix in other situations too?

[sunfishcode]

Is "OS's that use fds" anything other than just "Unix and Wasi"?

std::os::solid and std::os::fortanix_sgx also have similar types and traits. I'm not familiar with those platforms, but they also may want to use std::os::fd.

How close is Wasi to Unix? Would we treat Wasi similar to Unix in other situations too?

That's a complex question :-}. As a platform, WASI lacks some fundamental Unixy things, like fork, signals, users, an ambient filesystem namespace, "/dev", and more. But as a layer on top of that, wasi-libc emulates some of these features, like an ambient filesystem namespace, in order to make it easier to port code to it. This is oversimplifying things, but in short, does Rust want to primarily target the compatibility abstractions, or the actual platform? We can obviously do some amount of both in practice, but for cfg(unix) in particular we do need to make a binary choice. There isn't an inherently correct answer here, but my sense is that overall it's better to treat WASI as something new, and to factor out Unix concepts like fds where it makes sense to share.

[WhyNotHugo]

Has there been any discussion on impl std::io::Write for std::os::unix::io::OwnedFd?

[sunfishcode]

Has there been any discussion on impl std::io::Write for std::os::unix::io::OwnedFd?

I'm not aware of any. One way to do this today, if you're on Unix and know you're doing write's, might be to use std::fs::File instead of OwnedFd. Does that sound like a reasonable approach, or do you think it'd be valuable to have something else that can do Write?

[WhyNotHugo]

I have an OwnedFd to start with. This is more of a convenience that strictly necessary, but I also think it makes sense for OwnedFd to be Write, rather than requiring it to be converted into a File in order to do any IO.

[sunfishcode]

I can see the utility, but there is also the question of the overall API shape. Not all file descriptors support write(2), though one might say the worst that happens is you get EBADF or EINVAL, though if you say that's ok, then there's a question of what other operations OwnedFd should implement, and it's not clear where the boundary should be.

There has also been some discussion about possible "view" types; I have a crate which lets you do owned_fd.as_filelike_view<File>() providing a temporary &File view of an OwnedFd or similar. That part wasn't in the main I/O safety feature that was stabilized, though if something like that is useful in enough places, it could potentially be added in the future.

[programmerjake]

I think std::os::fd should be available on more platforms than just wasm and Unix, e.g. Windows (and other POSIX OSes, which covers a lot more than just Unix) also have file descriptors.

Since it's so commonly available, I propose just making it available everywhere and the rare OSes that don't support file descriptors in some fashion can just ignore it. this also provides the benefit of having a sneaky way to get a non-negative-one integer type:
#[repr(transparent)] struct NonNegOneInt(BorrowedFd<'static>)

[sunfishcode]

On Windows, there are several different plausible choices to be the file descriptor type: C runtime library file descriptors, Foundation HANDLEs, Winsock2 SOCKETs, or something like RawHandleOrSocket.

Please don't use BorrowedFd<'static> as an arbitrary NonNegOneInt.

[programmerjake]

On Windows, there are several different plausible choices to be the file descriptor type: C runtime library file descriptors, Foundation HANDLEs, Winsock2 SOCKETs, or something like RawHandleOrSocket.

i would assume file descriptors always mean c runtime library file descriptors, since i've never seen handles or sockets referred to as file descriptors.

[programmerjake]

we could define FDs to be the type used for C's close function, that way it should be more clear what's meant. these types could be moved to std::ffi to make that more clear since they are used primarily to interface with C (the open/write/etc. functions)

[sunfishcode]

Could you describe what use case(s) would motivate this? I myself don't know of any use cases that want to use the C runtime library APIs on Windows.

[programmerjake]

Could you describe what use case(s) would motivate this? I myself don't know of any use cases that want to use the C runtime library APIs on Windows.

when C libraries use posix file descriptors and you want to use those libraries from rust
e.g. gzdopen which afaict is available on windows

[sunfishcode]

My feedback here is that if you're serious about wanting this, the more detail you can provide about use cases that would benefit from it, the easier it would be to consider.

One thing to keep in mind is that if std::os::fd were added on Windows and it used C runtime file descriptors, nothing in std, not even File itself, would be able to implement AsFd, AsRawFd, IntoOwnedFd, IntoRawFd, etc., because everything in std uses HANDLEs and SOCKETs.

[programmerjake]

what I wanted to do was obtain a non-neg-one-int that would work on all platforms, the other reasons for wanting FDs are because I know it'll be useful on all POSIX OSes (including Windows), not because I have any plans to use it myself.

[programmerjake]

One thing to keep in mind is that if std::os::fd were added on Windows and it used C runtime file descriptors, nothing in std, not even File itself, would be able to implement AsFd, AsRawFd, IntoOwnedFd, IntoRawFd, etc., because everything in std uses HANDLEs and SOCKETs.

that's not entirely true, file descriptors can be converted to handles and back: _open_osfhandle and _get_osfhandle

[Ralith]

I stumbled upon the portability issue above today. I wrote sd-listen-fds with the intention that it gracefully no-op outside of Linux, allowing callers to use it in portable code without cfg gates. This was motivated by the absence of any feature gate for the fd module visible in the std::os docs. I'll have to make my API significantly worse to work around this issue (e.g. returning an i32 or a custom wrapper type).

[LunarLambda]

std::os::fd was stabilized in 1.66.0. Is there something else missing or should this be closed?

[sunfishcode]

Yes, std::os::fd is stabilized now, so this can be closed. Thanks!