-
Notifications
You must be signed in to change notification settings - Fork 28.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Missing fs(/promises) APIs: dup, dup2, fdopen #41733
Comments
Support for these would need to land in libuv first. |
Hm, I can see that for dup and dup2, but fdopen? Should I have filed a separate request for fdopen? |
All of them would need to be exposed by libuv and would need to deal with any platform-specific quirks. For example, all 3 of those functions exist on Windows, but they are deprecated aliases and so the underscore-prefixed versions of the functions should be used instead. There could be more platform-specific differences, but any/all of them should be dealt with at the libuv level. |
I'm confused why fsPromises.fdopen would need to use the C fdopen. I'm only proposing Node reuse the name. Looking at the implementation of fsPromises.open, I believe a correct implementation of fsPromises.fdopen would be
(note minor correction to originally proposed API - no 'mode' argument). I filed libuv/libuv#3448 for uv support for dup and dup2. |
Hang on, given the existence of a
It doesn't work with the copy of Node on the computer where I'm typing this, but that might just be because it's too old (v12). If this works, then it does what I'd want |
You can already pass a file descriptor to various Adding |
workaround on linux var fdDup = fs.openSync(`/proc/self/fd/0`); // os.dup(0) on linux use case for dup: fs.readSync with timeout example use in tokenpool-gnu-make-posix.cc (details) (javascript) (python) what does not work//var stdinDup = os.dup(0);
var stdinDup = fs.openSync(`/proc/self/fd/0`);
var closeTimer = setTimeout(() => stdinDup.close(), 100);
var buffer = Buffer.alloc(1);
try {
var bytesRead = fs.readSync(stdinDup); // can block until stdinDup.close
console.log(`done reading ${bytesRead} bytes from stdin: int ${buffer.readInt8()`);
}
catch (e) {
if (e.errno == -11) {
console.log("stdin is empty");
}
else throw e;
}
clearTimeout(closeTimer); ... but closeTimer is never called
hangs after |
I closed out libuv/libuv#3448 just now. I'll go ahead and close out this issue as well. If someone still wants to pursue this it's probably better to create a new issue because the discussion in this one is somewhat confusing and meandering. FWIW, dup() and dup2() have rather subtle semantics (file descriptors vs. file descriptions) and I don't necessarily agree it's a good idea to add them to node. @milahu Try opening the file with Not that bypassing node's stdin handling is necessarily a good idea so use at your own risk. |
What is the problem this feature will solve?
The
fs
module exposes most of the OS-level file operations one would expect, but a few are missing, and the missing operations that I particularly need today aredup
anddup2
.In addition,
fs/promises
does not currently offer a way to wrap aFileHandle
around an OS-level file descriptor. I suggest that the name for this operation should befdopen
, matching C/POSIX.The use case I have in mind today for all three of these functions is for programs that, in the Unix tradition, generate data and write it to
process.stdout
. Simply callingprocess.stdout.write
orreadable.pipe(process.stdout)
has two problems today. First, this can be extremely inefficient becauseprocess.stdout
does no buffering. Second, several methods of the globalconsole
object write toprocess.stdout
, so third-party libraries that call e.g.console.info
(perhaps only rarely) can corrupt your output, and there doesn't appear to be any official way to reconfigure the console object to not do that. Iffs
haddup
,dup2
, andfdopen
, I could work around these problems with a function likeThis rearranges things at the level of file descriptors such that output to
process.stdout
(including via the console) is now sent toprocess.stderr
, and then returns a normal (in particular, buffering) WriteStream that sends data to the originalprocess.stdout
. The data generator can be piped to this stream, and it can be closed without causing problems for unrelated code that assumesprocess.stdout
is always open.Other use cases for
fsPromises.fdopen
are common: any time a node.js program receives an OS-level file descriptor from external code, the first thing it probably wants to do with it is wrap it in a FileHandle. Just off the top of my head, existing APIs that might hand a node.js program a bare file descriptor includesd_listen_fds
mechanism: server processes are started with extra fds open (beyond stdin/out/err) and are expected to retrieve these fds and use them as listening socketsSCM_RIGHTS
special socket message).Use cases for
dup
anddup2
are more unusual but do exist, particularly in low-level code.What is the feature you are proposing to solve the problem?
Add the following functions to the
fs/promises
module. TypeScript notation used to document function arguments and return types:A
FileHandle.dup
method (returning another FileHandle, not a bare fd) would also be appropriate for completeness.What alternatives have you considered?
The
fs.createReadStream
andfs.createWriteStream
functions in the old fs module offer a 'fd' option; this makes them sufficient for many situations wherefsPromises.fdopen
would be useful, However, they involve using the old fs module rather thanfs/promises
, andFileHandle
objects have many other handy features that people might want.My specific use case for
dup
anddup2
could be addressed directly in the standard library -- essentially, addprepare_buffered_stdout
instead of the primitives that would make it possible for me to implement it myself. However, that solves only that problem and not any of the other reasons people might want the primitives.The text was updated successfully, but these errors were encountered: