Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: Add file descriptor support to *File() funcs #3163

Closed
wants to merge 1 commit into from
Closed

fs: Add file descriptor support to *File() funcs #3163

wants to merge 1 commit into from

Conversation

jwueller
Copy link
Contributor

@jwueller jwueller commented Oct 3, 2015

This is a re-targeted submission of the archived PR #8522, implementing the archived issue #8471.

The original description follows.


These changes affect the following functions and their synchronous counterparts:

  • fs.readFile()
  • fs.writeFile()
  • fs.appendFile()

If the first parameter is a uint32, it is treated as a file descriptor. In all other cases, the original implementation is used to ensure backwards compatibility. File descriptor ownership is never taken from the user.

The documentation was adjusted to reflect these API changes. A note was added to make the user aware of file descriptor ownership and the conditions under which a file descriptor can be used by each of these functions.

Tests were extended to test for file descriptor parameters under the conditions noted in the relevant documentation.

@mscdex mscdex added the fs Issues and PRs related to the fs subsystem / file system. label Oct 3, 2015
@jwueller
Copy link
Contributor Author

jwueller commented Oct 3, 2015

/cc @trevnorris @misterdjules

This one also includes a test for fs.writeFileSync(), where the original PR did not.

@ChALkeR ChALkeR added the semver-minor PRs that contain new features and should be released in the next minor version. label Oct 4, 2015
@@ -101,6 +101,10 @@ function nullCheck(path, callback) {
return true;
}

function isFd(path) {
return (path >>> 0) === path && path >= 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

path >= 0 can never be false here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You seem to be correct. This is probably residue from this discussion.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, if the user passes readFile('0xff'), would that be interpreted as a file name or file descriptor?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This implementation would treat '0xff' as a file name.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will all values where typeof === 'number' be treated as a number, all other values will be coerced to a string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original implementation does not seem to coerce at all. It errors with TypeError: path must be a string instead.

Nevertheless, >>> should always produce a number type, so the strict equality test can only ever succeed for typeof input === 'number', making everything else go down the file name branch, even if that results in another TypeError caused by the path argument not being a string. This behavior remains unchanged.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pondering how much coercion is allowed. e.g. should we throw on NaN? my instinct is yes. all non uint32 values should throw. Which is exactly what you have now. :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I generally avoid coercion as much as possible. It usually leads to rather unclear behavior. Throwing on NaN sounds like it would be expected. There is no sensible default we can choose in that case.

@ChALkeR
Copy link
Member

ChALkeR commented Oct 4, 2015

This is at least semver-minor, but actually looks more like a semver-major to me.
What if someone has done something like

var num = 1;

fs.writeFile(num++, );

Also, the logic is not very clear:

  1. fs.writeFile(1,…) tries to write to file descriptor 1.
  2. fs.writeFile(-1,…) tries to write to file name '-1'.
  3. fs.writeFile(4294967295,…) tries to write to file descriptor 4294967295.
  4. fs.writeFile(4294967296,…) tries to write to file name '4294967296'.

Understandable behavior IMO would be to check if the file name is a string, and if it's not a string nor a valid file descriptor — throw an Error. But that change would be clearly a semver-major.

Not sure what the others think.

Note: I left aside the actual feature introduced by this PR as I have no opinion on that, and covered only the API changes.

@jwueller
Copy link
Contributor Author

jwueller commented Oct 4, 2015

@ChALkeR First of all, thanks for reviewing this!

fs.writeFile(-1), etc. will actually fail with TypeError: path must be a string, so there does not seem to be a type coercion backwards compatibility problem here. Throwing a dedicated Error in case of neither a string or a plausible file descriptor uint would probably be helpful, though.

@trevnorris actually proposed the uint behavior here, as opposed to going the typeof route. Some additional discussion happened here. Thoughts?

Additionally, maybe @chrisdickinson could shed some light on the originally envisioned behavior as well (original issue link).

@trevnorris
Copy link
Contributor

What are potential blockers here?

@jwueller
Copy link
Contributor Author

jwueller commented Oct 8, 2015

Incompatibilities with existing code would definitely be a blocker. Those should not be possible, though. Tests for the original behavior still pass and remain unchanged.

Also, I'll need to do another rebase (and remove the redundant path >= 0 bit).

@ChALkeR
Copy link
Member

ChALkeR commented Oct 8, 2015

fs.writeFile(-1), etc. will actually fail with TypeError: path must be a string, so there does not seem to be a type coercion backwards compatibility problem here.

Ah, I overlooked that. So then there should be no backwards-incompatibility here, and this is indeed a semver-minor only?

@trevnorris
Copy link
Contributor

@ChALkeR This looks like an additive change. All previously accepted inputs are the same, with the addition of supporting uint32's for file descriptors. I'd say this is a semver-minor.

@jwueller
Copy link
Contributor Author

@ChALkeR @trevnorris Rebased and updated!

0o666,
req);
if (context.isUserFd) {
req.oncomplete.call(req, null, path);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put this in a nextTick. It's an expected async call, and must always run async.

@jwueller
Copy link
Contributor Author

@trevnorris Updated again with the discussed changes.


_Note: Specified file descriptors will not be closed automatically._

## fs.appendFileSync(file, data[, options])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that you have the note in fs.writeFileSync, but not here in fs.appendFileSync. Is it necessary here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what you are asking, exactly. The note is present in all asynchronous versions, while being omitted in the synchronous ones, since the latter segments only refer to the asynchronous documentation anyway.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoop. Sorry. Got lost in the diff. Can see that now.

@trevnorris
Copy link
Contributor

Almost there!

@jwueller
Copy link
Contributor Author

After some further digging, I noticed that we have – in fact – several different approaches to dealing with file descriptor arguments available right now:

Method fd Detection fd Ownership Transfer
fs.truncate() typeof path === 'number' none
this PR (path >>> 0) === path none
fs.createReadStream()
fs.createWriteStream()
supplied through options.fd, no type check via options.autoClose (enabled by default)

This makes the PR in it's current form differ from existing functionality. It seems to me we should discuss which of these approaches is the right way to go here.

The latter one does not seem to have existed at the time of writing the original PR (and I did not want to make things more complicated than necessary at the time), but is definitely the most flexible option now that it exists "in the wild".

The behavior of fs.truncate() does not seem desirable. It is behaving strangely, as it automatically delegates to fs.ftruncate(), while other functions with direct fs.f*() counterparts do not. If this is legacy behavior, it likely could be officially deprecated to make the API more uniform. This should probably be the subject of a separate Issue/PR, though. Introducing commit is 168a555, for the record.

@trevnorris
Copy link
Contributor

I believe how the functionality works in this PR is most appropriate of the differences. So I'd say we proceed with this then work on fixing the other implementations where necessary.

@jwueller
Copy link
Contributor Author

That seems reasonable. Updated again.

@trevnorris
Copy link
Contributor

Looks great. Running CI one last time for sanity then will land this thing: https://ci.nodejs.org/job/node-test-pull-request/508/

@jwueller
Copy link
Contributor Author

Thank you very much for reviewing everything!

var req = new FSReqWrap();
req.context = context;
req.oncomplete = readFileAfterOpen;

if (context.isUserFd) {
process.nextTick(function() {
req.oncomplete.call(req, null, path);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not req.oncomplete(null, path)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would not behave correctly. This call is supposed to emulate the behavior of binding.open(). Omitting the context would lead to incompatibilities with following code (ex: readFileAfterOpen(), where the context is extracted from this).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the default context is the object to the left of the . so I think @thefourtheye is right in this case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course he is. I actually tripped myself there. Sorry! I'll clean that up immediately.

@jwueller
Copy link
Contributor Author

@trevnorris One more fix. Thanks @thefourtheye for reviewing!

These changes affect the following functions and their synchronous
counterparts:

 * fs.readFile()
 * fs.writeFile()
 * fs.appendFile()

If the first parameter is a uint32, it is treated as a file descriptor.
In all other cases, the original implementation is used to ensure
backwards compatibility. File descriptor ownership is never taken from
the user.

The documentation was adjusted to reflect these API changes. A note was
added to make the user aware of file descriptor ownership and the
conditions under which a file descriptor can be used by each of these
functions.

Tests were extended to test for file descriptor parameters under the
conditions noted in the relevant documentation.
@trevnorris
Copy link
Contributor

Final CI for sanity: https://ci.nodejs.org/job/node-test-pull-request/518/

If CI is good let's land this thing! LGTM

@jwueller
Copy link
Contributor Author

@trevnorris Something seems to have failed, but it is not obvious to me what is happening, exactly. Can you help me out?

@trevnorris
Copy link
Contributor

@jwueller None of the failing tests are related to your PR. Everything seems fine.

trevnorris pushed a commit that referenced this pull request Oct 16, 2015
These changes affect the following functions and their synchronous
counterparts:

 * fs.readFile()
 * fs.writeFile()
 * fs.appendFile()

If the first parameter is a uint32, it is treated as a file descriptor.
In all other cases, the original implementation is used to ensure
backwards compatibility. File descriptor ownership is never taken from
the user.

The documentation was adjusted to reflect these API changes. A note was
added to make the user aware of file descriptor ownership and the
conditions under which a file descriptor can be used by each of these
functions.

Tests were extended to test for file descriptor parameters under the
conditions noted in the relevant documentation.

PR-URL: #3163
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
@trevnorris
Copy link
Contributor

Landed on 0803962. Thanks!

Note: This is labelled as semver-minor but it should not land on LTS.

@trevnorris trevnorris closed this Oct 16, 2015
@jwueller
Copy link
Contributor Author

@trevnorris Great! Thanks again!

@jwueller jwueller deleted the fs-optional-file-descriptor-args branch October 17, 2015 00:09
rvagg pushed a commit that referenced this pull request Oct 19, 2015
These changes affect the following functions and their synchronous
counterparts:

 * fs.readFile()
 * fs.writeFile()
 * fs.appendFile()

If the first parameter is a uint32, it is treated as a file descriptor.
In all other cases, the original implementation is used to ensure
backwards compatibility. File descriptor ownership is never taken from
the user.

The documentation was adjusted to reflect these API changes. A note was
added to make the user aware of file descriptor ownership and the
conditions under which a file descriptor can be used by each of these
functions.

Tests were extended to test for file descriptor parameters under the
conditions noted in the relevant documentation.

PR-URL: #3163
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
@rvagg rvagg mentioned this pull request Oct 21, 2015
rvagg added a commit to rvagg/io.js that referenced this pull request Oct 29, 2015
Notable changes:

* buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer,
  these have been deprecated for a long time (Sakthipriyan Vairamani) nodejs#2859.
* console: (Breaking) Values reported by console.time() now have 3 decimals of
  accuracy added (Michaël Zasso) nodejs#3166.
* fs:
  - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file
    descriptor as their first argument (Johannes Wüller) nodejs#3163.
  - (Breaking) In fs.readFile(), if an encoding is specified and the internal
    toString() fails the error is no longer thrown but is passed to the callback
    (Evan Lucas) nodejs#3485.
  - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding,
    callback) form), if the internal toString() fails the error is no longer
    thrown but is passed to the callback (Evan Lucas) nodejs#3503.
* http:
  - Fixed a bug where pipelined http requests would stall (Fedor Indutny) nodejs#3342.
  - (Breaking) When parsing HTTP, don't add duplicates of the following headers:
    Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition
    to the following headers which already block duplicates: Content-Type,
    Content-Length, User-Agent, Referer, Host, Authorization,
    Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location,
    Max-Forwards (James M Snell) nodejs#3090.
  - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a
    function or a TypeError is thrown (James M Snell) nodejs#3090.
  - (Breaking) HTTP methods and header names must now conform to the RFC 2616
    "token" rule, a list of allowed characters that excludes control characters
    and a number of separator characters. Specifically, methods and header names
    must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown
    (James M Snell) nodejs#2526.
* node:
  - (Breaking) Deprecated the _linklist module (Rich Trott) nodejs#3078.
  - (Breaking) Removed require.paths and require.registerExtension(), both had
    been previously set to throw Error when accessed
    (Sakthipriyan Vairamani) nodejs#2922.
* npm: Upgraded to version 3.3.6 from 2.14.7, see
  https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a
  major version bump for npm and it has seen a significant amount of change.
  Please see the original npm v3.0.0 release notes for a list of major changes
  (Rebecca Turner) nodejs#3310.
* src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary
  due to the V8 upgrade. Native add-ons will need to be recompiled
  (Rod Vagg) nodejs#3400.
* timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes
  a long-standing known issue where unrefed timers would perviously hold
  beforeExit open (Fedor Indutny) nodejs#3407.
* tls:
  - Added ALPN Support (Shigeki Ohtsu) nodejs#2564.
  - TLS options can now be passed in an object to createSecurePair()
    (Коренберг Марк) nodejs#2441.
  - (Breaking) The default minimum DH key size for tls.connect() is now 1024
    bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS
    option can be used to override the default. (Shigeki Ohtsu) nodejs#1831.
* util:
  - (Breaking) util.p() was deprecated for years, and has now been removed
    (Wyatt Preul) nodejs#3432.
  - (Breaking) util.inherits() can now work with ES6 classes. This is considered
    a breaking change because of potential subtle side-effects caused by a
    change from directly reassigning the prototype of the constructor using
    `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })`
    to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)`
    (Michaël Zasso) nodejs#3455.
* v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) nodejs#3351.
  - Implements the spread operator, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator
    for further information.
  - Implements new.target, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target
    for further information.
* zlib: Decompression now throws on truncated input (e.g. unexpected end of
  file) (Yuval Brik) nodejs#2595.

PR-URL: nodejs#3466
rvagg added a commit that referenced this pull request Oct 29, 2015
Notable changes:

* buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer,
  these have been deprecated for a long time (Sakthipriyan Vairamani) #2859.
* console: (Breaking) Values reported by console.time() now have 3 decimals of
  accuracy added (Michaël Zasso) #3166.
* fs:
  - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file
    descriptor as their first argument (Johannes Wüller) #3163.
  - (Breaking) In fs.readFile(), if an encoding is specified and the internal
    toString() fails the error is no longer thrown but is passed to the callback
    (Evan Lucas) #3485.
  - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding,
    callback) form), if the internal toString() fails the error is no longer
    thrown but is passed to the callback (Evan Lucas) #3503.
* http:
  - Fixed a bug where pipelined http requests would stall (Fedor Indutny) #3342.
  - (Breaking) When parsing HTTP, don't add duplicates of the following headers:
    Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition
    to the following headers which already block duplicates: Content-Type,
    Content-Length, User-Agent, Referer, Host, Authorization,
    Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location,
    Max-Forwards (James M Snell) #3090.
  - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a
    function or a TypeError is thrown (James M Snell) #3090.
  - (Breaking) HTTP methods and header names must now conform to the RFC 2616
    "token" rule, a list of allowed characters that excludes control characters
    and a number of separator characters. Specifically, methods and header names
    must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown
    (James M Snell) #2526.
* node:
  - (Breaking) Deprecated the _linklist module (Rich Trott) #3078.
  - (Breaking) Removed require.paths and require.registerExtension(), both had
    been previously set to throw Error when accessed
    (Sakthipriyan Vairamani) #2922.
* npm: Upgraded to version 3.3.6 from 2.14.7, see
  https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a
  major version bump for npm and it has seen a significant amount of change.
  Please see the original npm v3.0.0 release notes for a list of major changes
  (Rebecca Turner) #3310.
* src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary
  due to the V8 upgrade. Native add-ons will need to be recompiled
  (Rod Vagg) #3400.
* timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes
  a long-standing known issue where unrefed timers would perviously hold
  beforeExit open (Fedor Indutny) #3407.
* tls:
  - Added ALPN Support (Shigeki Ohtsu) #2564.
  - TLS options can now be passed in an object to createSecurePair()
    (Коренберг Марк) #2441.
  - (Breaking) The default minimum DH key size for tls.connect() is now 1024
    bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS
    option can be used to override the default. (Shigeki Ohtsu) #1831.
* util:
  - (Breaking) util.p() was deprecated for years, and has now been removed
    (Wyatt Preul) #3432.
  - (Breaking) util.inherits() can now work with ES6 classes. This is considered
    a breaking change because of potential subtle side-effects caused by a
    change from directly reassigning the prototype of the constructor using
    `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })`
    to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)`
    (Michaël Zasso) #3455.
* v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) #3351.
  - Implements the spread operator, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator
    for further information.
  - Implements new.target, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target
    for further information.
* zlib: Decompression now throws on truncated input (e.g. unexpected end of
  file) (Yuval Brik) #2595.

PR-URL: #3466
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fs Issues and PRs related to the fs subsystem / file system. semver-minor PRs that contain new features and should be released in the next minor version.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants