fs: Add file descriptor support to *File() funcs

[jwueller]

This is a re-targeted submission of the archived PR #8522, implementing the archived issue #8471.

The original description follows.

---

These changes affect the following functions and their synchronous counterparts:

• fs.readFile()
• fs.writeFile()
• fs.appendFile()

If the first parameter is a uint32, it is treated as a file descriptor. In all other cases, the original implementation is used to ensure backwards compatibility. File descriptor ownership is never taken from the user.

The documentation was adjusted to reflect these API changes. A note was added to make the user aware of file descriptor ownership and the conditions under which a file descriptor can be used by each of these functions.

Tests were extended to test for file descriptor parameters under the conditions noted in the relevant documentation.

[jwueller]

/cc @trevnorris @misterdjules

This one also includes a test for fs.writeFileSync(), where the original PR did not.

[ChALkeR]

path >= 0 can never be false here.

[jwueller]

You seem to be correct. This is probably residue from this discussion.

[trevnorris]

To clarify, if the user passes readFile('0xff'), would that be interpreted as a file name or file descriptor?

[jwueller]

This implementation would treat '0xff' as a file name.

[trevnorris]

Will all values where typeof === 'number' be treated as a number, all other values will be coerced to a string?

[jwueller]

The original implementation does not seem to coerce at all. It errors with TypeError: path must be a string instead.

Nevertheless, >>> should always produce a number type, so the strict equality test can only ever succeed for typeof input === 'number', making everything else go down the file name branch, even if that results in another TypeError caused by the path argument not being a string. This behavior remains unchanged.

[trevnorris]

I'm pondering how much coercion is allowed. e.g. should we throw on NaN? my instinct is yes. all non uint32 values should throw. Which is exactly what you have now. :)

[jwueller]

I generally avoid coercion as much as possible. It usually leads to rather unclear behavior. Throwing on NaN sounds like it would be expected. There is no sensible default we can choose in that case.

[ChALkeR]

This is at least semver-minor, but actually looks more like a semver-major to me.
What if someone has done something like

var num = 1;
…
fs.writeFile(num++, …);

Also, the logic is not very clear:

1. fs.writeFile(1,…) tries to write to file descriptor 1.
2. fs.writeFile(-1,…) tries to write to file name '-1'.
3. fs.writeFile(4294967295,…) tries to write to file descriptor 4294967295.
4. fs.writeFile(4294967296,…) tries to write to file name '4294967296'.

Understandable behavior IMO would be to check if the file name is a string, and if it's not a string nor a valid file descriptor — throw an Error. But that change would be clearly a semver-major.

Not sure what the others think.

Note: I left aside the actual feature introduced by this PR as I have no opinion on that, and covered only the API changes.

[jwueller]

@ChALkeR First of all, thanks for reviewing this!

fs.writeFile(-1), etc. will actually fail with TypeError: path must be a string, so there does not seem to be a type coercion backwards compatibility problem here. Throwing a dedicated Error in case of neither a string or a plausible file descriptor uint would probably be helpful, though.

@trevnorris actually proposed the uint behavior here, as opposed to going the typeof route. Some additional discussion happened here. Thoughts?

Additionally, maybe @chrisdickinson could shed some light on the originally envisioned behavior as well (original issue link).

[trevnorris]

What are potential blockers here?

[jwueller]

Incompatibilities with existing code would definitely be a blocker. Those should not be possible, though. Tests for the original behavior still pass and remain unchanged.

Also, I'll need to do another rebase (and remove the redundant path >= 0 bit).

[ChALkeR]

fs.writeFile(-1), etc. will actually fail with TypeError: path must be a string, so there does not seem to be a type coercion backwards compatibility problem here.

Ah, I overlooked that. So then there should be no backwards-incompatibility here, and this is indeed a semver-minor only?

[trevnorris]

@ChALkeR This looks like an additive change. All previously accepted inputs are the same, with the addition of supporting uint32's for file descriptors. I'd say this is a semver-minor.

[jwueller]

@ChALkeR @trevnorris Rebased and updated!

[trevnorris]

Put this in a nextTick. It's an expected async call, and must always run async.

[trevnorris]

return after the if then drop the else.

[jwueller]

I will adjust this where appropriate.

[trevnorris]

Clarify this with me. If an fd is passed then it isn't automatically closed?

[jwueller]

Correct. No fd ownership transfer takes place. We did not create the fd, so we should not close it IMHO. Doing so would prevent users from re-using existing file descriptors more than once, pretty much defeating the point of this PR.

[trevnorris]

This needs to be clearly documented. Possibly with big bold text. I can see users thinking they're handing off the fd.

[jwueller]

Yes, I agree that this is important information. The current patch adds the phrase

Specified file descriptors will not be closed automatically.

to all relevant locations. Do you think the whole thing should be bolded, or just the "not"? I am currently leaning towards emphasizing the whole phrase.

[trevnorris]

It should be in its own paragraph and prefixed with Note:. No need for that to be bold. But having that is common in documentation.

Thanks for working w/ me on this.

[trevnorris]

Right here. And also in every location where this is true. documentation duplication in all relevant locations is cool.

[jwueller]

Oops, I missed this comment. Please check my response above. How should this be modified, exactly?

[jwueller]

@trevnorris Updated again with the discussed changes.

[trevnorris]

I see that you have the note in fs.writeFileSync, but not here in fs.appendFileSync. Is it necessary here?

[jwueller]

I am not sure what you are asking, exactly. The note is present in all asynchronous versions, while being omitted in the synchronous ones, since the latter segments only refer to the asynchronous documentation anyway.

[trevnorris]

Whoop. Sorry. Got lost in the diff. Can see that now.

[trevnorris]

can remove this extra line.

[trevnorris]

ditto.

[trevnorris]

Mind adding that data written to fd will be appended?

Guess we don't expose an lseek() in fs so users couldn't reposition where data is written. Hm. Future feature.

[trevnorris]

nm. i see what's happening.

[jwueller]

Whether data will be appended depends on the flags that the fd was created with. The a flags will obviously append, but the w flags will create/truncate instead.

[trevnorris]

Yup. I crossed the following code from fs.append:

  if (isFd(path))
    options.flag = 'a';

Sorry

[trevnorris]

Almost there!

[jwueller]

After some further digging, I noticed that we have – in fact – several different approaches to dealing with file descriptor arguments available right now:

Method	fd Detection	fd Ownership Transfer
fs.truncate()	typeof path === 'number'	none
this PR	(path >>> 0) === path	none
fs.createReadStream() fs.createWriteStream()	supplied through options.fd, no type check	via options.autoClose (enabled by default)

This makes the PR in it's current form differ from existing functionality. It seems to me we should discuss which of these approaches is the right way to go here.

The latter one does not seem to have existed at the time of writing the original PR (and I did not want to make things more complicated than necessary at the time), but is definitely the most flexible option now that it exists "in the wild".

The behavior of fs.truncate() does not seem desirable. It is behaving strangely, as it automatically delegates to fs.ftruncate(), while other functions with direct fs.f*() counterparts do not. If this is legacy behavior, it likely could be officially deprecated to make the API more uniform. This should probably be the subject of a separate Issue/PR, though. Introducing commit is 168a555, for the record.

[trevnorris]

I believe how the functionality works in this PR is most appropriate of the differences. So I'd say we proceed with this then work on fixing the other implementations where necessary.

[jwueller]

That seems reasonable. Updated again.

[trevnorris]

Looks great. Running CI one last time for sanity then will land this thing: https://ci.nodejs.org/job/node-test-pull-request/508/

[jwueller]

Thank you very much for reviewing everything!

[thefourtheye]

Why not req.oncomplete(null, path)?

[jwueller]

It would not behave correctly. This call is supposed to emulate the behavior of binding.open(). Omitting the context would lead to incompatibilities with following code (ex: readFileAfterOpen(), where the context is extracted from this).

[trevnorris]

the default context is the object to the left of the . so I think @thefourtheye is right in this case.

[jwueller]

Of course he is. I actually tripped myself there. Sorry! I'll clean that up immediately.

[jwueller]

@trevnorris One more fix. Thanks @thefourtheye for reviewing!

[trevnorris]

Final CI for sanity: https://ci.nodejs.org/job/node-test-pull-request/518/

If CI is good let's land this thing! LGTM

[jwueller]

@trevnorris Something seems to have failed, but it is not obvious to me what is happening, exactly. Can you help me out?

[trevnorris]

@jwueller None of the failing tests are related to your PR. Everything seems fine.

[trevnorris]

Landed on 0803962. Thanks!

Note: This is labelled as semver-minor but it should not land on LTS.

[jwueller]

@trevnorris Great! Thanks again!