fs: refactor "options" processing as a function #7165

thefourtheye · 2016-06-05T21:37:06Z

Checklist

tests and code linting passes
the commit message follows commit guidelines

Affected core subsystem(s)

fs

Description of change

As it is the "options" processing is repeated in all the functions
which need it. That introduces checks which are inconsistent with
similar functions and produces slightly different error messages.

This patch moves the basic "options" validation and processing to a
separate function.

cc @nodejs/fs @nodejs/collaborators

Marking this as major, as this might break some code in the wild.

thefourtheye · 2016-06-05T21:38:30Z

CI Run: https://ci.nodejs.org/job/node-test-pull-request/2928/

mscdex · 2016-06-05T21:39:43Z

lib/fs.js

@@ -41,6 +41,25 @@ const isWindows = process.platform === 'win32';
 const DEBUG = process.env.NODE_DEBUG && /fs/.test(process.env.NODE_DEBUG);
 const errnoException = util._errnoException;

+function getOptions(options, defaultOptions) {
+  let actualType = typeof options;


FWIW v8 still does not (IIRC) optimize storing typeof foo in a variable. It's faster to just inline the typeof without using a variable (e.g. typeof actualType === 'function').

That's how I understand it too.

benjamingr · 2016-06-05T22:02:40Z

Generally +1 and LGTM, left a nit.

I think this can safely be a minor - this should only break code that parses error messages - right?

mscdex · 2016-06-05T22:22:32Z

AFAIK changing error message(s) still makes it a semver-major change.

seishun · 2016-06-06T05:40:45Z

While we're at it, either the error should say "object that isn't a function", or functions should be accepted as option bags too.

thefourtheye · 2016-06-06T12:24:11Z

@seishun The default options is used because, the functions which call getOptions handle the "options passed which are actually functions" case.

trevnorris · 2016-06-13T22:50:07Z

lib/fs.js

+                        typeof options + ' instead.');
+  }
+
+  options.encoding === 'buffer' || assertEncoding(options.encoding);


Can we please place this in an if (). no sense abusing JS's logical operators to run code.

Ya sure... I changed it to an if block.

silverwind · 2016-06-15T18:47:30Z

test/parallel/test-fs-read-stream-throw-type-error.js

-assert.throws(function() {
-  fs.createReadStream(example, 0);
-}, /"options" argument must be a string or an object/);
+}, /"options" must be a string or an object/);


Why not keep the other tests and adapt the string?

@silverwind Now, I just changed the strings and allowed only null.

silverwind · 2016-06-15T18:48:07Z

LGTM with question.

thefourtheye · 2016-06-30T17:23:45Z

Bump!

trevnorris · 2016-06-30T19:25:03Z

lib/fs.js

@@ -41,6 +41,24 @@ const isWindows = process.platform === 'win32';
 const DEBUG = process.env.NODE_DEBUG && /fs/.test(process.env.NODE_DEBUG);
 const errnoException = util._errnoException;

+function getOptions(options, defaultOptions) {
+  if (options == null || typeof options === 'function') {


Just want to double check that using == null is intentional?

Yes, @trevnorris. It is intentional.

Sometimes people pass null and sometimes undefined. That is why I wrote it like this. Eg: https://github.com/isaacs/node-graceful-fs/blob/master/graceful-fs.js#L89

(a bit late, sorry)
My only beef with this is that next time a developer sees this, they may become eager to "fix" this, causing either a bug or a new discussion about this pattern. My preference is to just be explicit and check for null and undefined explicitly with ===. Just my 2 cents.

@ronkorving Fair Enough. Changed it.

trevnorris · 2016-06-30T19:25:48Z

One question, otherwise LGTM

thefourtheye · 2016-07-12T02:27:59Z

Okay, added a commit to fix #7655 as well. PTAL.

Fishrock123 · 2016-07-12T09:53:46Z

lib/fs.js

+  if (options.encoding !== 'buffer')
+    assertEncoding(options.encoding);
+
+  return Object.assign(defaultOptions, options);


We should probably only do this where we need to modify it, otherwise there will be a hit to each call

@Fishrock123 You mean if we don't actually modify options, we don't want to create this copy?

Hmmm, that's what I am afraid of. Then the actual logic of options processing will not at the same place. Perhaps we can have a second parameter in the function which can be used to flag if a copy has to be made. What do you think?

@Fishrock123 I changed it to util._extend from Object.assign. Hope the hit will not be that much now, at least as per #7655 (comment)

How does util._extend solve this problem? By the way, Object.assign should've become quite a bit faster since V8 5.1 (see http://v8project.blogspot.jp/2016/04/v8-release-51.html). Now I do believe someone was adding a benchmark to compare the two, so I'm not making the assertion that util._extends should be avoided (yet). But a copy is a copy, and is probably more than we need in quite a few cases.

To think of it, it just copies things to defaultOptions. That object is constructed already in memory. Would this still be a problem?

micnic · 2016-07-13T10:44:32Z

LGTM
I'm just wondering if we should also fix #7655 for v4.x or not

MylesBorins · 2016-07-13T14:48:41Z

@micnic how do you think it could be done in a non semver major way?

micnic · 2016-07-15T09:15:04Z

@thealphanerd something like options = util._extend({}, options); as the first expression in the affected methods would do the job, I don't think it is a semver major because it's an expected behavior and everywhere in the core modules except for some fs methods the external objects are not modified. In any case the impact of this change would be minimal, I could discover this only by using Object.freeze(), most of the users are using inline defined objects without reusing or freezing them.

thefourtheye · 2016-07-21T23:13:19Z

Rebased. CI Run: https://ci.nodejs.org/job/node-test-pull-request/3380/

addaleax · 2016-08-27T11:02:34Z

@thealphanerd started a comparison CITGM run with v6.3.1 yesterday with the same vinyl-fs failures: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/371/ (from #8253 (comment)).

That shows the same failures, so the failures here are likely unrelated…

MylesBorins · 2016-08-27T11:34:16Z

/cc @phated

On Sat, Aug 27, 2016, 11:02 AM Anna Henningsen notifications@github.com
wrote:

@thealphanerd https://github.com/TheAlphaNerd started a comparison
CITGM run with v6.3.1 yesterday with the same vinyl-fs failures:
https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/371/ (from #8253
(comment)
#8253 (comment)).

That shows the same failures, so the failures here are likely unrelated…

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7165 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAecVyWwn5yzmRmF-aUFNFUWrqXpBP_iks5qkBlSgaJpZM4IudDT
.

thefourtheye · 2016-09-02T06:34:26Z

Squashed and Rebased.

thefourtheye · 2016-09-28T11:09:26Z

New CI Run: https://ci.nodejs.org/job/node-test-pull-request/4305/

jasnell · 2016-09-30T14:55:08Z

CI looks good other than a known flaky failure. @thefourtheye, do you want to go ahead and get this landed?

As it is, the "options" processing is repeated in all the functions which need it. That introduces checks which are inconsistent with other functions and produces slightly different error messages. This patch moves the basic "options" validation and processing to a seperate function.

thefourtheye · 2016-10-05T18:50:46Z

Landed in e8e969a. Thanks for the follow-up @jasnell :-)

As it is, the "options" processing is repeated in all the functions which need it. That introduces checks which are inconsistent with other functions and produces slightly different error messages. This patch moves the basic "options" validation and processing to a seperate function. PR-URL: #7165 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Roman Reiss <me@silverwind.io> Reviewed-By: Trevor Norris <trev.norris@gmail.com> Reviewed-By: Nicu Micleușanu <micnic90@gmail.com> Reviewed-By: Yorkie Liu <yorkiefixer@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

addaleax · 2016-10-05T18:54:59Z

Is this only semver-major due to the changed error messages? If so, would it make sense to pull this into v7.0.0? To me the diff looks like it could likely cause conflicts when backporting to v7 otherwise…

thefourtheye · 2016-10-05T20:18:12Z

@addaleax The encoding assertion happens for all the functions now. That also would qualify as a major change I believe.

addaleax · 2016-10-05T20:21:05Z

Okay then, that’s a bit more than just changed messages. I’m not sure, but maybe it’s still worth to ping @nodejs/ctc and see if anybody feels strongly about it?

Fishrock123 · 2016-10-05T21:10:52Z

yeah, prefer major.

If you want it in v7 we should discuss this now

jasnell · 2016-10-06T18:55:26Z

I'm ok with pulling this into v7. @nodejs/ctc ... thoughts?

addaleax · 2016-10-06T20:39:57Z

Seems okay to me, yes. 👍

jasnell · 2016-10-06T21:32:26Z

If there are no objections from @nodejs/ctc by Monday, I'll pull this in to v7.x-staging

As it is, the "options" processing is repeated in all the functions which need it. That introduces checks which are inconsistent with other functions and produces slightly different error messages. This patch moves the basic "options" validation and processing to a seperate function. PR-URL: #7165 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Roman Reiss <me@silverwind.io> Reviewed-By: Trevor Norris <trev.norris@gmail.com> Reviewed-By: Nicu Micleușanu <micnic90@gmail.com> Reviewed-By: Yorkie Liu <yorkiefixer@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

As it is, the "options" processing is repeated in all the functions which need it. That introduces checks which are inconsistent with other functions and produces slightly different error messages. This patch moves the basic "options" validation and processing to a seperate function. PR-URL: nodejs/node#7165 Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Roman Reiss <me@silverwind.io> Reviewed-By: Trevor Norris <trev.norris@gmail.com> Reviewed-By: Nicu Micleușanu <micnic90@gmail.com> Reviewed-By: Yorkie Liu <yorkiefixer@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

thefourtheye added fs Issues and PRs related to the fs subsystem / file system. semver-major PRs that contain breaking changes and should be released in the next major version. labels Jun 5, 2016

mscdex reviewed Jun 5, 2016
View reviewed changes

trevnorris reviewed Jun 13, 2016
View reviewed changes

thefourtheye force-pushed the refactor-fs-options branch from 211453d to 8de37d0 Compare June 15, 2016 10:50

silverwind reviewed Jun 15, 2016
View reviewed changes

thefourtheye force-pushed the refactor-fs-options branch 2 times, most recently from ffd1fb8 to 706316c Compare June 25, 2016 18:35

trevnorris reviewed Jun 30, 2016
View reviewed changes

thefourtheye force-pushed the refactor-fs-options branch from 706316c to 59ceb95 Compare July 11, 2016 14:16

thefourtheye mentioned this pull request Jul 11, 2016

External objects should not be modified #7655

Closed

Fishrock123 reviewed Jul 12, 2016
View reviewed changes

thefourtheye force-pushed the refactor-fs-options branch from 8608551 to 2c279ed Compare July 21, 2016 23:05

thefourtheye force-pushed the refactor-fs-options branch 3 times, most recently from 5c0feb3 to 845eeaf Compare July 22, 2016 13:00

thefourtheye force-pushed the refactor-fs-options branch from f502293 to 1b13514 Compare September 2, 2016 06:33

Trott force-pushed the master branch from b0df363 to c5ce7f4 Compare September 21, 2016 00:09

silverwind approved these changes Sep 28, 2016

View reviewed changes

thefourtheye force-pushed the refactor-fs-options branch from 1b13514 to 4693ee5 Compare October 5, 2016 18:42

thefourtheye closed this Oct 5, 2016

thefourtheye deleted the refactor-fs-options branch October 5, 2016 18:51

jasnell added this to the 7.0.0 milestone Oct 6, 2016

jasnell mentioned this pull request Oct 10, 2016

fs: don't alter user provided options object #7831

Closed

3 tasks

mrsimicsak mentioned this pull request Dec 20, 2016

OpenJSCAD throws fs.js error under node.js 7.0.0 jscad/OpenJSCAD.org#198

Closed

thefourtheye mentioned this pull request Mar 16, 2017

fs: getOption to populate passed in default values #11630

Closed

2 tasks

gibfahn mentioned this pull request Jun 15, 2017

Auditing for 6.11.1 nodejs/Release#230

Closed

3 tasks

corhere mentioned this pull request Aug 20, 2018

Normalize null options correctly jvilk/BrowserFS#254

Merged

fs: refactor "options" processing as a function #7165

fs: refactor "options" processing as a function #7165

Conversation

thefourtheye commented Jun 5, 2016

Checklist

Affected core subsystem(s)

Description of change

thefourtheye commented Jun 5, 2016

mscdex Jun 5, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjamingr commented Jun 5, 2016

mscdex commented Jun 5, 2016

seishun commented Jun 6, 2016

thefourtheye commented Jun 6, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

silverwind commented Jun 15, 2016

thefourtheye commented Jun 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trevnorris commented Jun 30, 2016

thefourtheye commented Jul 12, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thefourtheye Jul 13, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

micnic commented Jul 13, 2016

MylesBorins commented Jul 13, 2016

micnic commented Jul 15, 2016

thefourtheye commented Jul 21, 2016

addaleax commented Aug 27, 2016

MylesBorins commented Aug 27, 2016

thefourtheye commented Sep 2, 2016

thefourtheye commented Sep 28, 2016

jasnell commented Sep 30, 2016

thefourtheye commented Oct 5, 2016

addaleax commented Oct 5, 2016

thefourtheye commented Oct 5, 2016

addaleax commented Oct 5, 2016

Fishrock123 commented Oct 5, 2016

jasnell commented Oct 6, 2016

addaleax commented Oct 6, 2016

jasnell commented Oct 6, 2016

mscdex Jun 5, 2016 •

edited

thefourtheye Jul 13, 2016 •

edited