New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: pass err to callback if buffer is too big #3485

Merged
merged 1 commit into from Oct 23, 2015

Conversation

Projects
None yet
5 participants
@evanlucas
Member

evanlucas commented Oct 22, 2015

In fs.readFile, if an encoding is specified and toString fails, do not
throw an error. Instead, pass the error to the callback.

Fixes: #2767

@evanlucas

This comment has been minimized.

Member

evanlucas commented Oct 22, 2015

Went ahead and started the CI to make sure the test passes on all platforms.

https://ci.nodejs.org/job/node-test-pull-request/565/

@mscdex mscdex added the fs label Oct 22, 2015

@jasnell

View changes

lib/fs.js Outdated
if (context.encoding) {
const result = tryToString(buffer, context.encoding);
if (result.error)
return callback(result.error, buffer);

This comment has been minimized.

@jasnell

jasnell Oct 22, 2015

Member

In the error case, I'd much rather the second argument be undefined as opposed to it being a different type from the success case in line 405. Perhaps something like, callback(err,undefined,buffer), and in the success case, callback(null, result.buffer, buffer), so that the second and third arguments are always consistent?

This comment has been minimized.

@jasnell

jasnell Oct 22, 2015

Member

Ugh... nevermind on that, I see you're just copying the existing behavior on this.

This comment has been minimized.

@evanlucas

evanlucas Oct 22, 2015

Member

Yea, that is why I did that. And honestly, I feel like this is something that should go into v4.x if possible. We could make it better for v5?

This comment has been minimized.

@jasnell

jasnell Oct 22, 2015

Member

However, since you're returning in either case, why not simply pass the callback to the tryToString method...?

if (context.encoding)
  return tryToString(buffer, context.encoding, callback);
//...

function tryToString(buf, encoding, callback) {
  try {
    return callback(null, buf.toString(encoding));
  }
  catch (err) {
    return callback(err, buf);
  }
}

This comment has been minimized.

@evanlucas

evanlucas Oct 22, 2015

Member

Yea, I did it that way first, and then changed it to this way. Not sure why now... :]

Will update

This comment has been minimized.

@jasnell

jasnell Oct 22, 2015

Member

Given that it removes a throw, it's at least a semver-minor or major, which means it wouldn't make it in to v4.x

This comment has been minimized.

@evanlucas

evanlucas Oct 22, 2015

Member

ah, true. That really sucks...

@evanlucas evanlucas force-pushed the evanlucas:fixreadfileerr branch Oct 22, 2015

@evanlucas

This comment has been minimized.

Member

evanlucas commented Oct 22, 2015

ok, updated

@jasnell

This comment has been minimized.

Member

jasnell commented Oct 22, 2015

I'm going to tag this semver-major for now given the change in the throw. @nodejs/tsc if any of you feel it could be a minor, feel free to change it :-) ... otherwise, LGTM so long as CI is green

@trevnorris

View changes

test/parallel/test-fs-readfile-tostring-fail.js Outdated
const size = 269 * 1024 * 1024;
const a = new Array(1000000).join('a').toString();

This comment has been minimized.

@trevnorris

trevnorris Oct 22, 2015

Contributor

much less expensive to do const a = Buffer(1000000).fill('a').toString();

This comment has been minimized.

@evanlucas

evanlucas Oct 22, 2015

Member

good call. knocked about 1/2 a second off of it locally

This comment has been minimized.

@targos

targos Oct 23, 2015

Member

what about const a = 'a'.repeat(1000000) ?

This comment has been minimized.

@trevnorris

trevnorris Oct 23, 2015

Contributor

The operation itself is faster, but operating on that string is slower.

'a'.repeat(1000000):                    4.5 us/op
Buffer(1000000).fill('a').toString(): 180.8 us/op

Now turn those back into a Buffer to be written to disk:

Buffer('a'.repeat(1000000)):          7060.1 us/op
Buffer(''+Buffer(1000000).fill('a')):  721.8 us/op

Though there may be some bias from running these in a loop. So overall test time would be the best measurement.

@trevnorris

View changes

test/parallel/test-fs-readfile-tostring-fail.js Outdated
flags: 'a'
});
const size = 269 * 1024 * 1024;

This comment has been minimized.

@trevnorris

trevnorris Oct 22, 2015

Contributor

where does this number come from?

This comment has been minimized.

@evanlucas

evanlucas Oct 22, 2015

Member

That was the number suggested in the issue.

This comment has been minimized.

@trevnorris

trevnorris Oct 23, 2015

Contributor

I think the value is supposed to be process.binding('buffer').kStringMaxLength.

@trevnorris

View changes

test/parallel/test-fs-readfile-tostring-fail.js Outdated
stream.on('finish', common.mustCall(function() {
// make sure that the toString does not throw an error
fs.readFile(file, 'utf8', common.mustCall(function(err, buf) {
assert.strictEqual('toString failed', err.message);

This comment has been minimized.

@trevnorris

trevnorris Oct 22, 2015

Contributor

This test will be flaky on the pi's since there's higher likelihood that they'll run out of memory.

This comment has been minimized.

@evanlucas

evanlucas Oct 22, 2015

Member

Yea, I had considered just stubbing toString to return an error to verify that if it fails, doesn't throw, but passes it to the callback, but didn't see that as being that acceptable.

This comment has been minimized.

@evanlucas

evanlucas Oct 22, 2015

Member

I am definitely open to suggestions on improvement though :]

This comment has been minimized.

@trevnorris

trevnorris Oct 23, 2015

Contributor

my apologies. i'm not saying this is a bad test, only that it has a greater likelihood of failing on the pi's. much like the extern string tests were failing for a while until some recent refactoring.

@evanlucas

This comment has been minimized.

Member

evanlucas commented Oct 22, 2015

@trevnorris this isn't ideal, but works and runs in < 0.1 seconds:

'use strict';

const common = require('../common');
const assert = require('assert');
const fs = require('fs');
const path = require('path');

common.refreshTmpDir();

const file = path.join(common.tmpDir, 'toobig.txt');
const stream = fs.createWriteStream(file, {
  flags: 'a'
});

const a = new Buffer(100).fill('a').toString();

for (var i = 0; i < 20000; i++) {
  stream.write(a);
}

stream.end();
stream.on('finish', common.mustCall(function() {
  const orig = Buffer.prototype.toString;
  Buffer.prototype.toString = function() {
    throw new Error('toString failed');
  }
  // make sure that the toString does not throw an error
  fs.readFile(file, 'utf8', common.mustCall(function(err, buf) {
    Buffer.prototype.toString = orig;
    assert.strictEqual('toString failed', err.message);
  }));
}));

process.on('exit', function() {
  fs.unlinkSync(file);
});

@evanlucas evanlucas referenced this pull request Oct 23, 2015

Closed

v5.0.0 Planning & Checklist #3397

13 of 13 tasks complete
@trevnorris

View changes

lib/fs.js Outdated
try {
buf = buf.toString(encoding);
}
catch (err) {

This comment has been minimized.

@trevnorris

trevnorris Oct 23, 2015

Contributor

style? i thought the linter would catch this. catch should be on the same line as }.

This comment has been minimized.

@targos

targos Oct 23, 2015

Member

current version of the eslint doesn't catch it but it will after the update.

@trevnorris

This comment has been minimized.

Contributor

trevnorris commented Oct 23, 2015

@evanlucas Need to make sure that the amount written to disk is greater than process.binding('buffer').kStringMaxLength.

Oh, why are you toString()'ing the data being written to disk? You can simply pass the buffer directly.

@evanlucas evanlucas force-pushed the evanlucas:fixreadfileerr branch Oct 23, 2015

@evanlucas

This comment has been minimized.

Member

evanlucas commented Oct 23, 2015

// To make sure we don't leave a very large file
// on test machines in the event this test fails.
process.on('uncaughtException', function(err) {

This comment has been minimized.

@trevnorris

trevnorris Oct 23, 2015

Contributor

may want to add process.on('SIGINT' to capture any ^C. would be the same as here, except run process.exit() at the end. optional, non-blocker.

@trevnorris

This comment has been minimized.

Contributor

trevnorris commented Oct 23, 2015

one optional suggestion, but LGTM if CI is happy.

@evanlucas evanlucas force-pushed the evanlucas:fixreadfileerr branch Oct 23, 2015

@evanlucas

This comment has been minimized.

Member

evanlucas commented Oct 23, 2015

@evanlucas

This comment has been minimized.

Member

evanlucas commented Oct 23, 2015

yay, we at least got a yellow build instead of red.

@trevnorris

This comment has been minimized.

Contributor

trevnorris commented Oct 23, 2015

Awesome. Let's land it.

@evanlucas

This comment has been minimized.

Member

evanlucas commented Oct 23, 2015

k, landing shortly

fs: pass err to callback if buffer is too big
In fs.readFile, if an encoding is specified and toString fails, do not
throw an error. Instead, pass the error to the callback.

Fixes: #2767
PR-URL: #3485
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>

@evanlucas evanlucas force-pushed the evanlucas:fixreadfileerr branch to b620790 Oct 23, 2015

@evanlucas evanlucas closed this Oct 23, 2015

@evanlucas evanlucas deleted the evanlucas:fixreadfileerr branch Oct 23, 2015

@evanlucas evanlucas merged commit b620790 into nodejs:master Oct 23, 2015

@evanlucas

This comment has been minimized.

Member

evanlucas commented Oct 23, 2015

Landed in b620790. Thanks!

evanlucas added a commit that referenced this pull request Oct 26, 2015

fs: pass err to callback if buffer is too big
In fs.readFile, if an encoding is specified and toString fails, do not
throw an error. Instead, pass the error to the callback.

Fixes: #2767
PR-URL: #3485
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>

@rvagg rvagg referenced this pull request Oct 27, 2015

Merged

Release proposal: v5.0.0 #3466

rvagg added a commit to rvagg/io.js that referenced this pull request Oct 29, 2015

2015-10-29, Version 5.0.0 (Stable)
Notable changes:

* buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer,
  these have been deprecated for a long time (Sakthipriyan Vairamani) nodejs#2859.
* console: (Breaking) Values reported by console.time() now have 3 decimals of
  accuracy added (Michaël Zasso) nodejs#3166.
* fs:
  - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file
    descriptor as their first argument (Johannes Wüller) nodejs#3163.
  - (Breaking) In fs.readFile(), if an encoding is specified and the internal
    toString() fails the error is no longer thrown but is passed to the callback
    (Evan Lucas) nodejs#3485.
  - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding,
    callback) form), if the internal toString() fails the error is no longer
    thrown but is passed to the callback (Evan Lucas) nodejs#3503.
* http:
  - Fixed a bug where pipelined http requests would stall (Fedor Indutny) nodejs#3342.
  - (Breaking) When parsing HTTP, don't add duplicates of the following headers:
    Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition
    to the following headers which already block duplicates: Content-Type,
    Content-Length, User-Agent, Referer, Host, Authorization,
    Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location,
    Max-Forwards (James M Snell) nodejs#3090.
  - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a
    function or a TypeError is thrown (James M Snell) nodejs#3090.
  - (Breaking) HTTP methods and header names must now conform to the RFC 2616
    "token" rule, a list of allowed characters that excludes control characters
    and a number of separator characters. Specifically, methods and header names
    must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown
    (James M Snell) nodejs#2526.
* node:
  - (Breaking) Deprecated the _linklist module (Rich Trott) nodejs#3078.
  - (Breaking) Removed require.paths and require.registerExtension(), both had
    been previously set to throw Error when accessed
    (Sakthipriyan Vairamani) nodejs#2922.
* npm: Upgraded to version 3.3.6 from 2.14.7, see
  https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a
  major version bump for npm and it has seen a significant amount of change.
  Please see the original npm v3.0.0 release notes for a list of major changes
  (Rebecca Turner) nodejs#3310.
* src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary
  due to the V8 upgrade. Native add-ons will need to be recompiled
  (Rod Vagg) nodejs#3400.
* timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes
  a long-standing known issue where unrefed timers would perviously hold
  beforeExit open (Fedor Indutny) nodejs#3407.
* tls:
  - Added ALPN Support (Shigeki Ohtsu) nodejs#2564.
  - TLS options can now be passed in an object to createSecurePair()
    (Коренберг Марк) nodejs#2441.
  - (Breaking) The default minimum DH key size for tls.connect() is now 1024
    bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS
    option can be used to override the default. (Shigeki Ohtsu) nodejs#1831.
* util:
  - (Breaking) util.p() was deprecated for years, and has now been removed
    (Wyatt Preul) nodejs#3432.
  - (Breaking) util.inherits() can now work with ES6 classes. This is considered
    a breaking change because of potential subtle side-effects caused by a
    change from directly reassigning the prototype of the constructor using
    `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })`
    to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)`
    (Michaël Zasso) nodejs#3455.
* v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) nodejs#3351.
  - Implements the spread operator, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator
    for further information.
  - Implements new.target, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target
    for further information.
* zlib: Decompression now throws on truncated input (e.g. unexpected end of
  file) (Yuval Brik) nodejs#2595.

PR-URL: nodejs#3466

rvagg added a commit that referenced this pull request Oct 29, 2015

2015-10-29, Version 5.0.0 (Stable)
Notable changes:

* buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer,
  these have been deprecated for a long time (Sakthipriyan Vairamani) #2859.
* console: (Breaking) Values reported by console.time() now have 3 decimals of
  accuracy added (Michaël Zasso) #3166.
* fs:
  - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file
    descriptor as their first argument (Johannes Wüller) #3163.
  - (Breaking) In fs.readFile(), if an encoding is specified and the internal
    toString() fails the error is no longer thrown but is passed to the callback
    (Evan Lucas) #3485.
  - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding,
    callback) form), if the internal toString() fails the error is no longer
    thrown but is passed to the callback (Evan Lucas) #3503.
* http:
  - Fixed a bug where pipelined http requests would stall (Fedor Indutny) #3342.
  - (Breaking) When parsing HTTP, don't add duplicates of the following headers:
    Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition
    to the following headers which already block duplicates: Content-Type,
    Content-Length, User-Agent, Referer, Host, Authorization,
    Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location,
    Max-Forwards (James M Snell) #3090.
  - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a
    function or a TypeError is thrown (James M Snell) #3090.
  - (Breaking) HTTP methods and header names must now conform to the RFC 2616
    "token" rule, a list of allowed characters that excludes control characters
    and a number of separator characters. Specifically, methods and header names
    must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown
    (James M Snell) #2526.
* node:
  - (Breaking) Deprecated the _linklist module (Rich Trott) #3078.
  - (Breaking) Removed require.paths and require.registerExtension(), both had
    been previously set to throw Error when accessed
    (Sakthipriyan Vairamani) #2922.
* npm: Upgraded to version 3.3.6 from 2.14.7, see
  https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a
  major version bump for npm and it has seen a significant amount of change.
  Please see the original npm v3.0.0 release notes for a list of major changes
  (Rebecca Turner) #3310.
* src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary
  due to the V8 upgrade. Native add-ons will need to be recompiled
  (Rod Vagg) #3400.
* timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes
  a long-standing known issue where unrefed timers would perviously hold
  beforeExit open (Fedor Indutny) #3407.
* tls:
  - Added ALPN Support (Shigeki Ohtsu) #2564.
  - TLS options can now be passed in an object to createSecurePair()
    (Коренберг Марк) #2441.
  - (Breaking) The default minimum DH key size for tls.connect() is now 1024
    bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS
    option can be used to override the default. (Shigeki Ohtsu) #1831.
* util:
  - (Breaking) util.p() was deprecated for years, and has now been removed
    (Wyatt Preul) #3432.
  - (Breaking) util.inherits() can now work with ES6 classes. This is considered
    a breaking change because of potential subtle side-effects caused by a
    change from directly reassigning the prototype of the constructor using
    `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })`
    to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)`
    (Michaël Zasso) #3455.
* v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) #3351.
  - Implements the spread operator, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator
    for further information.
  - Implements new.target, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target
    for further information.
* zlib: Decompression now throws on truncated input (e.g. unexpected end of
  file) (Yuval Brik) #2595.

PR-URL: #3466
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment