New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: fix reads with pos > 4GB #21003

Closed
wants to merge 1 commit into
base: master
from

Conversation

@mafintosh
Member

mafintosh commented May 28, 2018

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines

This fixes an issue in node 10 where all file reads with position > 4GB returns the data stored at position 0 in the file.

I would appreciate a quick review/release on this as this can have pretty serious consequences obvs (I just corrupted some local databases because of it).

A test case is available here: https://gist.github.com/mafintosh/1f9adf37bbc7ea05f1d2da6ab2d0f7a1

I'm happy to add it but not sure if we expect the tests to run on file systems that all support sparse files. Otherwise it would be a very slow test case.

@addaleax

This comment has been minimized.

Member

addaleax commented May 28, 2018

This applies to readSync as well, I think :/

@mafintosh

This comment has been minimized.

Member

mafintosh commented May 28, 2018

@addaleax let me update it

@mafintosh

This comment has been minimized.

Member

mafintosh commented May 28, 2018

added fix for readSync also

@TimothyGu

Is it possible to write a test for this?

lib/fs.js Outdated
@@ -489,7 +489,7 @@ function readSync(fd, buffer, offset, length, position) {
validateOffsetLengthRead(offset, length, buffer.length);
if (!isUint32(position))
if (!Number.isNumber(position))

This comment has been minimized.

@TimothyGu

TimothyGu May 28, 2018

Member

isNumber? ;)

lib/fs.js Outdated
@@ -459,7 +459,7 @@ function read(fd, buffer, offset, length, position, callback) {
validateOffsetLengthRead(offset, length, buffer.length);
if (!isUint32(position))
if (!Number.isInteger(position))

This comment has been minimized.

@TimothyGu

TimothyGu May 28, 2018

Member

You probably want isSafeInteger, since non-safe integers are not guaranteed to represent what the user wants (and are thus invalid). We also use isSafeInteger in the validateInteger validator.

This comment has been minimized.

@mafintosh

mafintosh May 28, 2018

Member

Sure, just putting what was in there before

@mscdex

This comment has been minimized.

Contributor

mscdex commented May 28, 2018

Shouldn't it be possible to test this without large/sparse files by testing whether the large position argument reads from the current position or doesn't read anything? For example:

'use strict';
const fs = require('fs');
const assert = require('assert');

const fd = fs.openSync(__filename, 'r');
const nRead = fs.readSync(fd, Buffer.alloc(1), 0, 1, 900719925474099);
assert.strictEqual(nRead, 0);
@TimothyGu

Pending a test. Otherwise LGTM.

@mafintosh

This comment has been minimized.

Member

mafintosh commented May 28, 2018

@mscdex that's a nice compromise 👍

@mafintosh

This comment has been minimized.

Member

mafintosh commented May 28, 2018

Added a test for both read and readSync

@mafintosh

This comment has been minimized.

@jasnell

+1 for fast track

@addaleax addaleax added the fast-track label May 28, 2018

@addaleax

This comment has been minimized.

Member

addaleax commented May 28, 2018

I’m okay with fast-tracking as well, but in that case it’s a good idea to open an issue immediately afterwards for the fact that we want a (proper) regression test.

@@ -53,3 +53,15 @@ test(Buffer.allocUnsafe(expected.length),
test(new Uint8Array(expected.length),
new Uint8Array(expected.length),
Uint8Array.from(expected));
{
// Reading > 4gb should return no data

This comment has been minimized.

@mscdex

mscdex May 28, 2018

Contributor

This code comment is a little confusing because it's merely reading past the end of file that results in no data. That can happen no matter the file size.

Perhaps it should instead mention that it's testing whether >32-bit position arguments read from the absolute (good) or current (bad) position and that reading beyond the file is what should return no data.

This comment has been minimized.

@mafintosh

mafintosh May 28, 2018

Member

tweaked the comment

@mcollina

Lgtm. This should get info a release asap

@lpinca

lpinca approved these changes May 28, 2018

{
// Reading beyond file length (3 in this case) should return no data.
// This is a test for a bug where reads > uin32 would return

This comment has been minimized.

@mscdex

mscdex May 28, 2018

Contributor

s/uin32/uint32/

{
// Reading beyond file length (3 in this case) should return no data.
// This is a test for a bug where reads > uin32 would return
// the start of the file.

This comment has been minimized.

@mscdex

mscdex May 28, 2018

Contributor

s/the start of the file/data from the current position in the file/

This comment has been minimized.

@mafintosh

mafintosh May 28, 2018

Member

it seems to be the start of the file

This comment has been minimized.

@mafintosh

mafintosh May 28, 2018

Member

nope you're right.

@mscdex

This comment has been minimized.

Contributor

mscdex commented May 28, 2018

I think the test will need to be isolated (a separate openSync()) from the other read tests in that file because the preceding tests will have read the entire file, meaning the read()/readSync() would always return 0 because the current file pointer is at the end of the file.

@mafintosh

This comment has been minimized.

Member

mafintosh commented May 28, 2018

@mscdex you sure? the test fails for me if running it on node 10

@mscdex

This comment has been minimized.

Contributor

mscdex commented May 28, 2018

I forgot that specifying absolute file positions does not change the current file pointer, so that probably sounds right.

@mscdex

This comment has been minimized.

Contributor

mscdex commented May 28, 2018

It still might be a good idea to have a separate openSync() though, in case some other read test gets added that messes with the current file pointer.

@mafintosh

This comment has been minimized.

Member

mafintosh commented May 28, 2018

@mscdex made it use it's own fd anyway for good meassure

@mscdex

This comment has been minimized.

Contributor

mscdex commented May 28, 2018

LGTM now provided CI still passes

@MylesBorins MylesBorins referenced this pull request May 29, 2018

Merged

v10.3.0 proposal #21011

MylesBorins added a commit that referenced this pull request May 29, 2018

2018-05-29, Version 10.3.0 (Current)
Notable Changes:

* **deps**:
  - upgrade npm to 6.1.0 (Rebecca Turner)
    #20190
* **fs**:
  - fix reads with pos \> 4GB (Mathias Buus)
    #21003
* **net**:
  - new option to allow IPC servers to be readable and writable
    by all users (Bartosz Sosnowski)
    #19472
* **Added new collaborators**
  - John-David Dalton (https://github.com/jdalton)

PR-URL: #21011

MylesBorins added a commit that referenced this pull request May 29, 2018

2018-05-29, Version 10.3.0 (Current)
Notable Changes:

* **deps**:
  - upgrade npm to 6.1.0 (Rebecca Turner)
    #20190
* **fs**:
  - fix reads with pos \> 4GB (Mathias Buus)
    #21003
* **net**:
  - new option to allow IPC servers to be readable and writable
    by all users (Bartosz Sosnowski)
    #19472
* **stream**:
  - fix removeAllListeners() for Stream.Readable to work as expected
    when no arguments are passed (Kael Zhang)
    #20924
* **Added new collaborators**
  - John-David Dalton (https://github.com/jdalton)

PR-URL: #21011

MylesBorins added a commit that referenced this pull request May 29, 2018

2018-05-29, Version 10.3.0 (Current)
Notable Changes:

* **deps**:
  - upgrade npm to 6.1.0 (Rebecca Turner)
    #20190
* **fs**:
  - fix reads with pos \> 4GB (Mathias Buus)
    #21003
* **net**:
  - new option to allow IPC servers to be readable and writable
    by all users (Bartosz Sosnowski)
    #19472
* **stream**:
  - fix removeAllListeners() for Stream.Readable to work as expected
    when no arguments are passed (Kael Zhang)
    #20924
* **Added new collaborators**
  - John-David Dalton (https://github.com/jdalton)

PR-URL: #21011

MylesBorins added a commit that referenced this pull request May 29, 2018

2018-05-29, Version 10.3.0 (Current)
Notable Changes:

* **deps**:
  - upgrade npm to 6.1.0 (Rebecca Turner)
    #20190
* **fs**:
  - fix reads with pos \> 4GB (Mathias Buus)
    #21003
* **net**:
  - new option to allow IPC servers to be readable and writable
    by all users (Bartosz Sosnowski)
    #19472
* **stream**:
  - fix removeAllListeners() for Stream.Readable to work as expected
    when no arguments are passed (Kael Zhang)
    #20924
* **Added new collaborators**
  - John-David Dalton (https://github.com/jdalton)

PR-URL: #21011

shisama added a commit to shisama/node that referenced this pull request May 30, 2018

fs: fix reads with pos > 4GB
PR-URL: nodejs#21003
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>

shisama added a commit to shisama/node that referenced this pull request May 30, 2018

2018-05-29, Version 10.3.0 (Current)
Notable Changes:

* **deps**:
  - upgrade npm to 6.1.0 (Rebecca Turner)
    nodejs#20190
* **fs**:
  - fix reads with pos \> 4GB (Mathias Buus)
    nodejs#21003
* **net**:
  - new option to allow IPC servers to be readable and writable
    by all users (Bartosz Sosnowski)
    nodejs#19472
* **stream**:
  - fix removeAllListeners() for Stream.Readable to work as expected
    when no arguments are passed (Kael Zhang)
    nodejs#20924
* **Added new collaborators**
  - John-David Dalton (https://github.com/jdalton)

PR-URL: nodejs#21011

shisama added a commit to shisama/node that referenced this pull request May 30, 2018

fs: fix reads with pos > 4GB
PR-URL: nodejs#21003
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>

@cjihrig cjihrig referenced this pull request Jun 5, 2018

Merged

fs: fix promises reads with pos > 4GB #21148

3 of 3 tasks complete

shisama added a commit to shisama/node that referenced this pull request Jun 5, 2018

fs: fix reads with pos > 4GB
PR-URL: nodejs#21003
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>

shisama added a commit to shisama/node that referenced this pull request Jun 5, 2018

2018-05-29, Version 10.3.0 (Current)
Notable Changes:

* **deps**:
  - upgrade npm to 6.1.0 (Rebecca Turner)
    nodejs#20190
* **fs**:
  - fix reads with pos \> 4GB (Mathias Buus)
    nodejs#21003
* **net**:
  - new option to allow IPC servers to be readable and writable
    by all users (Bartosz Sosnowski)
    nodejs#19472
* **stream**:
  - fix removeAllListeners() for Stream.Readable to work as expected
    when no arguments are passed (Kael Zhang)
    nodejs#20924
* **Added new collaborators**
  - John-David Dalton (https://github.com/jdalton)

PR-URL: nodejs#21011

cjihrig added a commit to cjihrig/node-1 that referenced this pull request Jun 7, 2018

fs: fix promises reads with pos > 4GB
PR-URL: nodejs#21148
Fixes: nodejs#21121
Refs: nodejs#21003
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Ujjwal Sharma <usharma1998@gmail.com>

targos added a commit that referenced this pull request Jun 7, 2018

fs: fix promises reads with pos > 4GB
PR-URL: #21148
Fixes: #21121
Refs: #21003
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Ujjwal Sharma <usharma1998@gmail.com>

shisama added a commit to shisama/node that referenced this pull request Jun 9, 2018

fs: fix reads with pos > 4GB
PR-URL: nodejs#21003
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>

shisama added a commit to shisama/node that referenced this pull request Jun 9, 2018

2018-05-29, Version 10.3.0 (Current)
Notable Changes:

* **deps**:
  - upgrade npm to 6.1.0 (Rebecca Turner)
    nodejs#20190
* **fs**:
  - fix reads with pos \> 4GB (Mathias Buus)
    nodejs#21003
* **net**:
  - new option to allow IPC servers to be readable and writable
    by all users (Bartosz Sosnowski)
    nodejs#19472
* **stream**:
  - fix removeAllListeners() for Stream.Readable to work as expected
    when no arguments are passed (Kael Zhang)
    nodejs#20924
* **Added new collaborators**
  - John-David Dalton (https://github.com/jdalton)

PR-URL: nodejs#21011

shisama added a commit to shisama/node that referenced this pull request Jun 9, 2018

fs: fix promises reads with pos > 4GB
PR-URL: nodejs#21148
Fixes: nodejs#21121
Refs: nodejs#21003
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Ujjwal Sharma <usharma1998@gmail.com>

shisama added a commit to shisama/node that referenced this pull request Jun 11, 2018

fs: fix reads with pos > 4GB
PR-URL: nodejs#21003
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>

shisama added a commit to shisama/node that referenced this pull request Jun 11, 2018

2018-05-29, Version 10.3.0 (Current)
Notable Changes:

* **deps**:
  - upgrade npm to 6.1.0 (Rebecca Turner)
    nodejs#20190
* **fs**:
  - fix reads with pos \> 4GB (Mathias Buus)
    nodejs#21003
* **net**:
  - new option to allow IPC servers to be readable and writable
    by all users (Bartosz Sosnowski)
    nodejs#19472
* **stream**:
  - fix removeAllListeners() for Stream.Readable to work as expected
    when no arguments are passed (Kael Zhang)
    nodejs#20924
* **Added new collaborators**
  - John-David Dalton (https://github.com/jdalton)

PR-URL: nodejs#21011

shisama added a commit to shisama/node that referenced this pull request Jun 11, 2018

fs: fix promises reads with pos > 4GB
PR-URL: nodejs#21148
Fixes: nodejs#21121
Refs: nodejs#21003
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Ujjwal Sharma <usharma1998@gmail.com>

shisama added a commit to shisama/node that referenced this pull request Jun 12, 2018

fs: fix reads with pos > 4GB
PR-URL: nodejs#21003
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>

shisama added a commit to shisama/node that referenced this pull request Jun 12, 2018

2018-05-29, Version 10.3.0 (Current)
Notable Changes:

* **deps**:
  - upgrade npm to 6.1.0 (Rebecca Turner)
    nodejs#20190
* **fs**:
  - fix reads with pos \> 4GB (Mathias Buus)
    nodejs#21003
* **net**:
  - new option to allow IPC servers to be readable and writable
    by all users (Bartosz Sosnowski)
    nodejs#19472
* **stream**:
  - fix removeAllListeners() for Stream.Readable to work as expected
    when no arguments are passed (Kael Zhang)
    nodejs#20924
* **Added new collaborators**
  - John-David Dalton (https://github.com/jdalton)

PR-URL: nodejs#21011

shisama added a commit to shisama/node that referenced this pull request Jun 12, 2018

fs: fix promises reads with pos > 4GB
PR-URL: nodejs#21148
Fixes: nodejs#21121
Refs: nodejs#21003
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Ujjwal Sharma <usharma1998@gmail.com>

shisama added a commit to shisama/node that referenced this pull request Jun 17, 2018

fs: fix reads with pos > 4GB
PR-URL: nodejs#21003
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>

shisama added a commit to shisama/node that referenced this pull request Jun 17, 2018

2018-05-29, Version 10.3.0 (Current)
Notable Changes:

* **deps**:
  - upgrade npm to 6.1.0 (Rebecca Turner)
    nodejs#20190
* **fs**:
  - fix reads with pos \> 4GB (Mathias Buus)
    nodejs#21003
* **net**:
  - new option to allow IPC servers to be readable and writable
    by all users (Bartosz Sosnowski)
    nodejs#19472
* **stream**:
  - fix removeAllListeners() for Stream.Readable to work as expected
    when no arguments are passed (Kael Zhang)
    nodejs#20924
* **Added new collaborators**
  - John-David Dalton (https://github.com/jdalton)

PR-URL: nodejs#21011

shisama added a commit to shisama/node that referenced this pull request Jun 17, 2018

fs: fix promises reads with pos > 4GB
PR-URL: nodejs#21148
Fixes: nodejs#21121
Refs: nodejs#21003
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: Ujjwal Sharma <usharma1998@gmail.com>

codebytere added a commit to electron/node that referenced this pull request Aug 15, 2018

fix: fs.readSync with large position value
Backport of nodejs/node#21003; fixed after 10.3.0 so this does not also need to be applied to the 10.6.0 upgrade

codebytere added a commit to electron/node that referenced this pull request Aug 15, 2018

fix: fs.readSync with large position value
Backport to nodejs/node#21003; fixed after 10.3.0 so this does not also need to be applied to the 10.6.0 upgrade
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment