fs: work with any TypedArrays instead of only Uint8Arrays #22150

SirR4T · 2018-08-06T11:02:44Z

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Partially fixing #1826

~~Haven't added tests yet, since I'm not sure what behaviour to expect. Current tests are not breaking, though, so yay!~~
Tests have been added since.

SirR4T · 2018-08-06T11:07:40Z

cc: @TimothyGu

So, i'm not 100% sure what behaviour should be expected, when writing and reading to and from files, using TypedArrays. For instance, test/parallel/test-fs-write-file-uint8array.js fails if I replace the Uint8Array in the line

const input = Uint8Array.from(Buffer.from(s, 'utf8'));

with any other TypedArray. Why is that so? Due to this, I suspect a lot many tests should be added, to ensure read and write work correctly when using other TypedArrays.

targos · 2018-08-06T15:58:49Z

We need tests to make sure it's handled correctly on the C++ side

benjamingr

Actually, reading this again tests are missing - if we officially support those types we need to make sure we have tests making sure we don't break that.

benjamingr · 2018-08-06T22:17:55Z

If you haven't wrote tests for PRs before and want help with writing the tests or aren't sure what/where to do let me know and I'll elaborate on how that works.

SirR4T · 2018-08-07T04:11:02Z

@benjamingr I'm happy to write tests, but I'm not sure what the behaviour should be, when writing to files via TypedArrays or reading from them. Hence the hesitation. Maybe I could make copies of test/parallel/test-fs-write-file-uint8array.js with other typed arrays, and reviewers can take it from there?

benjamingr · 2018-08-07T07:36:18Z

@SirR4T basically, to check that you can perform each FS operations that supports other types of arrays and that it returns the value you expect - test-fs-write-file-uint8array.js is a good start though I'm not sure I'd create a file for every view type.

TimothyGu · 2018-08-07T22:00:00Z

test/parallel/test-fs-write-file-uint8array.js fails if I replace the Uint8Array in the line
const input = Uint8Array.from(Buffer.from(s, 'utf8'));
with any other TypedArray. Why is that so?

That is because .from() for TypedArray objects copy the data to conform to a new layout. E.g.

const a = new Uint8Array([1, 2, 3])
// In memory: 0x01 0x02 0x03

const b = Uint16Array.from(a);
// In memory 0x0001 0x0002 0x0003

However, if the test doesn't work with Int8Array, then there's a problem.

For testing, you could use getArrayBufferViews() to help simplify testing for many types of ArrayBufferViews. However, you need to make sure the buffer you pass in have a size that's a multiple of 8 bytes to make sure it works with all ArrayBufferViews (BigInt64Array and BigUint64Array have 8 bytes per element).

TimothyGu · 2018-08-07T22:00:52Z

Also, I'd be fine if you modify existing tests rather than creating new ones.

SirR4T

@benjamingr : going ahead with @TimothyGu 's suggestion, modifying the existing test case.

SirR4T · 2018-08-07T09:39:08Z

test/parallel/test-fs-file-typedarrays.js

+for (const type of types) {
+  const expect = convertToType(type, expectBuf);
+  fs.writeFileSync(filename, expect);
+  assert.strictEqual(convertToType(type, readInput(filename)), expect);


@benjamingr this test is currently failing with

assert.js:84 throw new AssertionError(obj); ^ AssertionError [ERR_ASSERTION]: Expected inputs to be strictly equal: + actual - expected ... Lines skipped Int8Array [ + 45, + 50, + 55, + 44, + 45, + 49, + 49, + 53, + 44, + 45, ... - -27, - -115, - -105, - -24, - -74, - -118, - -27, - -101, - -67, - -26, ...

What am i doing wrong?

SirR4T · 2018-08-08T05:27:09Z

@TimothyGu makes sense, going ahead with the current test case. Will modify the test case name too.

Currently, it is failing with

AssertionError [ERR_ASSERTION]: Expected inputs to be strictly equal:
+ actual - expected ... Lines skipped

+ '-27,-115,-105,-24,-74,-118,-27,-101,-67,-26,-104,-81,-27,-119,-115,50,48,51,-27,-71,-76,-24,-121,-77,-27,-119,-115,49,49,49,-27,-71,-76,-27,-83,-104,-27,-100,-88,-28,-70,-114,
... many more entries

...
- Int8Array [
-   -27,
-   -115,
-   -105,
-   -24,
-   -74,
-   -118,
-   -27,
-   -101,
-   -67,
-   -26,
-   -104,
-   -81,
-   -27,
-   -119,
-   -115,
-   50,
-   48,
-   51,
-   -27,
...
    at Object.<anonymous> (/Users/saratchandra/Play/Github/nodejs/node/test/parallel/test-fs-write-file-uint8array.js:24:10)

I notice that:

the expected value is a ~~raw Buffer~~ a string of raw buffer values concatenated 😲, whereas, I'm trying to match with a TypedArray (Int8Array here).
the actual entries for both are matching (the values in the raw buffer, i presume).

Can't shake the feeling i'm doing something terribly wrong here.
How do I ~~do exact matches between raw buffers and typed arrays~~ fix this?

SirR4T · 2018-08-10T05:20:00Z

@TimothyGu @benjamingr : The file contents are a string of numbers (positive and negative), concatenated by ,, when writing via TypedArrays (for eg., Int8Array). Is that expected behaviour?

outstanding issues

TimothyGu

Hope this answers your question.

TimothyGu · 2018-08-10T13:53:32Z

test/parallel/test-fs-write-file-uint8array.js

-assert.strictEqual(fs.readFileSync(filename, 'utf8'), s);
+for (const expectView of common.getArrayBufferViews(inputBuffer)) {
+  fs.writeFileSync(filename, expectView);
+  assert.strictEqual(fs.readFileSync(filename, 'utf8'), expectView);


fs.readFileSync(..., 'utf8') returns a string, so the assertion should compare with s.repeat(8) instead.

ah! indeed. so that will work for reads that are string based, that is, for utf8 strings, Buffers, Uint8Arrays. How should I test for the other typed arrays and data views?

For instance, when encoding is not specified fs.readFileSync() returns a Buffer. How do i match this with Int8Arrays or other typed arrays?

You don't. The test is testing if writeFile supports non-Buffer TypedArrays. It's not testing readFileSync, and you should always compare the result of readFileSync to the string.

aha! things are much more clearer now, thanks for the explanation 😄

TimothyGu · 2018-08-10T13:55:03Z

test/parallel/test-fs-write-file-uint8array.js

-}));
+    fs.readFile(filename, 'utf8', common.mustCall((err, data) => {
+      assert.ifError(err);
+      assert.strictEqual(data, expectView);


TimothyGu · 2018-08-10T13:55:41Z

test/parallel/test-fs-write-file-uint8array.js

@@ -17,13 +17,18 @@ const s = '南越国是前203年至前111年存在于岭南地区的一个国家
          '历经五代君主。南越国是岭南地区的第一个有记载的政权国家，采用封建制和郡县制并存的制度，' +
          '它的建立保证了秦末乱世岭南地区社会秩序的稳定，有效的改善了岭南地区落后的政治、##济现状。\n';

-const input = Uint8Array.from(Buffer.from(s, 'utf8'));
+const inputBuffer = Buffer.from(s.repeat(8), 'utf8');


Could you add a comment explaining why .repeat(8) is necessary? Specifically, that getArrayBufferViews expects it.

TimothyGu · 2018-08-16T20:38:19Z

Could you please rebase this branch on master?

SirR4T · 2018-08-17T05:13:01Z

@benjamingr tests added, and now passing.

@TimothyGu I can see that this diff of my feature branch against upstream/master is showing the correct files changed, but this PR is showing many more commits. How do I fix this? Should I close this and raise a different PR? This is after rebasing against latest upstream master.

TimothyGu · 2018-08-17T15:15:16Z

@SirR4T I see that your branch has this merge commit: SirR4T@590d218 You would need to get rid of that merge commit for things to work. I think just doing a git rebase with the upstream's master branch as base should be enough.

comments addressed

lundibundi · 2018-08-18T18:14:45Z

lib/fs.js

@@ -525,7 +524,7 @@ function write(fd, buffer, offset, length, position, callback) {
  const req = new FSReqCallback();
  req.oncomplete = wrapper;

-  if (isUint8Array(buffer)) {
+  if (ArrayBuffer.isView(buffer)) {


You should probably use https://github.com/nodejs/node/blob/master/lib/internal/util/types.js#L22 here and later.

good idea, will do that.

lundibundi · 2018-08-18T18:57:15Z

test/parallel/test-fs-write-file-typedarrays.js

-fs.writeFileSync(filename, input);
-assert.strictEqual(fs.readFileSync(filename, 'utf8'), s);
+for (const expectView of common.getArrayBufferViews(inputBuffer)) {
+  debuglog('Running test for: ', expectView[Symbol.toStringTag]);


I think we just use console.log in tests, don't we?

oh... I'm not sure what the current convention is, but following the discussion at #9314, I had assumed debug log was the consensus. Lemme know if I should change to console.log.

If I understand correctly, console.log statements are not printed when tests are run, but debuglog statements are printed atleast when tests fail.

TimothyGu

LGTM once @lundibundi's comments are addressed.

TimothyGu · 2018-08-18T19:13:48Z

CI: https://ci.nodejs.org/job/node-test-pull-request/16548/

TimothyGu · 2018-08-18T22:44:05Z

@SirR4T seems like the CI is failing on the new test. Could you take a look?

addaleax · 2018-08-28T20:17:43Z

Should this be backported to v10.x-staging? If yes please follow the guide and raise a backport PR, if not let me know or add the dont-land-on label.

targos · 2018-08-28T20:23:36Z

The conflict is trivial. I have a good version of this change locally. Currently compiling/running tests.

PR-URL: #22150 Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Using the same filename for different async tests could lead to race conditions. Example failure: https://travis-ci.com/nodejs/node/jobs/143351655 Refs: nodejs#22150

Using the same filename for different async tests could lead to race conditions. Example failure: https://travis-ci.com/nodejs/node/jobs/143351655 Refs: #22150 PR-URL: #22659 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

PR-URL: #22150 Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Using the same filename for different async tests could lead to race conditions. Example failure: https://travis-ci.com/nodejs/node/jobs/143351655 Refs: #22150 PR-URL: #22659 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

Notable changes: * child_process: * `TypedArray` and `DataView` values are now accepted as input by `execFileSync` and `spawnSync`. #22409 * coverage: * Native V8 code coverage information can now be output to disk by setting the environment variable `NODE_V8_COVERAGE` to a directory. #22527 * deps: * The bundled npm was upgraded to version 6.4.1. #22591 * Changelogs: [6.3.0-next.0](https://github.com/npm/cli/releases/tag/v6.3.0-next.0) [6.3.0](https://github.com/npm/cli/releases/tag/v6.3.0) [6.4.0](https://github.com/npm/cli/releases/tag/v6.4.0) [6.4.1](https://github.com/npm/cli/releases/tag/v6.4.1) * fs: * The methods `fs.read`, `fs.readSync`, `fs.write`, `fs.writeSync`, `fs.writeFile` and `fs.writeFileSync` now all accept `TypedArray` and `DataView` objects. #22150 * A new boolean option, `withFileTypes`, can be passed to to `fs.readdir` and `fs.readdirSync`. If set to true, the methods return an array of directory entries. These are objects that can be used to determine the type of each entry and filter them based on that without calling `fs.stat`. #22020 * http2: * The `http2` module is no longer experimental. #22466 * os: * Added two new methods: `os.getPriority` and `os.setPriority`, allowing to manipulate the scheduling priority of processes. #22394 * process: * Added `process.allowedNodeEnvironmentFlags`. This object can be used to programmatically validate and list flags that are allowed in the `NODE_OPTIONS` environment variable. #19335 * src: * Deprecated option variables in public C++ API. #22392 * Refactored options parsing. #22392 * vm: * Added `vm.compileFunction`, a method to create new JavaScript functions from a source body, with options similar to those of the other `vm` methods. #21571 * Added new collaborators: * [lundibundi](https://github.com/lundibundi) - Denys Otrishko PR-URL: #22716

PR-URL: #22150 Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Using the same filename for different async tests could lead to race conditions. Example failure: https://travis-ci.com/nodejs/node/jobs/143351655 Refs: #22150 PR-URL: #22659 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Michaël Zasso <targos@protonmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

Notable changes: * child_process: * `TypedArray` and `DataView` values are now accepted as input by `execFileSync` and `spawnSync`. #22409 * coverage: * Native V8 code coverage information can now be output to disk by setting the environment variable `NODE_V8_COVERAGE` to a directory. #22527 * deps: * The bundled npm was upgraded to version 6.4.1. #22591 * Changelogs: [6.3.0-next.0](https://github.com/npm/cli/releases/tag/v6.3.0-next.0) [6.3.0](https://github.com/npm/cli/releases/tag/v6.3.0) [6.4.0](https://github.com/npm/cli/releases/tag/v6.4.0) [6.4.1](https://github.com/npm/cli/releases/tag/v6.4.1) * fs: * The methods `fs.read`, `fs.readSync`, `fs.write`, `fs.writeSync`, `fs.writeFile` and `fs.writeFileSync` now all accept `TypedArray` and `DataView` objects. #22150 * A new boolean option, `withFileTypes`, can be passed to to `fs.readdir` and `fs.readdirSync`. If set to true, the methods return an array of directory entries. These are objects that can be used to determine the type of each entry and filter them based on that without calling `fs.stat`. #22020 * http2: * The `http2` module is no longer experimental. #22466 * os: * Added two new methods: `os.getPriority` and `os.setPriority`, allowing to manipulate the scheduling priority of processes. #22407 * process: * Added `process.allowedNodeEnvironmentFlags`. This object can be used to programmatically validate and list flags that are allowed in the `NODE_OPTIONS` environment variable. #19335 * src: * Deprecated option variables in public C++ API. #22515 * Refactored options parsing. #22392 * vm: * Added `vm.compileFunction`, a method to create new JavaScript functions from a source body, with options similar to those of the other `vm` methods. #21571 * Added new collaborators: * [lundibundi](https://github.com/lundibundi) - Denys Otrishko PR-URL: #22716

SimonSchick · 2018-09-07T13:06:40Z

Does this also affect appendFile and the likes?

nodejs-github-bot added the fs Issues and PRs related to the fs subsystem / file system. label Aug 6, 2018

SirR4T force-pushed the fix1826-ArrayBuffersInFs branch from a53af6e to 5204651 Compare August 6, 2018 12:04

jasnell previously approved these changes Aug 6, 2018

View reviewed changes

benjamingr approved these changes Aug 6, 2018

View reviewed changes

benjamingr previously requested changes Aug 6, 2018

View reviewed changes

SirR4T force-pushed the fix1826-ArrayBuffersInFs branch from 5204651 to ae2d513 Compare August 7, 2018 09:36

TimothyGu self-requested a review August 7, 2018 22:02

SirR4T force-pushed the fix1826-ArrayBuffersInFs branch from ae2d513 to bcaacaa Compare August 8, 2018 05:19

SirR4T commented Aug 8, 2018

View reviewed changes

SirR4T force-pushed the fix1826-ArrayBuffersInFs branch from bcaacaa to bc3b58e Compare August 8, 2018 06:04

TimothyGu reviewed Aug 10, 2018

View reviewed changes

SirR4T force-pushed the fix1826-ArrayBuffersInFs branch from 590d218 to 0b78626 Compare August 17, 2018 15:21

lundibundi reviewed Aug 18, 2018

View reviewed changes

TimothyGu approved these changes Aug 18, 2018

View reviewed changes

BeniCheni mentioned this pull request Aug 28, 2018

string_decoder : support typed array or data view #22562

Closed

4 tasks

TimothyGu mentioned this pull request Aug 28, 2018

Accept ArrayBuffer (and typed array/data view?) anywhere Buffer is allowed in the API #1826

Closed

addaleax added the backport-requested-v10.x label Aug 28, 2018

targos removed author ready PRs that have at least one approval, no pending requests for changes, and a CI started. backport-requested-v10.x labels Aug 28, 2018

addaleax mentioned this pull request Sep 2, 2018

test: fix flaky parallel/test-fs-write-file-typedarrays #22659

Closed

3 tasks

targos mentioned this pull request Sep 5, 2018

Release proposal: v10.10.0 #22716

Merged

This was referenced Sep 6, 2018

2018-09-06 Version 10.10.0 (Current) @targos beevelop/docker-android-nodejs#236

Closed

2018-09-06 Version 10.10.0 (Current) @targos beevelop/docker-nodejs#221

Closed

targos mentioned this pull request Sep 19, 2018

how to do feature detection #22585

Closed

bsiegel mentioned this pull request Nov 9, 2018

@types/fs-extra -- Updates needed for Node v10.10+ DefinitelyTyped/DefinitelyTyped#30398

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fs: work with any TypedArrays instead of only Uint8Arrays #22150

fs: work with any TypedArrays instead of only Uint8Arrays #22150

SirR4T commented Aug 6, 2018 •

edited

Loading

SirR4T commented Aug 6, 2018

targos commented Aug 6, 2018

benjamingr left a comment

benjamingr commented Aug 6, 2018

SirR4T commented Aug 7, 2018

benjamingr commented Aug 7, 2018

TimothyGu commented Aug 7, 2018

TimothyGu commented Aug 7, 2018

SirR4T left a comment

SirR4T Aug 7, 2018

SirR4T commented Aug 8, 2018 •

edited

Loading

SirR4T commented Aug 10, 2018

TimothyGu left a comment

TimothyGu Aug 10, 2018

SirR4T Aug 16, 2018

TimothyGu Aug 16, 2018

SirR4T Aug 17, 2018

TimothyGu Aug 10, 2018

TimothyGu Aug 10, 2018

TimothyGu commented Aug 16, 2018

SirR4T commented Aug 17, 2018 •

edited

Loading

TimothyGu commented Aug 17, 2018

lundibundi Aug 18, 2018

SirR4T Aug 19, 2018

lundibundi Aug 18, 2018

SirR4T Aug 19, 2018

TimothyGu left a comment

TimothyGu commented Aug 18, 2018

TimothyGu commented Aug 18, 2018

addaleax commented Aug 28, 2018

targos commented Aug 28, 2018

SimonSchick commented Sep 7, 2018

fs: work with any TypedArrays instead of only Uint8Arrays #22150

fs: work with any TypedArrays instead of only Uint8Arrays #22150

Conversation

SirR4T commented Aug 6, 2018 • edited Loading

Checklist

SirR4T commented Aug 6, 2018

targos commented Aug 6, 2018

benjamingr left a comment

Choose a reason for hiding this comment

benjamingr commented Aug 6, 2018

SirR4T commented Aug 7, 2018

benjamingr commented Aug 7, 2018

TimothyGu commented Aug 7, 2018

TimothyGu commented Aug 7, 2018

SirR4T left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SirR4T commented Aug 8, 2018 • edited Loading

SirR4T commented Aug 10, 2018

TimothyGu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TimothyGu commented Aug 16, 2018

SirR4T commented Aug 17, 2018 • edited Loading

TimothyGu commented Aug 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TimothyGu left a comment

Choose a reason for hiding this comment

TimothyGu commented Aug 18, 2018

TimothyGu commented Aug 18, 2018

addaleax commented Aug 28, 2018

targos commented Aug 28, 2018

SimonSchick commented Sep 7, 2018

SirR4T commented Aug 6, 2018 •

edited

Loading

SirR4T commented Aug 8, 2018 •

edited

Loading

SirR4T commented Aug 17, 2018 •

edited

Loading