Integrate Test262 #654

jugglinmike · 2017-07-29T23:19:46Z

Introduce a GNU Make target for retrieving TC-39's Test262 suite and
validating parsing of the files it contains. Interpret each file as a
parser test in accordance with that project's INTERPRETING.md
document. Allow for the specification of allowed failures via a
"whitelist" file so that the test suite may help prevent regressions in
this project in situations where this project has known bugs. Initialize
the "whitelist" file with a listing of all tests that are currently
failing. Extend the continuous integration environment's configuration
to automatically run these tests.

Q	A
Bug fix?	no
Breaking change?	no
New feature?	no
Deprecations?	no
Spec compliancy?	no
Tests added/pass?	yes
Fixed tickets	gh-633
License	MIT

Hey @hzoo!

Here's my first attempt at integrating Test262. There's a lot of new code here,
so I'm prepared to make a log of changes according to review feedback.

One thing I would prefer to do is split the run_test262_utils.js file into
multiple files, possibly within a dedicated directory. There's not much
precedent for that within the scripts directory, so I thought I'd hold off on
that until I heard back from you.

I'd also like to point out that the code within the scripts directory is not
currently being linted by the npm "lint" command. I manually satisfied the
linter anyway, with one exception: trailing commas. (I'm developing in Node.js
current LTS release where trailing commas are not supported.) Is there anything
you would like to change about this?

Introduce a GNU Make target for retrieving TC-39's Test262 suite and validating parsing of the files it contains. Interpret each file as a parser test in accordance with that project's `INTERPRETING.md` document. Allow for the specification of allowed failures via a "whitelist" file so that the test suite may help prevent regressions in this project in situations where this project has known bugs. Initialize the "whitelist" file with a listing of all tests that are currently failing. Extend the continuous integration environment's configuration to automatically run these tests.

nicolo-ribaudo · 2017-07-30T10:02:59Z

Travis reported this error:

$ if [ "$JOB" = "test262-test" ]; then make test-test262; fi
node scripts/run_test262.js
{ Error: ENOENT: no such file or directory, stat '/home/travis/build/babel/babylon/build/test262/test'
    at Error (native)
  errno: -2,
  code: 'ENOENT',
  syscall: 'stat',
  path: '/home/travis/build/babel/babylon/build/test262/test' }

Is it intended?

nicolo-ribaudo · 2017-07-30T10:14:11Z

scripts/run_test262.js

+  })
+  .catch(function(err) {
+    console.error(err);
+    process.statusCode = 1;


This should be process.exitCode?

Yes. This is critical--thank you for spotting it.

nicolo-ribaudo · 2017-07-30T10:22:57Z

scripts/test262_whitelist.txt

@@ -0,0 +1,7972 @@
+# This file lists tests that are known to produce incorrect results when parsed
+# with Babyline:


nit: Babylon

nicolo-ribaudo · 2017-07-30T10:41:08Z

I see there are a lot of failing tests which babylon actually supports via plugins, for example language/expressions/async-generator/args-trailing-comma-multiple.js.
I think there should be a way to specify which plugins enable for a given file/folder.
Maybe we should use the features flag?

jugglinmike · 2017-07-30T17:03:41Z

I think there should be a way to specify which plugins enable for a given
file/folder. Maybe we should use the features flag?

I'd be open to try that, but at some point, I think we should consider
introducing a YAML parser. The ad-hoc regular expressions I've written seem
"close enough" for the current use case, but using RegExps to interpret dynamic
values like that is much more error prone. I've been trying to avoid adding any
dependencies for this patch, so I'm wondering how the project maintainers would
feel about a new "devDependency" on a YAML parser.

jugglinmike · 2017-07-30T17:44:19Z

Alternatively, we could unconditionally enable all plugins for all tests. This is certainly less precision, but since Test262 generally doesn't include any "future hostile" tests, it may be an acceptable compromise.

hzoo · 2017-08-01T17:24:33Z

We just did something like this for our printer tests (generator) babel/babel#6018 with explicit flags. We could do unconditional for all tests though for now, if someone runs the stage 0 preset, it would be the same thing and none of the plugins should be incompatible.

hzoo · 2017-08-01T17:25:28Z

scripts/run_test262_utils.js

+  const sourceType = test.isModule ? "module" : "script";
+
+  try {
+    parse(test.content, { sourceType: sourceType });


Just need to add the ones in https://github.com/babel/babylon#plugins (excluding flow/jsx/typescript/estree)

hzoo · 2017-08-01T17:27:06Z

is there already a script/way to remove the whitelist tests automatically if things are fixed later?

Jessidhia · 2017-08-02T01:27:57Z

scripts/run_test262.js

+const whitelistFile = path.join(__dirname, "test262_whitelist.txt");
+
+Promise.all([utils.getTests(testDir), utils.getWhitelist(whitelistFile)])
+  .then(function([tests, whitelist]) {


This breaks in Node 4 (do we care?)

I think it's ok to only run on latest for these?

Jessidhia · 2017-08-02T01:29:09Z

scripts/run_test262.js

+    const badnews = [];
+    const badnewsDetails = [];
+
+    [


Lines starting with a [ are pretty scary, regardless of semicolons.

Maybe save this into a const, or make it the right side of a for of?

For consistency, I'd prefer to keep a functional style. But I see your point. I've made this the right-hand side of a void expression--that will guard against ASI-related bugs, and it should also more clearly indicate to the reader that this literal value is not used beyond this statement.

Jessidhia · 2017-08-02T01:30:11Z

scripts/run_test262.js

+      badnews.push(desc);
+      badnewsDetails.push(desc + ":");
+      badnewsDetails.push(
+        ...tests.map(function(test) {


This also breaks in Node 4 😄

Jessidhia · 2017-08-02T01:32:36Z

scripts/run_test262_utils.js

+
+function readDir(dirName) {
+  return new Promise(function(resolve, reject) {
+    fs.readdir(dirName, function(err, contents) {


Any objections to adding a devDependency on util.promisify?

It's available natively in Node >= 8, the package just shims it if it's missing.

I wouldn't object, but I believe this question was @hzoo, so I'm going to hold off unless he agrees.

I don't think we need to run on anything other than latest node (8)? We can add it too if we really want to but I think it's fine to run one one version of node like we do for coverage and lint

Works for me!

Jessidhia · 2017-08-02T01:35:17Z

scripts/run_test262_utils.js

+    test.actualError = true;
+  }
+
+  test.result = test.expectedError ^ test.actualError ? "fail" : "pass";


Better to use !==.

Jessidhia · 2017-08-02T01:37:23Z

scripts/run_test262_utils.js

+        return line.replace(/#.*$/, "").trim();
+      })
+      .filter(function(line) {
+        return line.length > 0;


.filter(Boolean) would work too but I guess this is clearer

jugglinmike · 2017-08-05T19:42:25Z

is there already a script/way to remove the whitelist tests automatically if
things are fixed later?

There was not, but that's a good idea. Maintaining the white list in JSHint has
been a pain point for me. I've updated the patch to include a command-line
flag and dedicated Makefile target.

So I think that's everything--this should be ready for another round of review.

jugglinmike · 2017-08-06T02:27:47Z

I've updated the job to run in the latest release of Node.js, and I've taken @Kovensky's advice regarding the "util.promisify" module. My thinking is that many contributors likely prefer to develop in the LTS release. Even though they may not all care to run this particular test suite, the module's dependency tree is light enough to be justify the convenience.

The babel-test job is failing. Is there something we should do about that in this patch?

hzoo · 2017-08-06T02:32:20Z

There was not, but that's a good idea.

Like it would basically run the whitelist tests and if they pass correctly instead of erroring/not erroring it could tell you to remove them (or it is done automatically), similar to updating a snapshot/fixture

hzoo

lgtm, thanks for this work! Will help us a lot for regressions in the future! And hopefully get us started on the path for test262 in the transforms as well actually; ideally we can further more make compat-table 2.0 with test262

hzoo · 2017-08-06T02:36:19Z

oops I just merged babel/babel#6056, which means we need to update yarn at least?

either way unrelated to your pr

jugglinmike · 2017-08-06T03:04:57Z

Whoops; before you updated the comment, I thought you were saying I should update the yarn lock file. I guess wasn't necessary, after all (I'm not familiar with that tool). Should I revert this latest commit?

hzoo · 2017-08-06T03:06:45Z

I updated Babel to use a new feature in yarn; I think we have to add https://github.com/babel/babel/pull/6056/files#diff-354f30a63fb0907d4ad57269548329e3L15 since the yarn version it's using now is older. Not sure it's going to fix but it's an issue due to us targeting master. Either way it would of broke in someone else's PR

hzoo · 2017-08-06T03:20:56Z

Yeah we don't need that commit, I can try this locally and push a change

danez

Nice work. It's crazy how much tests are not working with babylon.

danez · 2017-08-06T11:42:17Z

I just tested this also locally and it doesn't work and stops with EMFILE: too many open files. Seems reading all the tests async without limit is not a good idea, but works on travis for some reason.

nicolo-ribaudo · 2017-08-06T11:55:40Z

We could use graceful-fs, which handles EMFILE errors automatically.

hzoo · 2017-08-06T14:10:03Z

Yeah graceful-fs sounds good

existentialism

Awesome @jugglinmike!

Note, I bumped our yarn version to support workspaces and switched to graceful-fs.

After this lands, we can drop the rest-parameter test262 test fixtures I added a while back

hzoo · 2017-08-07T01:19:46Z

Thanks for starting this @jugglinmike!

hzoo · 2017-08-07T01:23:31Z

Makefile

+
+bootstrap-test262: clean
+	mkdir ./build
+	git clone https://github.com/tc39/test262.git ./build/test262


could we do a depth=x here as well?

hzoo · 2017-08-07T01:27:00Z

chicoxyzzy

<3

jugglinmike · 2017-08-07T23:50:13Z

It's been my pleasure :)

jugglinmike mentioned this pull request Jul 29, 2017

Incorporating Test262 #633

Closed

nicolo-ribaudo reviewed Jul 30, 2017

View reviewed changes

jugglinmike added 3 commits July 30, 2017 12:28

fixup! Integrate Test262

110ace2

fixup! Integrate Test262

496029a

fixup! Integrate Test262

8b0018d

hzoo added Priority: High Tag: Internal labels Aug 1, 2017

hzoo mentioned this pull request Aug 1, 2017

Create aug-02.md babel/notes#29

Merged

hzoo reviewed Aug 1, 2017

View reviewed changes

Jessidhia reviewed Aug 2, 2017

View reviewed changes

jugglinmike added 3 commits August 5, 2017 15:23

fixup! Integrate Test262

2957990

fixup! Integrate Test262

03bfc28

fixup! Integrate Test262

48486ab

fixup! Integrate Test262

10aadef

hzoo approved these changes Aug 6, 2017

View reviewed changes

fixup! Integrate Test262

81b210b

danez approved these changes Aug 6, 2017

View reviewed changes

existentialism added 2 commits August 6, 2017 09:50

use graceful-fs and latest yarn on travis

78b3465

Merge branch 'master' into test262

908fa7b

existentialism approved these changes Aug 6, 2017

View reviewed changes

hzoo merged commit 0466504 into babel:master Aug 7, 2017

hzoo reviewed Aug 7, 2017

View reviewed changes

chicoxyzzy reviewed Aug 7, 2017

View reviewed changes

hzoo mentioned this pull request Aug 10, 2017

Run tc39/test262 against Babel babel/babel#4987

Closed

nicolo-ribaudo mentioned this pull request Aug 19, 2017

Fix flow test runner #680

Merged

jugglinmike mentioned this pull request Sep 9, 2017

testStream(): A Node.js API for visiting "compiled" tests tc39/test262-harness#92

Closed

This was referenced Nov 13, 2017

Integrate Test262 acornjs/acorn#622

Closed

Integrate Test262 jquery/esprima#1880

Closed

chicoxyzzy mentioned this pull request Dec 30, 2017

test262 conformance as an line item the compat table compat-table/compat-table#830

Open

littledan mentioned this pull request Aug 6, 2018

[WIP] Early Errors > Grammar #1343 tc39/test262#1651

Closed

6 tasks

		@@ -0,0 +1,7972 @@
		# This file lists tests that are known to produce incorrect results when parsed
		# with Babyline:

Integrate Test262 #654

Integrate Test262 #654

Conversation

jugglinmike commented Jul 29, 2017

nicolo-ribaudo commented Jul 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolo-ribaudo commented Jul 30, 2017 • edited Loading

jugglinmike commented Jul 30, 2017

jugglinmike commented Jul 30, 2017

hzoo commented Aug 1, 2017

hzoo Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

hzoo commented Aug 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jugglinmike commented Aug 5, 2017

jugglinmike commented Aug 6, 2017

hzoo commented Aug 6, 2017

hzoo left a comment

Choose a reason for hiding this comment

hzoo commented Aug 6, 2017 • edited Loading

jugglinmike commented Aug 6, 2017

hzoo commented Aug 6, 2017

hzoo commented Aug 6, 2017

danez left a comment

Choose a reason for hiding this comment

danez commented Aug 6, 2017

nicolo-ribaudo commented Aug 6, 2017

hzoo commented Aug 6, 2017

existentialism left a comment • edited Loading

Choose a reason for hiding this comment

hzoo commented Aug 7, 2017

Choose a reason for hiding this comment

hzoo commented Aug 7, 2017

chicoxyzzy left a comment

Choose a reason for hiding this comment

jugglinmike commented Aug 7, 2017

nicolo-ribaudo commented Jul 30, 2017 •

edited

Loading

hzoo Aug 1, 2017 •

edited

Loading

hzoo commented Aug 6, 2017 •

edited

Loading

existentialism left a comment •

edited

Loading