Need to run tests in both strict mode and non-strict mode

[bterlson]

If a test doesn't indicate that it's specifically for sloppy or strict mode it should be run in both. This change should be made at least for the web harness, but also perhaps for the python harness.

[bterlson]

(@erights, just fyi)

Investigated this a bit today. Unfortunately this is not easy as many test cases are noStrict due to verifications they do. For example, ES5 15.2.3.5-4-214.js:

function testcase() {
        var descObj = {
            writable: ""
        };

        var newObj = Object.create({}, {
            prop: descObj
        });
        var hasProperty = newObj.hasOwnProperty("prop") && typeof newObj.prop === "undefined";

        newObj.prop = 121;

        return hasProperty && typeof newObj.prop === "undefined";
    }
runTestCase(testcase);

Similar cases exist when ensuring a non-extensible object can't be extended and a non-configurable property can't be deleted.

I don't have an exact count of cases affected but I can confidently say it's over 100.

A quick fix might be to tag these tests noStrict, but ideally the verification steps should be fixed to work in strict and non-strict.

[bterlson]

I think at this point we can track individual test failures as bugs. Thanks to lots of work from @anba and @smikes we should be mostly strict clean right now!

[anba]

we should be mostly strict clean right now!

I don't want to be the bearer of bad news, but there are still more than 1200 tests failing when running in strict mode. 😢

[bterlson]

@anba noooo! I guess I messed up my harness command or something :(

[anba]

Just FYI
Test failures: https://gist.github.com/anba/98fff692ea5b2e7a6358
With stacktraces: https://gist.github.com/anba/c46ce3357cbab4b92be6

[smikes]

@anba I see those too. I believe there is a race in test262-harness that causes failures when many tests are queued, because when I rerun them individually they succeed, eg

$ test262-harness test/built-ins/RegExp/*.js
[lots of failures]
FAIL test/built-ins/RegExp/S15.10.3.1_A2_T2.js
     Checking if using dafined variable "x = 1" as flags leads to throwing the correct exception
     Exp: no error
     Got: ReferenceError: __patterns is not defined
       at evalmachine.<anonymous>:13:69
       at Socket.<anonymous> (/Users/smikes/src/github/test262-harness/lib/runners/nodehost.js:33:12)
       at Socket.emit (events.js:95:17)
       at Socket.<anonymous> (_stream_readable.js:764:14)
       at Socket.emit (events.js:92:17)
       at emitReadable_ (_stream_readable.js:426:10)
       at emitReadable (_stream_readable.js:422:5)
       at readableAddChunk (_stream_readable.js:165:9)
       at Socket.Readable.push (_stream_readable.js:127:10)
       at Pipe.onread (net.js:528:21)

Ran 407 tests
373 passed
34 failed
Took 2.371 seconds
$ test262-harness test/built-ins/RegExp/S15.10.3.1_A2_T2.js 
Ran 1 tests
1 passed
0 failed
Took 0.102 seconds

[anba]

@smikes __patterns only appears in RegExp/15.10.2_A1_T1, so the file name in the harness output is definitely wrong. Also: Running 15.10.2_A1_T1 with the --testStrict option should report a failure for strict mode. That works for me when using -r jsshell. It did not work for me with -r node and -r node-ip.

Update: It's still broken with node@0.12, but it works as expected in io.js@0.12

[erights]

What's the status of this?

[anba]

What's the status of this?

The numbers from my earlier comment still apply, so ~1200 failing tests.

[bterlson]

@anba, any comments on how much further to go after your latest PRs?

[anba]

@anba, any comments on how much further to go after your latest PRs?

• ~240 files in built-ins/Object: All remaining Object tests require changes to use propertyHelper functions.
• 72 files in language: I still need to decide how to handle tests using eval. I think I'll convert eval("var x = 0;"); if (x !== 0) { ... } to var result = eval("var x = 0; x); if (result !== 0) { ... }, but I haven't yet made a final decision. (See language/line-terminators and language/white-space for examples where this is necessary.)

[smikes]

I can take on the Object tests , if that would help.

[anba]

I can take on the Object tests , if that would help.

Sure. Only a handful of test cases were changed in #249, the rest still needs some work.

[smikes]

Working on that this morning.

[bterlson]

What's left? :-D

[anba]

What's left? :-D

All tests work now in both modes! 😄

The only thing left is updating the python runner to use the new default.

diff --git a/tools/packaging/test262.py b/tools/packaging/test262.py
index b9172b0..ed8939e 100755
--- a/tools/packaging/test262.py
+++ b/tools/packaging/test262.py
@@ -66,9 +66,7 @@ def BuildOptions():
                     help="Test only strict mode")
   result.add_option("--non_strict_only", default=False, action="store_true",
                     help="Test only non-strict mode")
-  # TODO: Once enough tests are made strict compat, change the default
-  # to "both"
-  result.add_option("--unmarked_default", default="non_strict",
+  result.add_option("--unmarked_default", default="both",
                     help="default mode for tests of unspecified strictness")
   result.add_option("--logname", help="Filename to save stdout to")
   result.add_option("--junitname", help="Filename to save test results in JUnit XML format")

[smikes]

🎉

[bterlson]

Done!!! 🍰 🍨 🎈 🎉

[bterlson]

@anba, @smikes, thanks! ❤️