Test streamlining/browser support for W3C and other tests

[brettz9]

Although we have a recent addition of W3C-adapted tests for Node, I'd like us to:

• Node tests
  • Use an automated process for converting the original browser-based W3C tests to Node tests
  • The latest version of ws is breaking our worker code; ought to investigate rather than just
    keeping the last working version hard-coded
• In-browser tests
  • Convert the other fake, IndexedDBMock, and old W3C tests we've added to work for the browser
  • Directly incorporate the original browser-based W3C tests for direct browser testing (as individual files)
  • Get all browser tests to be able to launch together from within the browser and get W3C test server working (if nothing else than for WebIDL tests; our tests are now copying the source of the WebIDLParser.js file directly, so no need to build)
    • Modifying the beginning of FileHandler get_data in web-platform-tests/tools/wptserve/wptserve/handlers.py to the snippet below works but waiting on Code injection config web-platform-tests/wpt#5133 . Alternatively, there are the following changes (they have the disadvantage of adding .any.js files as per our npm run w3c-add-wrap routine and duplicating work in runner but the runner has the advantage of adding polyfill links alongside non-polyfill ones and was the recommended approach): Code injection config web-platform-tests/wpt#5133 We can instead use npm run wrap even though it unfortunately messes up the web-platform-tests/IndexedDB directory, requiring a reset of those files.
  • Although there are Grunt tasks for auto-deleting db files, the tests themselves should really do the deletions on clean-up, ensuring proper browser testing (and less dead weight)
  • Move "Old W3C" tests to own repository just in case ever need again, but as new W3C is set up for Node and browser now, we shouldn't need them anymore
  • Get interfaces.worker.js and any other worker or** iframe tests (opaque tests) using polyfill in browser tests
  • Get new W3C tests working on rawgit.com, ideally optionally with a runner that can work statically (without Python).
• Headless browsers
  • Directly incorporate the original browser-based W3C tests for (Nightmare) headless browser testing
  • Add Nightmare headless testing of fake/indexeddbmock/mocha/qunit (already have Node and browser)
• Actual testing
  • Grab the low-hanging fruit in getting failing tests to pass (Need to see why Chrome and Safari are failing where the Node implementation is not)
  • Report any new issues that cannot be readily solved

Three long-term goals (which are not milestone 3 but which I'll track here for now):

• For our pre-existing tests, maybe also streamlining to choose either Mocha in place of QUnit, unless there is some compelling advantage for QUnit tests.
• We might also try to return our tests to the W3C tests unless they are specific to our implementation only.
• See the todo markers in code for minor lower priority issues not being tracked

• Snippet

        if byte_ranges is None:
            if (path.find('IndexedDB') > -1 and path.find('.htm') > -1):
                shim = """
                <script src='http://localhost:9999/dist/indexeddbshim-UnicodeIdentifiers.js'></script>
                <script>
                    shimIndexedDB.__setConfig({
                        fullIDLSupport: true,
                        replaceNonIDBGlobals: true
                    });
                    shimIndexedDB.__useShim();
                </script>
                """
                fc = open(path, 'rb').read()
                return fc.replace("<script", shim + "<script", 1)
            else:
                return open(path, 'rb')

** Could add the following to workers in theory (and in Safari 10, in the workers only, the old bug of needing to define indexedDB in a closure as an alias to shimIndexedDB was also present):

importScripts('http://localhost:9999/dist/indexeddbshim-noninvasive.js');
setGlobalVars(null, {
    fullIDLSupport: true,
    replaceNonIDBGlobals: true
});
shimIndexedDB.__setConnectionQueueOrigin();
shimIndexedDB.__useShim();

However, Safari apparently removed its support of IndexedDB from workers (as did Chrome) as per https://bugs.chromium.org/p/chromium/issues/detail?id=434740

[brettz9]

Note that it is now possible to set up a robust testing environment for the latest W3C tests (along with our own good number of tests), both for Node and the browser.

The W3C test runner for the browser still has to be patched (I've submitted a couple PRs as per web-platform-tests/wpt#5133 (comment) ), but if you get the web-platform-tests test environment working (a submodule of this repo), the tweaks are simple enough to do by hand pending any incorporation of the PRs.

We could really use help, with WebSQL-supporting browsers (modern Chrome (57) and Safari 10) attempting to fix the issues they have in failing those tests that are working in Node.js (the ones still failing in our Node implementation are pretty much unresolvable, particularly in the browser). Chrome would probably be best to start with as it is easier to test and it has fewer failures that are not present in Node.

The test runner is not as useful in Safari which chokes on even native testing, so in that case, it will be necessary to test the file one-by-one or by regex (adding .any.html to the file names as discussed in the above-mentioned web-platform-tests issue).

As far as iOS9, I intend to post shortly to #277 in this regard.

[brettz9]

Here's the code I currently have for an alternative testharness runner, which I've placed at testharness_runner_polyfill.html at the root of web-platform-tests along with the above-mentioned patches:

<!DOCTYPE html>
<meta charset="utf-8" />
<link rel="shortcut icon" href="data:image/x-icon;," type="image/x-icon" />

<script src="http://localhost:9999/dist/indexeddbshim-noninvasive.js"></script>
<iframe id="ifr" style="width:100%; height: 680px;" src="http://web-platform.test:8000/tools/runner/index.html"></iframe>
<script>
var win = document.querySelector('#ifr').contentWindow;
win.addEventListener('load', function () {
    var oldLink = win.VisualOutput.prototype.link;
    win.VisualOutput.prototype.link = function(href) {
        var link = oldLink.apply(this, arguments);
        link.target = '_blank';
        var linkPolyfill = link.cloneNode(true);
        var suffix = '.any.html';
        if (!(/\.any\.html$/).test(href)) {
            linkPolyfill.href += suffix;
            linkPolyfill.textContent = 'polyfill';
        }
        var span = document.createElement('span');
        span.appendChild(link);
        span.appendChild(document.createTextNode(' ('));
        span.appendChild(linkPolyfill);
        span.appendChild(document.createTextNode(')'));
        return span;
    };
    win.testharness_properties = Object.assign({}, win.testharness_properties, {
        callback: function (win) {
            setGlobalVars(win, {
                fullIDLSupport: true,
                replaceNonIDBGlobals: true
            });
            win.shimIndexedDB.__useShim();
            // Object.defineProperty(win.IDBTransaction, 'READ_WRITE', {configurable: true, value: 55555}); // Use with `/IndexedDB/historical.html` to confirm we're injecting
        }
    });
});
</script>

Also adding this to web-platform-tests' .gitignore is helpful:

# Polyfill-related additions
IndexedDB/*.any.js
IndexedDB/_indexeddbshim-loader.html
testharness_runner_polyfill.html

[brettz9]

Nevermind the above instructions--just follow the instructions in docs/TESTING.md

[brettz9]

Tracking this issue for new package at #15

[axemclion]

@brettz9 - should I archive this repo, so that we can move to the new location ?

[brettz9]

See my comment at #353 (comment) . Thanks!