tools/wpt testsuite failures on Windows

[gsnedders]

Excluding the tests mentioned in #12934, running this gets the following:

(wpt-env) C:\Users\geoff\Documents\projects\wpt\tools\wpt [master ≡]> tox -e py27 -- --no-cov -vv -k 'not test_tests_aff
ected'
py27 installed: asn1crypto==0.24.0,atomicwrites==1.2.1,attrs==18.2.0,blessings==1.7,certifi==2018.8.24,cffi==1.11.5,chardet==3.0.4,colorama==0.3.9,coverage==4.5.1,cryptography==2.3.1,enum34==1.1.6,funcsigs==1.0.2,html5lib==1.0.1,hypothesis==3.71.3,idna==2.7,ipaddress==1.0.22,marionette-driver==2.6.0,mock==2.0.0,mohawk==0.3.4,more-itertools==4.3.0,mozcrash==1.0,mozdebug==0.1,mozdevice==0.52,mozdownload==1.24,mozfile==1.2,mozinfo==0.10,mozInstall==1.16.0,mozleak==0.1,mozlog==3.8,moznetwork==0.27,mozprocess==0.26,mozprofile==1.1.0,mozrunner==7.0.0,mozterm==1.0.0,mozversion==1.5,pathlib2==2.3.2,pbr==4.2.0,pluggy==0.7.1,progressbar2==3.38.0,py==1.6.0,pycparser==2.18,pyOpenSSL==18.0.0,pytest==3.8.0,pytest-cov==2.6.0,python-utils==2.3.0,redo==1.6,requests==2.19.1,requests-hawk==1.0.0,scandir==1.9.0,selenium==3.14.0,six==1.11.0,treeherder-client==4.0.0,urllib3==1.22,webencodings==0.5.1
py27 runtests: PYTHONHASHSEED='112'
py27 runtests: commands[0] | pytest --cov --no-cov -vv -k 'not test_tests_affected'
============================= test session starts =============================
platform win32 -- Python 2.7.15, pytest-3.8.0, py-1.6.0, pluggy-0.7.1 -- c:\users\geoff\documents\projects\wpt\tools\wpt\.tox\py27\scripts\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('C:\\Users\\geoff\\Documents\\projects\\wpt\\tools\\wpt\\.hypothesis\\examples')
rootdir: C:\Users\geoff\Documents\projects\wpt\tools, inifile: pytest.ini
plugins: cov-2.6.0, mozlog-3.8, hypothesis-3.71.3
collected 22 items / 2 deselected

tests\test_run.py::test_check_environ_fail[Windows] PASSED               [  5%]
tests\test_run.py::test_check_environ_fail[Linux] PASSED                 [ 10%]
tests\test_run.py::test_check_environ_fail[Darwin] PASSED                [ 15%]
tests\test_wpt.py::test_missing PASSED                                   [ 20%]
tests\test_wpt.py::test_help PASSED                                      [ 25%]
tests\test_wpt.py::test_list_tests FAILED                                [ 30%]
tests\test_wpt.py::test_list_tests_missing_manifest FAILED               [ 35%]
tests\test_wpt.py::test_list_tests_invalid_manifest FAILED               [ 40%]
tests\test_wpt.py::test_run_firefox xfail                                [ 45%]
tests\test_wpt.py::test_run_chrome xfail                                 [ 50%]
tests\test_wpt.py::test_run_zero_tests xfail                             [ 55%]
tests\test_wpt.py::test_run_failing_test xfail                           [ 60%]
tests\test_wpt.py::test_run_verify_unstable xfail                        [ 65%]
tests\test_wpt.py::test_install_chromedriver xfail                       [ 70%]
tests\test_wpt.py::test_install_firefox xfail                            [ 75%]
tests\test_wpt.py::test_files_changed xfail                              [ 80%]
tests\test_wpt.py::test_files_changed_null xfail                         [ 85%]
tests\test_wpt.py::test_files_changed_ignore PASSED                      [ 90%]
tests\test_wpt.py::test_files_changed_ignore_rules PASSED                [ 95%]
tests\test_wpt.py::test_serve xfail                                      [100%]

================================== FAILURES ===================================
_______________________________ test_list_tests _______________________________

manifest_dir = 'c:\\users\\geoff\\appdata\\local\\temp\\tmpgakg1p'

    @pytest.mark.slow
    def test_list_tests(manifest_dir):
        """The `--list-tests` option should not produce an error under normal
        conditions."""

        with pytest.raises(SystemExit) as excinfo:
            wpt.main(argv=["run", "--metadata", manifest_dir, "--list-tests",
>                          "--yes", "chrome", "/dom/nodes/Element-tagName.html"])

tests\test_wpt.py:101:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
wpt.py:108: in main
    venv = setup_virtualenv(main_args.venv, props)
wpt.py:86: in setup_virtualenv
    venv.install(name)
virtualenv.py:50: in install
    call(self.pip_path, "install", *requirements)
utils.py:44: in call
    return subprocess.check_output(args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (('C:\\Users\\geoff\\Documents\\projects\\wpt\\_venv\\Scripts\\pip.exe', 'install', 'requests'),)
kwargs = {}, process = <subprocess.Popen object at 0x041BD110>, output = ''
unused_err = None, retcode = -1073741515
cmd = ('C:\\Users\\geoff\\Documents\\projects\\wpt\\_venv\\Scripts\\pip.exe', 'install', 'requests')

    def check_output(*popenargs, **kwargs):
        r"""Run command with arguments and return its output as a byte string.

        If the exit code was non-zero it raises a CalledProcessError.  The
        CalledProcessError object will have the return code in the returncode
        attribute and output in the output attribute.

        The arguments are the same as for the Popen constructor.  Example:

        >>> check_output(["ls", "-l", "/dev/null"])
        'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

        The stdout argument is not allowed as it is used internally.
        To capture standard error in the result, use stderr=STDOUT.

        >>> check_output(["/bin/sh", "-c",
        ...               "ls -l non_existent_file ; exit 0"],
        ...              stderr=STDOUT)
        'ls: non_existent_file: No such file or directory\n'
        """
        if 'stdout' in kwargs:
            raise ValueError('stdout argument not allowed, it will be overridden.')
        process = Popen(stdout=PIPE, *popenargs, **kwargs)
        output, unused_err = process.communicate()
        retcode = process.poll()
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd, output=output)
E           CalledProcessError: Command '('C:\\Users\\geoff\\Documents\\projects\\wpt\\_venv\\Scripts\\pip.exe', 'install', u'requests')' returned non-zero exit status -1073741515

c:\python27\Lib\subprocess.py:223: CalledProcessError
------------------------------ Captured log call ------------------------------
utils.py                    47 CRITICAL ('C:\\Users\\geoff\\Documents\\projects\\wpt\\_venv\\Scripts\\pip.exe', 'install', u'requests') exited with return code -1073741515
utils.py                    48 CRITICAL
______________________ test_list_tests_missing_manifest _______________________

manifest_dir = 'c:\\users\\geoff\\appdata\\local\\temp\\tmpgakg1p'

    @pytest.mark.slow
    def test_list_tests_missing_manifest(manifest_dir):
        """The `--list-tests` option should not produce an error in the absence of
        a test manifest file."""

        os.remove(os.path.join(manifest_dir, "MANIFEST.json"))

        with pytest.raises(SystemExit) as excinfo:
            wpt.main(argv=["run",
                           # This test triggers the creation of a new manifest
                           # file which is not necessary to ensure successful
                           # process completion. Specifying the current directory
                           # as the tests source via the --tests` option
                           # drastically reduces the time to execute the test.
                           "--tests", here,
                           "--metadata", manifest_dir,
                           "--list-tests",
                           "--yes",
>                          "firefox", "/dom/nodes/Element-tagName.html"])

tests\test_wpt.py:123:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
wpt.py:108: in main
    venv = setup_virtualenv(main_args.venv, props)
wpt.py:86: in setup_virtualenv
    venv.install(name)
virtualenv.py:50: in install
    call(self.pip_path, "install", *requirements)
utils.py:44: in call
    return subprocess.check_output(args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (('C:\\Users\\geoff\\Documents\\projects\\wpt\\_venv\\Scripts\\pip.exe', 'install', 'requests'),)
kwargs = {}, process = <subprocess.Popen object at 0x03531E70>, output = ''
unused_err = None, retcode = -1073741515
cmd = ('C:\\Users\\geoff\\Documents\\projects\\wpt\\_venv\\Scripts\\pip.exe', 'install', 'requests')

    def check_output(*popenargs, **kwargs):
        r"""Run command with arguments and return its output as a byte string.

        If the exit code was non-zero it raises a CalledProcessError.  The
        CalledProcessError object will have the return code in the returncode
        attribute and output in the output attribute.

        The arguments are the same as for the Popen constructor.  Example:

        >>> check_output(["ls", "-l", "/dev/null"])
        'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

        The stdout argument is not allowed as it is used internally.
        To capture standard error in the result, use stderr=STDOUT.

        >>> check_output(["/bin/sh", "-c",
        ...               "ls -l non_existent_file ; exit 0"],
        ...              stderr=STDOUT)
        'ls: non_existent_file: No such file or directory\n'
        """
        if 'stdout' in kwargs:
            raise ValueError('stdout argument not allowed, it will be overridden.')
        process = Popen(stdout=PIPE, *popenargs, **kwargs)
        output, unused_err = process.communicate()
        retcode = process.poll()
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd, output=output)
E           CalledProcessError: Command '('C:\\Users\\geoff\\Documents\\projects\\wpt\\_venv\\Scripts\\pip.exe', 'install', u'requests')' returned non-zero exit status -1073741515

c:\python27\Lib\subprocess.py:223: CalledProcessError
------------------------------ Captured log call ------------------------------
utils.py                    47 CRITICAL ('C:\\Users\\geoff\\Documents\\projects\\wpt\\_venv\\Scripts\\pip.exe', 'install', u'requests') exited with return code -1073741515
utils.py                    48 CRITICAL
______________________ test_list_tests_invalid_manifest _______________________

manifest_dir = 'c:\\users\\geoff\\appdata\\local\\temp\\tmpgakg1p'

    @pytest.mark.slow
    def test_list_tests_invalid_manifest(manifest_dir):
        """The `--list-tests` option should not produce an error in the presence of
        a malformed test manifest file."""

        manifest_filename = os.path.join(manifest_dir, "MANIFEST.json")

>       assert os.path.isfile(manifest_filename)
E       AssertionError: assert False
E        +  where False = <function isfile at 0x02804EF0>('c:\\users\\geoff\\appdata\\local\\temp\\tmpgakg1p\\MANIFEST.json')
E        +    where <function isfile at 0x02804EF0> = <module 'ntpath' from 'c:\users\geoff\documents\projects\wpt\tools\wpt\.tox\py27\lib\ntpath.pyc'>.isfile
E        +      where <module 'ntpath' from 'c:\users\geoff\documents\projects\wpt\tools\wpt\.tox\py27\lib\ntpath.pyc'> = os.path

tests\test_wpt.py:135: AssertionError
WARNING: Coverage disabled via --no-cov switch!
============================== warnings summary ===============================
C:\Users\geoff\Documents\projects\wpt\tools\wpt\tests\test_run.py:18: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  assert "wpt make-hosts-file" in excinfo.value.message

C:\Users\geoff\Documents\projects\wpt\tools\wpt\tests\test_run.py:18: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  assert "wpt make-hosts-file" in excinfo.value.message

C:\Users\geoff\Documents\projects\wpt\tools\wpt\tests\test_run.py:18: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  assert "wpt make-hosts-file" in excinfo.value.message

  Coverage disabled via --no-cov switch!

unknown file:0: RemovedInPytest4Warning: config.warn has been deprecated, use warnings.warn instead

-- Docs: https://docs.pytest.org/en/latest/warnings.html
== 3 failed, 7 passed, 2 deselected, 10 xfailed, 5 warnings in 14.72 seconds ==
ERROR: InvocationError for command 'C:\\Users\\geoff\\Documents\\projects\\wpt\\tools\\wpt\\.tox\\py27\\Scripts\\pytest.EXE --cov --no-cov -vv -k not test_tests_affected' (exited with code 1)
___________________________________ summary ___________________________________
ERROR:   py27: commands failed
(wpt-env) C:\Users\geoff\Documents\projects\wpt\tools\wpt [master ≡]>

[foolip]

I think this is blocking #13342.

[mdittmer]

friendly ping. @foolip is this still blocking other work? @gsnedders is this still on your radar?

[foolip]

@gsnedders and I discussed this week. Conclusion was to mark current failures as expected and only proactively fix the ones we know result in differences in the actual test results.

@gsnedders should we just close this, or does an umbrella bug serve a purpose?

[gsnedders]

@foolip I'd rather have something open to track the XFAILs and to point at from them.