The Python requirements for running Firefox are broken

[lddubeau]

What I describe below happens with master at a0b8550

About an hour ago I cloned WPT for the first time and tried to run the tests with my local Firefox installation. After performing the necessary setup, I immediately ran into a problem while running ./wpt run firefox dom/nodes/Node-normalize.html I got this error:

 0:39.20 CRITICAL Traceback (most recent call last):
  File "/home/ldd/src/git-repos/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 132, in start_runner
    executor = executor_cls(browser, **executor_kwargs)
  File "/home/ldd/src/git-repos/web-platform-tests/tools/wptrunner/wptrunner/executors/executormarionette.py", line 496, in __init__
    self.protocol = MarionetteProtocol(self, browser, capabilities, timeout_multiplier)
  File "/home/ldd/src/git-repos/web-platform-tests/tools/wptrunner/wptrunner/executors/executormarionette.py", line 343, in __init__
    do_delayed_imports()
  File "/home/ldd/src/git-repos/web-platform-tests/tools/wptrunner/wptrunner/executors/executormarionette.py", line 50, in do_delayed_imports
    from marionette_driver import marionette, errors
  File "/home/ldd/src/git-repos/web-platform-tests/_venv/lib/python2.7/site-packages/marionette_driver/__init__.py", line 9, in <module>
    from marionette_driver import (
  File "/home/ldd/src/git-repos/web-platform-tests/_venv/lib/python2.7/site-packages/marionette_driver/expected.py", line 10, in <module>
    from .marionette import HTMLElement
  File "/home/ldd/src/git-repos/web-platform-tests/_venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 24, in <module>
    from .geckoinstance import GeckoInstance
  File "/home/ldd/src/git-repos/web-platform-tests/_venv/lib/python2.7/site-packages/marionette_driver/geckoinstance.py", line 17, in <module>
    from mozdevice import DMError
ImportError: cannot import name DMError

DMError is indeed not provided by mozdevice 1.0.0.

I was able to solve it by editing tools/wptrunner/requirements_firefox.txt to add:

mozdevice == 0.52

If I do this, I get a series of warnings of this sort:

mozrunner 7.0.1 has requirement mozdevice>=1.*, but you'll have mozdevice 0.52 which is incompatible.

But I am able to run the suite. So the above works for me as a workaround, but I doubt that it is the correct solution. I don't know enough about how the various pieces interrelate to readily figure what needs to be updated here, or even if an issue should be filed elsewhere.

[jugglinmike]

Confirmed. This has also interrupted results collection for https://wpt.fyi, but only the most recent builds (i.e. those scheduled for 8:00 EST). Now bisecting...

[jugglinmike]

The regression was introduced in commit 8b9f853:

commit 8b9f853fd06ca887b340e77e342d323c32635a13
Author: pyup-bot <github-bot@pyup.io>
Date:   Mon Apr 30 21:55:58 2018 +0100

    Update mozrunner from 7.0.0 to 7.0.1

diff --git a/tools/wptrunner/requirements_firefox.txt b/tools/wptrunner/requirements_firefox.txt
index 0eb26e7..e9c79d0 100644
--- a/tools/wptrunner/requirements_firefox.txt
+++ b/tools/wptrunner/requirements_firefox.txt
@@ -2,7 +2,7 @@ marionette_driver==2.6.0
 mozprofile==1.1.0
 mozprocess == 0.26
 mozcrash == 1.0
-mozrunner == 7.0.0
+mozrunner==7.0.1
 mozleak == 0.1
 mozinstall == 1.15
 mozdownload == 1.23

That was merged via gh-10721, which unfortunately has no information about why this change was made or who made it.

@jgraham or @gsnedders may be able to say more since their handles are the only non-bot names on that pull request. Then again, various automation tools perform actions (e.g. adding labels) using human GitHub handles, so that's not conclusive. I'll keep digging!

[jugglinmike]

I haven't been able to find any relevant bugs filed againsst mozrunner, and the project's page on Pypy doesn't include a change log.

I did find the commit to gecko-dev that introduced the change, though, which lead me to a BugZilla ticket with more information (via the ticket that created the 7.0.1 release. I'm not sure why those wasn't turned up by the BugZilla search. Anyway:

Releasing mozdevice 1.0.0 breaks consumers of mozrunner 6.15 which was
released with mozdevice >= 0.37 on dep on pypi.

IDK if there's really a way around that, but since mozrunner 7.0.0 is not yet
released on pypi, the only fix is to specify the mozdevice version as < 1 in
apps that depend on mozrunner 6.15.

Which isn't a huge deal, but it's probably better if you release 7.0.0. So,
please release mozrunner 7.0.0. Or whoever has the ability to do that should.

and:

Actually, mozrunner 7.0.0 is on pypi.

Unfortunately, I updated the mozrunner deps on mozrunner 7.0.0 without
bumping the mozrunner version and re-releasing. I think that is the issue:
I'll bump to 7.1.0 (or 7.0.1? I'm not sure how to interpret this in semver
terms) and release to pypi.

Sorry for the disruption.

This suggests that it is safe to revert the change. However, it also seems to say that version 7.0.0 is invalid, but this project updated to that release on Friday without any problems. This casts some doubt about whether reverting this morning's commit is the correct solution. I'll update here when I know more.

[jugglinmike]

It appears that we are able to use mozrunner at version 7.0.0 because there are no references to mozdevice throughout all of WPT:

$ git log -n1
commit a77f2be0145f9a1fceeab3cdf11071095d84704b
Merge: 6b65b4b 757f74c
Author: Servo WPT Sync <32481905+servo-wpt-sync@users.noreply.github.com>
Date:   Tue May 1 10:26:14 2018 -0400

    Merge pull request #10733 from w3c/sync_757f74c1e0fd7d59d4ca0503a767b74839205a33

    Merge pull request #10733 from sync_757f74c1e0fd7d59d4ca0503a767b74839205a33
$ git grep mozdevice . | wc -l
0

...and because the dependencies of mozdevice module did not change between version 1.0 and the previous release.

I think this proves that mozrunner at version 7.0.0 is suitable for our purposes, and we can safely revert the patch.

[jugglinmike]

Looks like @gsnedders beat me to the punch!