Add the test suite

[jwilk]

This is the test suite I've sent you in private, and which Debian has been carrying for a while (hi @spwhitton).

I don't expect you to merge it right now. Unfortunately, it doesn't work at the moment:

• xdotool, which the test suite uses to interact with Firefox, is broken.
• The Firefox UI for certificate errors has been just redesigned in Firefox 44 (passport dude is gone!) and I haven't updated the code to take this into account.

Do we want this to be a part of the xpi file? If yes, then I should probably move it to src/.

For the moment the test requires that the extension is installed system-wide. It would be cool if it was possible to also test against the unpacked source. I haven't figured out how to do it yet.

[jwilk]

I've fixed support for Firefox 44.

[spwhitton]

Thanks for a nice summary of why the test suite is failing, jwilk. I hope xdotool gets fixed soon.

[jwilk]

I've implemented work-around for the xdotool bug.
Wheee!

[jwilk]

I forogot to mention that now I'm getting these warnings from xdotool:

XGetWindowProperty[_NET_WM_DESKTOP] failed (code=1)

They seem harmless, but I'll investigate further.

[spwhitton]

Hello @jwilk. I'm getting this error when I try to run the test suite:

[1] untrusted connection ... ok
[1] open "add security exception" dialog ... Your windowmanager claims not to support _NET_ACTIVE_WINDOW, so the attempt to activate the window was aborted.

According to this thread this is because xvfb's built-in window manager doesn't support EWMHs. Fair enough, but in that case how did you ever make the test suite run? I think I must be missing something.

[spwhitton]

Ah, that problem was caused by your xdotool workaround which I needed to drop. Unfortunately, the test suite is now timing out:

autopkgtest [09:12:56]: test testsuite: [-----------------------
[1] untrusted connection ... ok
[1] open "add security exception" dialog ... ok
[1] add security exception ... TIMEOUT
Traceback (most recent call last):
  File "./test/run", line 109, in <module>
    main()
  File "./test/run", line 61, in main
    _main()
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/autopkgtest.ndvXJ4/autopkgtest_tmp/test/firefox.py", line 147, in clean_home_dir
    yield
  File "./test/run", line 61, in main
    _main()
  File "./test/run", line 83, in _main
    'alt+c',
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "./test/run", line 46, in check
    yield
  File "./test/run", line 83, in _main
    'alt+c',
  File "/tmp/autopkgtest.ndvXJ4/autopkgtest_tmp/test/firefox.py", line 93, in talk
    ipc.check_call(cmdline)
  File "/usr/lib/python3.5/subprocess.py", line 576, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.5/subprocess.py", line 559, in call
    return p.wait(timeout=timeout)
  File "/usr/lib/python3.5/subprocess.py", line 1658, in wait
    (pid, sts) = self._try_wait(0)
  File "/usr/lib/python3.5/subprocess.py", line 1608, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
  File "./test/run", line 36, in raise_
    raise Timeout
__main__.Timeout
autopkgtest [09:13:21]: test testsuite: -----------------------]

Is there some way I can make the output more verbose?

[jwilk]

I probably only tested under a real X server. Oops.
Lack of _NET_ACTIVE_WINDOW support can be worked around by running a window manager under Xvfb. For example, this seems to work for me:

xvfb-run sh -c 'i3 & test/run' 

The timeout is expected if 0c224ce was not applied.
The test waits for a window to appear, but it will never happen, because xdotool sent the wrong keys...

[spwhitton]

Thanks for the feedback.

After installing i3, I tried running the command you suggested but it
still times out while adding the security exception. I have 0c224ce
applied (as a quilt patch, with the stack pushed).

Could you the version of Firefox you're using?

Sean Whitton

[jwilk]

45.2.0esr-1 I should probably try a newer version...

[jwilk]

ca7b79e fixed it at least for Firefox 47.0.

[jwilk]

In d43c78c, I fixed compatibility with Firefox 48.

[spwhitton]

Hello @jwilk. The test suite is failing at present. I'm not yet sure if this is a temporary failure or an incompatibility with Firefox 50.

[jwilk]

The tests work for me when run against Firefox 50, so that's probably something else.

[spwhitton]

Looks like it was a temporary failure. Thanks for your feedback.

…

-- Sean Whitton