Adding autopilot (UI) tests

[gdm85]

I have noticed that there are no UI tests in this repository, nor upstream at major distros.

autopilot seems to be the best choice for GTK-based applications, for an example see http://bazaar.launchpad.net/~ubuntu-testcase/ubuntu-autopilot-tests/trunk/view/head:/ubuntu_autopilot_tests/gedit/test_gedit.py (there you can browse tests for other applications like firefox etc)

• would Geany benefit from such tests? is this repository a natural and correct place to add them?
• shall a Github project be opened to track this?
• can we easily integrate such tests in Travis PR builder by using some Xvfb-based container or qemu?

Obviously I think there would be some benefit, as regression tests can progressively be added during development. I can probably be of help to bootstrap the GH integration to make this happen with some dummy tests, but I would likely not have the momentum/desire to add hundreds of tests at once and rather expect that new developments on the UI front would also add some autopilot tests for completeness (if that becomes an accepted good practice).

[elextr]

It would be nice to have UI tests, after all Geany is almost entirely a UI application.

But my suspicion is that it will be subject to the problem that you identified, neither you nor me nor anybody else has the desire to add lots of tests for the existing UI. And as anything beyond minimal UI changes are very rare, tests won't grow very much just from UI changes.

[gdm85]

My idea in proposing this was to cover hard-to-reproduce bugs like #1252 or other weirdnesses. I see it also as a "safeguard" if/when plugins start to have a bigger role in feature extension (imagine the tests Mozilla does for the browser extensions).

[elextr]

Well, I'm not sure that "weirdness" is the most likely use-case for such tests, since weirdness usually means difficulty in reproducing the problem, which means its going to be difficult to write a test to reproduce it.

I'm not sure how the proposed test harness works, can it see what changes happen on the edit buffer? Thats where most results appear.

As Geany recently had accessibility capabilities added, perhaps some of the testing mechanisms that use accessibility to read the screen may be appropriate, but AFAIK nobody has looked at doing it.

Note that very few plugins are part of Geany itself, most are third party maintained, see the Geany-Plugins repository and others are purely listed on the plugin website. So its not really up to Geany itself to apply such testing.

[gdm85]

I am no test fanatic myself but reproducing is always the first step to understand (and then solve the problem); if it were easily possible to "code" the steps to reproduce a test, we'd get for free some coverage against regressions; but don't want to reiterate, we can also just close this (or wait some other feedback, at your option).

(NOTE: I have never used autopilot myself but I am familiar with other UI testsuites) From what I had understood on my initial research Autopilot already has "introspection" features (see https://developer.ubuntu.com/api/devel/ubuntu-14.04/autopilot/api/introspection.html) so that one could access GTK objects without relying on A11Y features of the application.

As for the plugins, you're right, any discussion for a "plugins testsuite" for 3rd-parties is probably worth be separate.