Testing Toga

[Ocupe]

Testing Toga

Divide and conquer

The vision is to come up with two sets of testing tools. One inward facing, and one outward facing set of tests.

Inward Facing Tests

These tests are mostly for Toga.core

Toga core unit tests

This set of tests takes care of the question does Toga core work as expected. For example, If I create a Button with a label, is the label set correctly? This means that we test platform and implementation independent.

How to implement it

This part of the testing suite is pretty much your normal testing routine. Import a class from toga.core and run your unit tests against it.

Implementation checks

Performs checks whether the platform specific implementation of toga.core on iOS, Windows, Linux etc. specifies all the required widgets, classes, methods and properties that are defined in toga.core.

This could also be used to automate the documentation to reflect the current status of all implemented Toga backends. For example, it could check whether the Button with all its functionality is already implemented on Windows. If the widget is implemented, change the docs to reflect that. If only a part of the widget is implemented show what part and what parts still need implementation.

How to implement it

The implementation checks are pretty much a series of checks if some defined structure is fulfilled or not. For example, if we wanted to do a check for toga.button than we would have to specify what functions, classes and property are required to fulfil our specifications. The test would walk through the different implementation of toga and would check if a button.py file with a class ```Button''' and the required functionality exists.

I imagine a functionality where you can set a flag whether or not to test for a certain implementation. This way it only tests the parts that someone claimed to be implemented. Otherwise each test run would cause failures because of not implemented parts.

Outward Facing Tests

I'm not sure if I got this right? Feedback is more than appreciated :)

With these tests you can test if your application behaves in the expected way. One discussed way of tackling the problem of platform independent testing is to create yet another backend that integrates toga.core (like toga.iOS, toga.cocoa etc.). This so called dummy or testing backend allows us to test headless (without opening any window/app), this simplifies the testing suite enormously. By not having to open a app you also lose the platform specific way of how to interact with the app (moving the mouse, clicking, touch on mobile ...).

This dummy backend is also useful for users to test their apps against.

A known problem of the "dummy" backend solution is the we can not test how the Toga.core implementation, for example in iOS, interacts with the platform specific API. The good thing about the here proposed solution is, it doesn't prevent us from adding platform specific tests in the future.

File Structure for Tests

# possible structure for inward facing tests
tests
|___test_core
|   |   test_app.py
|   |   test_window.py
|   |   ...
|   |___test_widgets
|       |   test_button.py
|       |   test_label.py
|       |   ...
|___test_implementation_ckecks # 
    |   test_app.py
    |   test_window.py
    |   ...
    |___test_widgets
        |   test_button.py
        |   test_label.py
        |   ...

[freakboy3742]

I think you're on the right track here, but a little more elaboration would help confirm this.

What might be helpful is to show (possibly at a high level, with plenty of handwaving around details if necessary), what this would look like for a simple widget feature (like setting a label on a button). What does the core unit test, implementation test, and outward test look like in those cases?

[Ocupe]

@freakboy3742 Sorry for the late answer, it is finals week for me. I don't know if this is detailed enough to see where I'm heading. If not let me know.

Example on the basis of Button

This examples is a high level / pseudo code / text description of how the different tests could look like.

Example: Toga core unit tests

Testing Toga core widgets is often done in few test cases. This will change when the implementation of more complex APIs (Settings API, Actions/Menus API etc.) follows. Nevertheless there should be test in place which cache "pluming" errors.

• Is the toga.core.button imported in the interface package?
• Can I import the button?
• Does the Button logic function like expected?

from src.core.toga.interface import Button

class TestCoreButton(unittest.TestCase):
    def test_label(self):
        btn = Button('label_name')
        self.assertEqual(btn._label, 'label_name')
    ...
    ...

Example: Implementation checks

To check whether or not backends have implemented toga.core as expected/defined.
I would split the test logic into two parts. One part that defines the requirements the other one tests if the defined requirements are meet. This way we only have to define the requirements once and can use it to test all the implementations of Button for every backend. This leads to a consistent implementation and naming of functions/classes/etc.. To test for classes, methods etc. I would make use of the inspect module form the standard library.

# in define_button.py
# 
# Things that have to be present in the implementation
# of a platform specific toga.Button.
- class Button
- def create()
- def _set_label()
- def _set_enabled()
- def _set_on_press()

------------------------------

# in test_implementation_button.py

import inspect
# import the definitions from define_button.py
from define_button import definitions


# which platforms should be testet?
cocoa = True
iOS = True
windows = False
linux = True
...
platforms = [cocoa, iOS, windows, linux, ...]
...

# check for existents
for platform in platforms:
    if platform:
        for definition in definitions:
            # test if the class/function/property etc. for button exists.
            methodes = inspect.getmembers(Button, predicate=inspect.ismethod)
            
            
# integration for automatic documentation updates here possible.

Example: Outward Facing Tests (work in progress)

As I mentioned before this part need some time and brain power and is out of scope for GSoC. But as I see this issue as more as only a GSoC application I would like to share my thoughts on this as well.

Some questions that I have asked myself:

• What would one like to test, that is not possible to test without running the app.
  • User interaction (button.on_press, on_change, ...)
  • App state changes (on_load, will_load, is_loaded etc.)

# dummy_backend
# in toga.src.dummy.interface.widget.button.py

class Button(Widget):
    # button class implements the basic logic that is required by a toga.core.Button
    # on top of that it also has logic on board that is needed for testing.
    # All the work that is normally done by the platform specific backend
    # has to be done by the dummy backend. 
    

[Ocupe]

I think this was addressed by the GSoC project 2017