IDLE functional/integration testing

[roseman]

BPO	24845
Nosy	@terryjreedy, @kbkaiser, @taleinat, @serwy, @roseman
PRs	bpo-37903: IDLE: Shell sidebar with prompts (take #2) #22682
Files	functionaltests.patch functionaltests.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2015-08-11.20:38:01.069>
labels = ['expert-IDLE', 'type-feature', '3.7']
title = 'IDLE functional/integration testing'
updated_at = <Date 2020-10-15.04:15:20.114>
user = 'https://github.com/roseman'

bugs.python.org fields:

activity = <Date 2020-10-15.04:15:20.114>
actor = 'taleinat'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2015-08-11.20:38:01.069>
creator = 'markroseman'
dependencies = []
files = ['40164', '40165']
hgrepos = []
issue_num = 24845
keywords = ['patch']
message_count = 6.0
messages = ['248428', '248429', '297864', '297987', '297994', '298007']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'kbk', 'taleinat', 'roger.serwy', 'markroseman']
pr_nums = ['22682']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24845'
versions = ['Python 3.6', 'Python 3.7']

[roseman]

This is a placeholder issue for adding automated functional/integration tests to complement the existing unit tests.

[roseman]

I've attached functionaltests.patch which provides a starting point, using Tk introspection and event generation to exercise the running application. The heart of it is the (very much in progress) TkTestCase class which provides a bunch of Tkinter-specific utilities to be used for tests.

One thing I'm not sure about is that because these run somewhat slower than the existing unit tests, should they be shut off by default if someone is running a repository-wide test?

[terryjreedy]

I have accepted what I think is the core idea -- testing functional pathways starting with user actions instead or in addition to pure unit tests. For instance, test_configdialog.GeneralTest (newly revised in PR 2612) invokes buttons and inserts entries. These simulated user actions change Variable values; the trace callback is then called to add an option to the changes data structure. The test is that the right addition was made. I will not bother to test the callback separately as a unit.

Test speed is very important to me as I rerun a module's tests constantly when editing a module or test_module. I try to efficiently cover the code, including taking branches both ways, without heaps of redundancy.

The search box tests use tk introspecton, but I didn't like it as it. For me, it made the tests less clear. I would rather trade space for speed and clarity by keeping named references to all python objects needed for tests.

So for the moment, I prefer my implementation of the core idea.

[terryjreedy]

A different take on the proposal: A functional integration test for IDLE should open IDLE, rename .idlerc, open a new file, insert some text, and, for instance, at some point open the options dialog for testing. As part this, the font face change would simulate button clicks and then check that the the font had changed in the editor *and* in .idlerc/config-main.cfg. I have done this manually. A complete functional test would do what I also occasionally do. Exercise at least once every menu function, all dialog widgets, and a few invisible functions. Unless suppressed, even the print function should be included, verified by the user affirming that the printed page appeared.

Such a test should be separate from the unittest test suite, just as is the htest module. Only a few buildbots run gui tests anyway, but some core devs run the suite regularly, some with gui enabled. I don't want to impose on them in any way, neither another minute or two of test time, nor flickers and bells. Running such a test suite should be strictly voluntary.

A few files could go in idle_test, like htest.py. A large number might justify a separate func_test directory.

I don't want the mode of invocation and operation and the code style to be limited to that of unittest.

I envision IDLE and the functest code running in one process, with one Tk instance. Which starts first is to be decided. But if IDLE starts first, then Run Func Test can be a menu item. I have already though of adding Run Unit Tests (in a subprocess) and Run Visual Tests (in the same process) to the Help Menu.

The main use of setup and teardown in IDLE unit tests is to create and destroy Tk roots; with one permanent root, these functionsgo away.

If test_xyz functions are defined at module scope, they are easy to collect and run. This is the function I use in my own project.

def main(namespace):
    for name, obj in namespace.items():
        if name.startswith('test_') and hasattr(obj, '__call__'):
            print(name)
            obj()

The obj() call could be wrapped with

try: obj()
except Exception as e: print(e, file=output_window)

where output_window is an instance of IDLE's OutputWindow. A helper equal(x,y) function could send an error message to the same place.

[terryjreedy]

To clarify the 'tk introspection' I did not like was asking a widget for its children, getting a list of tk ids, and then testing the widgets by id. I prefer keeping and and directly using a python reference. I was not referring querying a widget for its configuration.

[Not part of functional testing, but... Current unittests do not check that every widget is managed (placed, packed, or gridded). I just verified that commenting out a widget's .pack call did not affect the current test of the widget. We currently use the htests for that.]

[terryjreedy]

The problem with this proposal, and with writing complete unittests, is that event_generate seems to be badly broken. I have spent hours doing experiments that mostly fail. I have read Stackoverflow questions and there seems to be no dependable rule for success.