Idle: make human-mediated GUI tests usable #62304
Comments
|
23 of the 62 idlelib/*.py files have an 'if __name__ ...' test that brings up a tkinter gui widget and waits for the human tester to do ... something. Problems:
Running all is much worse given that I found no list; doing the above with all 62 files to find the 23 is something few would do.
My overall idea expands on the example of the dialog tests, such as I am thinking to suffix human-only tests with '_H' and put 'test_xyy_H' in a subdirectory such as idle_test/human. Some test files might test several different modules, such as 'test_dialog_H'. |
|
Since you *don't* want these to be autodiscovered by testing frameworks, perhaps h_test_xxxx.py would be a better naming scheme. |
|
What I want is for the tests to be discovered when they should be and not when they should not ;-) -- without jumping through too many hoops. I probably once read 'recursing into subdirectories' but forgot. I like 'h_test...' for 'human test ...'. Thinking about it more, unittest does not not add much to this type of test. If dialogs maps dialog to test information used by h_test_dialog, then a first draft of testing all (there are 6 that I know of, 3 with display tests) could amount to |
|
While improving the self-test for confixSectionNameDialog.py after fixing the error that stopped it from running, I discovered an error in the NameOk method: names were not really stripped before being tested. so duplicate section names were erroneously accepted if initially surrounded by whitespace. After fixing that, I removed unneeded code, including 5 pairs of parentheses in one statement, and added recommended spaces. I also centered the action buttons in the dialog. Attached is a drop-in replacement for the file (produced with Idle on my non-developement Win7 machine, so probably with the extra line end char). I think it is good enough to commit, but I am curious if the instructions added to the test are clear enough. (The dialog is the one seen if Save as New Custom Theme (Highlighting tab) or Save as New Custom Key Set (Key tab) are selected on the Options / Preferences dialog. For new custom test files, the root window should be open farther from the edge of the screen. |
|
I opened bpo-18130 for an updated configSectionNameDialog.py patch. It also adds to idle_tests two files, mock_tk.py and test_config_name.py. The latter uses the former for gui-free automated tests of some of the dialog methods. I plan to commit in a couple of days. The fixed human test is required for this issue. |
|
Some follow-up after revising the test for GetCfgSectionNameDialog in configSectionNameDialog.py (bpo-18130):
if __name__...:
<run unittests, as usual>
<if file defines visible widget, also run human test>where <run human test> currently means to open a tk window with a button to start the test, but which can instead just be closed [x] to skip the test. My idea is that after editing, one should always run the automated test (if there is one, which there eventually will be) and maybe run the visual test.
I need to pick, say, 3 tests and do some experiments. |
|
I thought through my design criteria a bit, and in the process, decided on a specific concrete proposal to test.
from idlelib.idle_test.htest import run
run(GetCfgSectionNameDialog) # or other class
Something like the following may work for both criteria. In idle_test/htest.py GetCfgSectionNameDialog_spec = {
'kwds': {'title':'Get Name',
'message':'Enter something',
'used-names': {'abc'}},
'msg': "After the text entered with [Ok] is stripped, <nothing>, "
"'abc', or more that 30 chars are errors. "
"Close with a valid entry (printed), [Cancel], or [X]"}
def run(klas):
root = tk.Tk()
klas_spec = globals[klas.__name__+'_arg']
klas_spec.kwds['parent'] = root # does Idle use 'parent' consistently?
def run_klas():
widget = klas(**klas_spec.kwds)
try:
print(widget.result)
except AttributeError:
pass
Message(root, text=klas_spec.msg).pack()
Button(root, text='Test ' + klas.__name__, command=run_klas).pack()
root.mainloop()
filter(lambda ob: hasattr(ob, '__name__') and '_args' in ob.__names__, globals()) |
|
After correcting bugs in the 'explaining with code' posted before, and making a few other changes, the attached 3.3 diff works. The main substantive change is that I added a parameter to the class to move the tested dialog under the toplevel driver so the test message would be visible. The default for this dialog is centered over the parent (normally the config dialog). If I still like this design after adding a couple more tests and a run_all function, I will commit it. |
|
The new patch adds a docstring with spec template, a second example, and a crude runall(). I am inclined to push this as a base for further patches -- at least one to improve runall and others to test more widget classes. These might be separate issues. |
|
New patch. I refined the definition of the parameter for run() in the process of adding a test for EditorWindow.HelpDialog. This cannot be tested directly. This example is also a reminder that some files define more than one display widget. I should try template and framework with non-dialog, but the example above pretty well convinces me that the framework should be flexible enough. |
|
Defective patches removed and replaced with one that includes current htest.py and should have proper (unix) line endings. |
|
Here are my notes on modules that should probably be included in htest. Most have some end-of-module test now, many with errors. |
|
New changeset 460203eaf731 by Terry Jan Reedy in branch '2.7': New changeset 617656f7b538 by Terry Jan Reedy in branch '3.4': |
|
New changeset aea4f427902f by Terry Jan Reedy in branch '3.4': New changeset b7bc43e96041 by Terry Jan Reedy in branch 'default': New changeset 7cdb38fa191e by Terry Jan Reedy in branch 'default': |
|
The revised patch, as pushed, will serve as a base for this portion of Saimadhav's GSOC project. bpo-21477 proposes changes to htest itself. Adding tests from other modules will be dependencies of that issue. |
|
New changeset 0a6d51ccff54 by Terry Jan Reedy in branch '2.7': New changeset 6d2982ff441f by Terry Jan Reedy in branch '3.4': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: