find_best_control_matches causes ComboBox flutter

[bernd-wechner]

This is a tad bothersome to say the least. When i try to find a control by its name like:

control = win["name"]

all the ComboBoxes on my form flutter. It took me some serious drill down effort to nail this to:

ComboBoxWrapper.texts

which literally expands the combobox and collapses it again all in the name, it seems of finding three child controls that turn out to be:

uia_controls.ListViewWrapper - '', ListBox
uia_controls.StaticWrapper - '', Static
uia_controls.ButtonWrapper - 'Close', Button

But I'm looking for a panel in fact (Pane10), and I find it and it's all good and the software alas has no title or name on that control, and so forth, but in finding it I have to endure unwanted ComboBox flutter?

What is the work around here? I'm curious. I will explore it some more, but I really don't need all the ComboBox children to be checked when looking for a control unless I'm looking for one of those children. In short I kind of expect and guess this is a hope for, a setting through which I can say: Don't expand controls in the best match search, the one I want is visible thank you.

Indeed the whole search seems a tad inefficient as find_best_control_matches calls build_unique_dict on all the controls in a window (hundreds) to build a name map for finding. But seriously it can break out and bail the minute it finds a perfect match I would guess.

I'm hoping I can find a better way to get the control for this pane that is reliable and avoid using this best_match approach.

[bernd-wechner]

Ironically, looking deeper:

WindowSpecification.print_control_identifiers(self, depth=None, filename=None)

accepts a depth. And yet, it applies a depth check only to the print stage, building a list of all controls to the lowest depth first. Hmmmm ... definite room for improvement here, to respect depth on the control walk, so that I might conceivably use depth as a means of finding Pane10 without ever seeing CombBoxes expand because they are one tier deeper than the the pane I'm after.

Ideally I suspect that the best_match criteria should include one called "expand = True" which we could pass to a best_match search or request in some manner, which conducts the best match search without expanding controls. Just on basis of the unexpanded controls.

But seriously it would be nice if the walk of controls also respected a depth setting rather than just the printing of them which does.

[vasily-v-ryabov]

Hi @bernd-wechner thanks for all the suggestions. We aware about some methods which are too invasive. They are blockers for our script recorder architecture (PR #701) as well.

Reducing best_match algorithm complexity is another direction we're trying to work on. But it moves not so fast.

As a fast alternative, more detailed criteria like .child_window(title="...", control_type="...") can be used. Preliminary filtering by control_type is extremely fast because it doesn't require cross-process interaction while getting text may take 2 cross-process requests (depending on backend). So getting texts for reduced set of elements is a good idea we use while searching by particular property.

Also there are some inconsistencies between search criteria and class *ElementInfo properties that will be fixed in next major release. Though some part of backward compatibility will be lost.

[bernd-wechner]

@vasily-v-ryabov Thanks for the feedback and the tip. I just gave child_window a go, but no luck alas. A couple of curious emerged. Notably documentation suggests:

dlg = app.child_window(title="your title", classname="your class", ...)
yet that fails for me while:

dlg = app.top_window().child_window(title="your title", classname="your class", ...)

works and checking application.py, yes, child_window is a method on WindowSpecification not on Application, suggesting docs (internal and external are out of syn with the code a little?).

By the by, as it's easy to find it (pywinauto is not a huge package by any means and fairly easy to search), the bigger problem I face is that the control I'm looking for refuses to identify itself in any simple way.
print_control_identifiers() suggests:

   |    |    | Pane - ''    (L-890, T568, R-710, B1112)
   |    |    | ['7', 'Pane10']
   |    |    | child_window(auto_id="13176374", control_type="Pane")

But the auto_id seems to vary between runs (not be a hard property of the control but much as the name suggests, something generated by pywinauto for differentiation). Still, pywinauto finds it on a best_match on "Pane10" and I can check its element_info. But alas it contains no unique identifiers and my guess is Pane10 is itself a name generated by pywinauto (as in the 10th Pane in the list of controls it builds, would need to drill down further to find out if it's build_unique_dict() that adds these or some precursor). In any case it seems not to be a property of the control itself with lacks any identifiers.

[vasily-v-ryabov]

We have plans to permit creating WindowSpecification from wrapper object: see #570. After that we can remove or change behavior of top_window() (#399) because it may confuse in case the top window is changed, but WindowSpecification returned by .top_window() still points to previous one.

auto_id property is usually stable for many apps like WPF or WinForms. I'm curious which kind of application you're automating.

BTW, the mentioned pane can be identified by child_window(control_type="Pane", found_index=9) as an example.

[vasily-v-ryabov]

More wide implementation plan for 0.7.0 (the most critical issues) can be found here:
pywinauto 0.7.0 milestone with label "recorder_0_7_0_critical".

[vasily-v-ryabov]

I've found right way to enumerate all combo box items for WPF without fluttering. See FindItemByProperty method of ItemContainerPattern. Remarks section tells that we can pass PropertyId == 0 to enumerate all items. This pattern is supported for ListView and other WPF containers as well (@airelil it might be useful for VirtualizedItems enumaration in explorer.exe's ListView).

Unfortunately WinForms don't implement this pattern provider, but we can use non-invasive workarounds from backend="win32".

The most complicated case is Qt5. Both ways are infeasible there. I couldn't even get expanded/collapsed state. Getting texts are invasive, and nothing tells about any alternative way so far.

[vasily-v-ryabov]

@bernd-wechner could you please try the fix from latest master branch?

[bernd-wechner]

@vasily-v-ryabov:

@bernd-wechner could you please try the fix from latest master branch?

Apologies for going cold on this trail. Busy, busy, busy as they say. Am provisionally back on pywinauto and even submitted a small PR to test the water. On this one, alas I'm only using the UIA backend at the moment, and I haven't touched on a combobox yet, but I'll try this out again some time don't fret.

[claesmk]

I have a case where I have multiple ComboBoxes on a window, some of these have names and some don't (control.window_text() == ''). The ones without names will flutter anytime I perform an operation which uses a magic lookup on that windows. This was painful to debug in PyCharm because, apparently, there isn't a way to disable automatic variable evaluation in the debugger and, if you try to step through things, whenever the variables evaluate, the ComboBoxes flutter. I ended up adding some tracebacks to determine what was happening.

I put a hack into get_contro_names just to see if I could make this better:

    elif control.has_title and friendly_class_name != 'TreeView':
        try:
            if friendly_class_name == 'ComboBox':
                names.append('ComboBoxNoFlutter')
            else:
                for text in control.texts()[1:]:
                    names.append(friendly_class_name + text)

So, it's the call to control.texts() which expands the ComboBox and causes the flutter. This would not be so bad if it only happened once per window, however, as I mentioned above, it happens repeatedly.

I'm happy to try to come up with a real fix for this but could use some guidance from someone more familiar with the code such as @vasily-v-ryabov so I don't break something.

At a glance, to be sure not to break something, it would seem to require adding a new method or parameter which would set a new control.combo_expand property to either True to preserve existing functionality or False to avoid the fluttering.

Another option might be to preserve the information from the first call to texts() so that subsequent calls are not required. Expanding all ComboBoxes without names one time would not be that bad, but this might have a greater chance of breaking something.

[vasily-v-ryabov]

@claesmk this case could be filed as a separate issue after checking with atspi branch which is current active branch for next major release. To install from atspi branch:

pip uninstall -y pywinauto
pip install https://github.com/pywinauto/pywinauto/archive/refs/heads/atspi.zip