Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Announcement of suggested contacts in outlook 2010 #2816

Closed
nvaccessAuto opened this issue Nov 21, 2012 · 18 comments
Closed

Announcement of suggested contacts in outlook 2010 #2816

nvaccessAuto opened this issue Nov 21, 2012 · 18 comments
Labels
Milestone

Comments

@nvaccessAuto
Copy link

Reported by anubhav on 2012-11-21 06:54
When typing any e-mail address in “To” or “Cc” feed of Microsoft 2010, NVDA should announce the list of name/e-mail address suggested by Outlook.
Blocked by #3354
Blocking #3134

@nvaccessAuto
Copy link
Author

Attachment bug 2816 - outlook contacts auto complete spoken.patch added by manish on 2013-06-06 09:56
Description:

@nvaccessAuto
Copy link
Author

Comment 2 by manish on 2013-06-06 10:00
The attached patch fixes:

  • autocomplete list of addresses in the "to" and other address fields is now read using up and down arrow keys
  • If multiple email accounts are configured in outlook, the "from" address dropdown is now correctly read.

This has the following 3 problems still:

  1. If there are a large number of suggested contacts (some are not visible on screen), the the first item in the autocomplete address list is announced only after 3-4 characters. This appears to be a bug in the outlook MSAA implementation because the state changed event for that item is not raised for the first few key presses.
  2. Several state changed events are raised by the autocomplete list window that do not eventually yield an accessible object when oleacc.GetAccessibleObjectFromEvent is called. This also appears to be a bug in the IAccessible implementation because the child IDs being received in these events are very different from those received when an accessible object is correctly obtained by oleacc.
  3. Object navigation does not work in the autocomplete list. On moving the navigator object to any of the list items causes the list to close and the currently selected item to be entered in the "to" field as if the tab character was pressed. The same behavior is observed if i try to print obj.DevInfo() for any of the list item objects. This again looks like a buggy IAccessible outlook implementation.

I haven't found a workaround for these so far.
These are however, significantly minor as compared to the current problem where NVDA is not able to read the contents of the autocomplete box at all.

@nvaccessAuto
Copy link
Author

Comment 3 by jteh on 2013-06-07 00:21
Thanks for the patch!

I'm reluctant to add classes which might not be application specific (the name NetUIHWND doesn't suggest it's specific to Outlook) to badUIAWindowClassNames unless there really is no other way. Are you certain the UIA control is broken; i.e. doesn't fire events or report the right information? If it isn't broken, the overlay class can be changed to be generic so it works for both IAccessible and UIA, thus avoiding the need for it to be blacklisted.

Just a minor code review point: when testing whether something has one of a set of values, you should do:
blah in (a, b, c)
instead of:
blah == a or blah == b or blah == c
The former is more efficient and easier to read.
Changes:
Milestone changed from None to 2013.2

@nvaccessAuto
Copy link
Author

Comment 4 by manish (in reply to comment 3) on 2013-06-08 13:11
NetUIHWND is a class name used by a lot of windows in the ms office 2010 set of applications and not just in outlook. None of these are UIA, however. I will try to explore if office 2013, which implements UIA also uses this class name.

Is there an example of how to make the same overlay class work for both IAccessible and UIA that I can follow here?

For your code review comment, I will start using it going forward. Being a c# programmer primarily, I wasn't familiar with the "in" construct.
Replying to jteh:

Thanks for the patch!

I'm reluctant to add classes which might not be application specific (the name NetUIHWND doesn't suggest it's specific to Outlook) to badUIAWindowClassNames unless there really is no other way. Are you certain the UIA control is broken; i.e. doesn't fire events or report the right information? If it isn't broken, the overlay class can be changed to be generic so it works for both IAccessible and UIA, thus avoiding the need for it to be blacklisted.

Just a minor code review point: when testing whether something has one of a set of values, you should do:

blah in (a, b, c)

instead of:

blah == a or blah == b or blah == c

The former is more efficient and easier to read.

@nvaccessAuto
Copy link
Author

Comment 5 by kyogesh on 2013-06-12 14:06
No UIA event is fired while typing in To/Cc edit fields in MS Outlook 2010.

Here is the log for it:

INFO - NVDAObjects.UIA.new (19:28:04):
(<class 'NVDAObjects.UIA.UIA'>, 67430L, <POINTER(IUIAutomationElement) ptr=0x2e7d838 at 651bbc0>)
INFO - NVDAObjects.UIA.UIA.init (19:28:04):
(<NVDAObjects.UIA.UIA object at 0x06595770>, 67430L, <POINTER(IUIAutomationElement) ptr=0x2e7d838 at 651bbc0>)
INFO - NVDAObjects.UIA.UIA.findOverlayClasses (19:28:04):

INFO - NVDAObjects.UIA.UIA._get_TextInfo (19:28:04):

INFO - NVDAObjects.UIA.UIA._get_UIATextPattern (19:28:04):

INFO - NVDAObjects.UIA.UIA._getUIAPattern (19:28:04):

INFO - NVDAObjects.UIA.UIA._get_role (19:28:04):

INFO - NVDAObjects.UIA.UIA.findOverlayClasses (19:28:04):
'NVDAObjects.UIA.UIA'>
INFO - NVDAObjects.UIA.UIA._get_processID (19:28:04):

INFO - NVDAObjects.UIA.UIA._get_states (19:28:04):

INFO - NVDAObjects.UIA.UIA._get_firstChild (19:28:04):

INFO - NVDAObjects.UIA.new (19:28:04):
(<class 'NVDAObjects.UIA.UIA'>, None, <POINTER(IUIAutomationElement) ptr=0x2e7d878 at 651bb70>)
INFO - NVDAObjects.UIA.UIA.init (19:28:04):
(<NVDAObjects.UIA.UIA object at 0x06595650>, None, <POINTER(IUIAutomationElement) ptr=0x2e7d878 at 651bb70>)
INFO - NVDAObjects.UIA.UIA.findOverlayClasses (19:28:04):

INFO - NVDAObjects.UIA.UIA._get_TextInfo (19:28:04):

INFO - NVDAObjects.UIA.UIA._get_UIATextPattern (19:28:04):

INFO - NVDAObjects.UIA.UIA._getUIAPattern (19:28:04):

INFO - NVDAObjects.UIA.UIA.findOverlayClasses (19:28:04):
'NVDAObjects.UIA.UIA'>
INFO - NVDAObjects.UIA.UIA._get_processID (19:28:04):

INFO - NVDAObjects.UIA.UIA._get_role (19:28:04):

INFO - NVDAObjects.UIA.UIA._get_states (19:28:04):

INFO - NVDAObjects.UIA.UIA.correctAPIForRelation (19:28:04):
(<NVDAObjects.UIA.UIA object at 0x06595770>, <NVDAObjects.UIA.UIA object at 0x06595650>, None)
INFO - NVDAObjects.UIA.UIA._get_next (19:28:04):

INFO - speech.speakTypedCharacters (19:28:07):
y
INFO - core.main (19:28:12):
Exiting
INFO - config.save (19:28:12):
Configuration saved
INFO - nvda (19:28:13):
NVDA exit

@nvaccessAuto
Copy link
Author

Comment 6 by manish (in reply to comment 3) on 2013-06-13 13:43
UIA is not broken for this control. On deriving the AutoCompleteListItem class from UIA, I am able to handle the name changed event for the various list items when they become highlighted for the first time (which solves problem 1 mentioned in my original comment).
However, when I move using the arrow keys back to an item for which the UIA NameChange event has already been raised when it was first highlighted, no stateChange event gets fired even though the state of the item goes from unselected to selected.
When the AutoCompleteListItem was derived from IAccessible, then the correct StateChange event was raised on moving with the arrow keys.

So, I guess, I need a class that implements both UIA and IAccessible. I wasn't able to figure out how to do that. Will appreciate any pointers on how to create a class that can handle both UIA and IAccessible events.

Also, is there a way to find out which all UIA events are being fired when I am moving using the arrow keys - I am trying to see if there is another event instead of stateChange that gets raised in UIA.
Replying to jteh:

Thanks for the patch!

I'm reluctant to add classes which might not be application specific (the name NetUIHWND doesn't suggest it's specific to Outlook) to badUIAWindowClassNames unless there really is no other way. Are you certain the UIA control is broken; i.e. doesn't fire events or report the right information? If it isn't broken, the overlay class can be changed to be generic so it works for both IAccessible and UIA, thus avoiding the need for it to be blacklisted.

Just a minor code review point: when testing whether something has one of a set of values, you should do:

blah in (a, b, c)

instead of:

blah == a or blah == b or blah == c

The former is more efficient and easier to read.

@nvaccessAuto
Copy link
Author

Comment 7 by jteh (in reply to comment 6) on 2013-06-20 02:36
Replying to manish:

NetUIHWND is a class name used by a lot of windows in the ms office 2010 set of applications and not just in outlook. None of these are UIA, however.

If you had to add that class to badUIAWindowClassNames, that suggests they are native UIA.

Is there an example of how to make the same overlay class work for both IAccessible and UIA that I can follow here?

You just inherit from NVDAObjects.window.Window instead of NVDAObjects.IAccessible.IAccessible and make sure your chooseNVDAObjectOverlayClasses isn't checking for anything IAccessible specific. Of course, your overlay class can't do anything IAccessible specific either.

For your code review comment, I will start using it going forward. Being a c# programmer primarily, I wasn't familiar with the "in" construct.

No problem.

Replying to manish:

So, I guess, I need a class that implements both UIA and IAccessible. I wasn't able to figure out how to do that.

You can't really do that. Because Windows/the app often proxies between the two and fires both sets of events, NVDA has to ignore one or the other. You could set up your own low level event handlers, but that's fairly tricky.

Also, is there a way to find out which all UIA events are being fired when I am moving using the arrow keys - I am trying to see if there is another event instead of stateChange that gets raised in UIA.

I'd probably use Microsoft's Accessible Event Watcher tool. That removes NVDA from the equation.

@nvaccessAuto
Copy link
Author

Comment 8 by manish (in reply to comment 7) on 2013-06-20 12:04
I used Accessible event watcher to track all the events being raised by UIA and IAccessible for this.
The findings are the same as above:

  1. UIA raises Name change events only once for each item when it gets selected the first time. No UIA events are raised on selection change using arrow keys.
  2. The window raises IAccessible state change events for selection changes using the arrow keys. however, no IAccessible event is raised if an item gets selected automatically because of a match with user input untill that is the only item in the list.

For now, I suggest we use the IAccessible events for this because that is significantly better than the current state where the items are not read at all.
Questions:

  1. How can I have NVDA subscribe to IAccessible events for this window without putting "NetUIHWND" in the BadUIAClassNames list because this class name is used by many other windows as well, where UIA may not be broken as is the case here.
  2. The window ID of the parent window is always the same - Can that be used along with the window class name to somehow choose IAccessible as the best API instead of UIA for this?
  3. Does the app module get a chance to decide the best API for a window - similar to the ChooseOverlayClasses method, is there a ChooseAPI method as well?

Thanks for your help.

-Manish
Replying to jteh:

Replying to manish:

NetUIHWND is a class name used by a lot of windows in the ms office 2010 set of applications and not just in outlook. None of these are UIA, however.

If you had to add that class to badUIAWindowClassNames, that suggests they are native UIA.

Is there an example of how to make the same overlay class work for both IAccessible and UIA that I can follow here?

You just inherit from NVDAObjects.window.Window instead of NVDAObjects.IAccessible.IAccessible and make sure your chooseNVDAObjectOverlayClasses isn't checking for anything IAccessible specific. Of course, your overlay class can't do anything IAccessible specific either.

For your code review comment, I will start using it going forward. Being a c# programmer primarily, I wasn't familiar with the "in" construct.

No problem.

Replying to manish:

So, I guess, I need a class that implements both UIA and IAccessible. I wasn't able to figure out how to do that.

You can't really do that. Because Windows/the app often proxies between the two and fires both sets of events, NVDA has to ignore one or the other. You could set up your own low level event handlers, but that's fairly tricky.

Also, is there a way to find out which all UIA events are being fired when I am moving using the arrow keys - I am trying to see if there is another event instead of stateChange that gets raised in UIA.

I'd probably use Microsoft's Accessible Event Watcher tool. That removes NVDA from the equation.

@nvaccessAuto
Copy link
Author

Comment 9 by jteh on 2013-06-26 03:00
Changes:
Milestone changed from 2013.2 to next

@nvaccessAuto
Copy link
Author

Comment 10 by jteh (in reply to comment 8) on 2013-06-28 00:20
Replying to manish:

For now, I suggest we use the IAccessible events for this because that is significantly better than the current state where the items are not read at all.

That sounds ideal, though it's problematic to implement.

  1. How can I have NVDA subscribe to IAccessible events for this window without putting "NetUIHWND" in the BadUIAClassNames list because this class name is used by many other windows as well, where UIA may not be broken as is the case here.

NVDA doesn't provide a way to do this because it would result in duplicate events in most cases, and even where it doesn't, it'd fire events on two different types of objects (one UIA, one IAccessible). The only way to do this is to create your own low level win event hook.

  1. The window ID of the parent window is always the same - Can that be used along with the window class name to somehow choose IAccessible as the best API instead of UIA for this?

Do you mean the window handle or the window control ID? If you mean the window handle, it will change for each application session (if not for each new message), so this can't be used. Control ID can't be used for this yet either, though there's at least a chance if that remains constant.

  1. Does the app module get a chance to decide the best API for a window - similar to the ChooseOverlayClasses method, is there a ChooseAPI method as well?

No. This isn't possible; this choice needs to happen when handling the events from the low level API, which has to happen before an NVDAObject is even created.

I'll keep thinking on possible solutions.

@nvaccessAuto
Copy link
Author

Comment 11 by jteh on 2013-06-28 03:11
After a very long discussion with Mick, we decided to just add NETUIHWND to badUIAWindowClassNames as your original patch does. It doesn't seem like we lose anything doing this (we checked a few other places that use this class) and the alternatives are far too ugly and tedious. However, before we do this, we do need to fix an issue with handling of badUIAWindowClassNames which will cause problems for the ribbon if we don't.

@nvaccessAuto
Copy link
Author

Comment 12 by manish (in reply to comment 11) on 2013-06-28 11:17
We cannot do this because this will break the fix for ticket #649. The status bar window in office applications has the window class name "netUIHWND". and putting it in BadUIAWindowClassNames will prevent the status bar from being identified.

I'll think more to see if I can come up with some alternative.
Replying to jteh:

After a very long discussion with Mick, we decided to just add NETUIHWND to badUIAWindowClassNames as your original patch does. It doesn't seem like we lose anything doing this (we checked a few other places that use this class) and the alternatives are far too ugly and tedious. However, before we do this, we do need to fix an issue with handling of badUIAWindowClassNames which will cause problems for the ribbon if we don't.

@nvaccessAuto
Copy link
Author

Comment 13 by Michael Curran <mick@... on 2013-07-12 04:44
In [cf706c4]:

Outlook 2010 auto completes now work due to now using MSAA, not UIA in this situation. Re #2816

Specifically: UIAHandler: treet a window with a class of NetUIHWND who's root window's class is Net UI Tool Window, as having a bad UIA implementation (we will fallback to IAccessible).
At the same time, slightly restructure UIAHandler.handler.isUIAWindow moving the choice logic in to a new _isUIAWindowHelper function, so as to not get confused by the outer cashing code.

@nvaccessAuto
Copy link
Author

Comment 14 by Michael Curran <mick@... on 2013-07-12 04:49
In [d344235]:

Merge branch 't2816' into next

incubates #2816

Changes:
Added labels: incubating

@nvaccessAuto
Copy link
Author

Comment 15 by mdcurran on 2013-07-12 04:55
For reference, the patch on this ticket was committed to branch t2816 in c0195a2. However, only changes to the appModule was included, not the UIAHandler changes as this was handled differently in the next commit.

@nvaccessAuto
Copy link
Author

Comment 16 by mdcurran on 2013-07-12 04:57
Its also worth noting that the solution does not break Office 2010 status bars as we are being very specific about the NetUIHwnd window in question.

@nvaccessAuto
Copy link
Author

Comment 18 by Michael Curran <mick@... on 2013-07-25 03:24
In [49fcafb]:

Outlook 2010 auto completes now work due to now using MSAA, not UIA in this situation. Fixes #2816

Specifically: UIAHandler: treet a window with a class of NetUIHWND who's root window's class is Net UI Tool Window, as having a bad UIA implementation (we will fallback to IAccessible).
At the same time, slightly restructure UIAHandler.handler.isUIAWindow moving the choice logic in to a new _isUIAWindowHelper function, so as to not get confused by the outer cashing code.

Changes:
Removed labels: incubating
State: closed

@nvaccessAuto
Copy link
Author

Comment 19 by mdcurran on 2013-07-25 03:26
Changes:
Milestone changed from next to 2013.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant