Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read Outlook's auto-complete suggestions in the To, CC & BCC fields of new emails #8502

Open
Qchristensen opened this Issue Jul 13, 2018 · 50 comments

Comments

Projects
None yet
@Qchristensen
Copy link

Qchristensen commented Jul 13, 2018

Steps to reproduce:

In Outlook, when you start typing a contact's name, Outlook displays a list of recently used email addresses and contacts which match what has been typed so far. So, if "Mar" is typed, Outlook will list any Mary's or Martin's in your contacts, or anyone with "mar" in their email address.

To navigate through the list, use the arrow keys and to use the selected contact or address, press enter or tab.

Actual behavior:

When using the arrow keys to move through the list of contacts, NVDA reads "Blank". The only way to have NVDA read the contacts is to hover the mouse over, then the information is read.

Expected behavior:

A user would expect NVDA to read the contacts as the focus moves to each.

System configuration:

Windows 10 fast insider build 17713.1000
Microsoft Office 365 / Outlook 2016 Version 1806 (Build 10228.20104 Click to run) - updated just now.

NVDA Installed/portable/running from source:

NVDA 2018.2.1 portable
NVDA Alpha 15527,9c3f5bc8 installed
NVDA 2016.4 temporary

(Same results with all three versions)

Other information about your system:

In the current build of Windows 10, Narrator does announce "Suggestions available" when the suggestions appear then proceeds to read the first one, and the current suggestion when moving through the list.

Other questions:

Does the issue still occur after restarting your PC?

Have you tried any other versions of NVDA?

I thought this used to work. Reading issue #8055 which talks about the autocomplete list (but encounters an error with Braille), it seems to indicate it was working then (if it wasn't, that surely would have been worth mentioning in that ticket), so it's possibly a regression in a recent build of Outlook.

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Jul 13, 2018

@Qchristensen

This comment has been minimized.

Copy link
Author

Qchristensen commented Jul 13, 2018

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Jul 13, 2018

@Qchristensen

This comment has been minimized.

Copy link
Author

Qchristensen commented Jul 13, 2018

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Jul 13, 2018

Hi,

This feature worked as advertised in earlier Office 365 releases (including Office 2013 I believe). I'm thinking the control has changed and we didn't keep up with that change.

CC @derekriemer

@Adriani90

This comment has been minimized.

Copy link
Collaborator

Adriani90 commented Jul 13, 2018

@Novalis7747, are you planning to raise a new pull request with your code?

@Brian1Gaff

This comment has been minimized.

Copy link

Brian1Gaff commented Jul 13, 2018

@Ednunp

This comment has been minimized.

Copy link

Ednunp commented Oct 14, 2018

Guys, I'm confused about the state of this issue. It is being reported as closed, but I for one am still seeing the issue with latest NVDA and office 365 update. IN the issue that it is linked to in the merge, it doesn't seem to clearly say whether the speech side of it has been resolved. Can someone comment either way please and if not resolved, are we any closer to resolving this with speech? not having the list of names complete is extremely annoying as it can take significantly longer to enter addresses in a large organisation.

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Oct 14, 2018

@Ednunp

This comment has been minimized.

Copy link

Ednunp commented Nov 21, 2018

Guys, is this ever going to get resolved? What is the current status? is MS looking into it, are NVDA looking into it, is anyone looking at it? it was opened in July and is by all accounts dead in the water. I want to stress again this is a significant issue when working in a large org with a lot of names and addresses. Currently the only workaround such as it is, is to enter a few letters of someones name, then hit enter, see if the name is correct, delete it if not, enter the same few letters of the name again, scroll down once and hit enter to try it all over again, repeat until you get the name right. This is truely ridiculous if you're entering 30 addresses in one of these fields. Can we please have a resolution plan or an indication of what's going on? Thanks, Ed.

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 21, 2018

Hi,

Since we haven't heard from the individual who did a pull request on this, I'll take this on.

Because of its high impact, I propose raising the priority of this one to P2 at least. If I do come up with a PR, it'll be based on beta unless @michaelDCurran says otherwise.

Thanks.

@Ednunp

This comment has been minimized.

Copy link

Ednunp commented Nov 21, 2018

Thanks Joseph apreciate this. I know of at least 3 people who wil be extremely happy once this is fixed.

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 21, 2018

Hi,

Further investigation says: yes, my initial hypothesis was right: I think we've been so complacent to a point where we cost someone his or her productivity.

Technical: the "To:" field and friends have way better UIA implementation. This is evidenced by the fact that:

  1. Narrator announces suggestions.
  2. Controller for event is raised.

Several solutions come to my mind:

  1. Tell NVDA that message address fields are good UIA controls (outlook.AppModule.isGoodUIAWindow). This then allows UIA to take over. Results announcement can be taken care of through NVDAObjects.UIA.SearchField.
  2. Add an IAccessible class for auto-complete fields and ask if it has a UIA implementation (UIA.kwargsFromSuper(dict)). For older Outlook versions, it should fail, but since 2016/Office 365 (which includes Outlook 2019), these fields should present a UIA version of themselves. This guarantees backward compatibility and is an application of UIA grid row strategy used a few years back to deal with Outlook message list.
  3. Same as second solution, but this time add suggestion announcement capability (NvDAObjects.behaviors.EditableTextWithSuggestions). Then while handling name change events, see if the underlying UIA implementation says anything about controller for array, and if it changes for some reason, raise suggestionsOpen/Closed events as needed. If chosen, this solution becomes an application of search field handler outside of Windows 10, also taking care of another issue about hiding suggestion sounds checkbox in object presentation panel outside of Windows 10.

I imagine all of these (and more possibilities) have downsides, especially when it comes to supporting Outlook 2007 and earlier:

  1. If first solution is chosen, we need to tell NVDA to use UIA for auto-complete fields on Outlook 2013 and above unless someone using Outlook 2010 says it is working as well.
  2. The second solution needs to catch possible exceptions.
  3. The third one is perhaps the hardest to implement but is a bit more flexible.

CC @michaelDCurran, @jcsteh, and @leonardder in case you have something to say.

Thanks.

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 21, 2018

Hi,

Technical: although name change event may seem promising, it won't cover certain situations. Also, the root culprit is the fact that state change event for auto-complete list item is not raised in some cases, but when using UIA, it works.

I'll compile a try build for you all to try (the code is based on latest master snapshot).

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 21, 2018

Hi,

Try build:

https://ci.appveyor.com/api/buildjobs/3967ub6tudlp64xa/artifacts/output%2Fnvda_snapshot_try-outlookAutoCompleteUIA-16299%2Ccd98032c.exe

Don't worry about error tones when entering address into "To:" field in Outlook.

Please test this build under the following combinations:

  1. Windows 10 and all of the following Outlook versions if possible: 2007, 2010, 2013, 2016, 2019, Office 365
  2. Windows 8.x and the following Outlook versions: 2007, 2010, 2013, 2016, Office 365 (Office 2019 does not support releases other than Windows 10).
  3. Windows 7 and Outlook 2007, 2010, 2013, 2016, Office 365

Notes:

  1. Windows releases earlier than 7 SP1 won't be supported.
  2. Outlook 2003 is end of life, therefore I recommend not supporting it.
  3. Support for Outlook 2007 is contingent on community feedback (especially from enterprises).
  4. Outlook 2010 support may or may not make the cut depending on whether UIA implementation used in this try build is even usable.
  5. Full support is expected for Outlook 2013 and later (including recent iterations of Office 365/2019).

Scenarios while entering address into "To:" field and friends:

  1. Type the name of a known contact and see if NVDA announces the top suggestion.
  2. As you type addresses, see if NVDA announces refined search results.
  3. While doing all these, see if NVDA plays search suggestion sound (to configure this, go to NVDA Menu/Settings/Object presentation/Play a sound when auto-suggestions appear checkbox must be checked).
  4. If you have a braille display, see if suggestions appear on the display.

For the first two scenarios, if one of them fails, let us know how to reproduce the problem (note that my computer has Office 365/2016 at the moment), along with Windows version and Outlook version in use. The third item shouldn't be a problem unless you've got Windows 7 or 8.x, and the fourth item is applicable only for those using a braille display.

Thanks.

@Adriani90

This comment has been minimized.

Copy link
Collaborator

Adriani90 commented Nov 21, 2018

Hey @josephsl, I've tested with MS Outlook 2016 and 365 with speech. Suggestion sound plays properly and NVDA announces the suggestions as expected. It works in to, cc and bcc.
Using Windows 10 1809 update.

I have noticed that the address in the field is recognized as button. This is also announced when navigating with left arrow to the address. It says "button" and "button end". This is not the case in NVDA 2018.4 Beta 1.

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 21, 2018

@govsta1981

This comment has been minimized.

Copy link

govsta1981 commented Nov 21, 2018

@Novalis7747

This comment has been minimized.

Copy link

Novalis7747 commented Nov 21, 2018

running Office 365 under Win 1809. the suggestion in the TO field works fine when typing. When arrowing down the suggested email addresses they are spoken correctly, but when up arrowing in the list it only says blank and arrowing doen again only brings me the value I typed in the address field.

@Novalis7747

This comment has been minimized.

Copy link

Novalis7747 commented Nov 21, 2018

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 21, 2018

@Novalis7747

This comment has been minimized.

Copy link

Novalis7747 commented Nov 21, 2018

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 21, 2018

@Novalis7747

This comment has been minimized.

Copy link

Novalis7747 commented Nov 21, 2018

@Novalis7747

This comment has been minimized.

Copy link

Novalis7747 commented Nov 22, 2018

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 22, 2018

@Novalis7747

This comment has been minimized.

Copy link

Novalis7747 commented Nov 22, 2018

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 22, 2018

@Ednunp

This comment has been minimized.

Copy link

Ednunp commented Nov 22, 2018

HI. Apologies I've not been able to test this yet as don't have my work laptop with me. Does this last comment mean we can't implement this code then? that the fix must remain disabled? Also another question if I may, how do I turn UIA on? is it a setting or on by default?

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 22, 2018

Hi,

Based on feedback so far, the new code, if even chosen, will work only on Office 2016 and beyond with some changes. Of course that could change based on your test results.

UIA is always enabled by default. One way to check is opening NVDA log (NVDA+F1) and checking UI Automation line at the top.

Thanks.

@Ednunp

This comment has been minimized.

Copy link

Ednunp commented Nov 22, 2018

It's a pain because office 2016 is where the problem is. No way of getting it to work out which version of Office is running and then working accordingly?

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 22, 2018

Hi,

Let's see if I can give you a slightly simplified picture:

As long as UIA is active, NVDA will use UIA support subsystem if a control says it is a UI Automation element. However, we (NV Access people and third-party developers) ran into incomplete UIA implementation for many controls in the past, thus we added several methods to block usage of UIA for certain controls (some of the most important UIA controls reside in Microsoft Office, and it was found that some of them had no useful UIA routines for NVDA to use). These days, Microsoft added enhancements for UIA controls in Office, and we're beginning to see the impact of this (especially in Office 2013 and beyond).

As for Office 2016 version detection, it is possible. However, given many Office 2016/2019 releases now (especially thanks to Office 365 release channels), it isn't enough to just tell NVDA to use UIA code if major version of Outlook is 16.0 (we also need to keep an eye on build numbers, too). In short, the answer to your question is "yes with caveats".

Thanks.

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 22, 2018

Hi,

Basically, if we do bring UIA, it'll be supported on more recent Office 2016/2019 builds. As noted on the pull request linked above, I asked Mick if he can take a look at feedback so far and do something about this issue.

Thanks.

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Nov 22, 2018

Hi,

Follow-up to my last comment: yes, the pull request referenced above will fix this issue for you once it shows up on alpha builds (after going through reviews of course).

By the way, I'd like to take back my word about complacency (apologies for not doing thorough research before making a swift conclusion).

Thanks.

@Ednunp

This comment has been minimized.

Copy link

Ednunp commented Jan 9, 2019

Hi guys, sorry to keep banging on about this but this is still not fixed in the alpha builds. Can someone please please review it and get it added? last update was in November.

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Jan 9, 2019

Hi,

At the moment NV Access people and several third-party developers are looking into this, including resolving problems we've encountered while testing the pull request and others. Sorry for the inconvenience.

Thanks.

@nvaccessAuto nvaccessAuto added this to the 2019.1 milestone Jan 15, 2019

@Ednunp

This comment has been minimized.

Copy link

Ednunp commented Jan 15, 2019

Hi. i'm confused why this has been closed, to my knowledge this has not been resolved yet. What does adding it to the milestone mean? Thanks.

@michaelDCurran

This comment has been minimized.

Copy link
Contributor

michaelDCurran commented Jan 15, 2019

@Novalis7747

This comment has been minimized.

Copy link

Novalis7747 commented Jan 15, 2019

@Ednunp

This comment has been minimized.

Copy link

Ednunp commented Jan 15, 2019

HI, I've just downloaded alpha 16555,5 and it is not working. Is there a new alpha version coming out with it in that I need to get later? thanks.

@leonardder

This comment has been minimized.

Copy link
Collaborator

leonardder commented Jan 15, 2019

@Ednunp

This comment has been minimized.

Copy link

Ednunp commented Jan 15, 2019

OH yeah baby, it's back :) thank you so much, you have just made my day about a million times easier when sending emails ;) Very happy this is now closed

@leonardder

This comment has been minimized.

Copy link
Collaborator

leonardder commented Jan 15, 2019

I"m afraid this regressed for me when #8919 was merged.

Steps to reproduce (note that this depends on your available contacts)

  1. Open a new message
  2. IN the to field, type "mick"

Expected

NVDA announces Michael Curran <e-mail address>

Actual result

NVDA stays silent

System details

Alpha 16557, Windows 10 1809, Outlook 16.0.11126.20196

@Ednunp

This comment has been minimized.

Copy link

Ednunp commented Jan 15, 2019

That is true actually. I was just about to send a comment. You can write in the to field and scroll up and down for the names, but it will not automatically announce. Also, if you hit control shift q to open up a new meeting invite, even scrolling up and down sometimes does not always work until the 2nd or 3rd press of the arrow keys. in general the functionality is a lot better however.

@zstanecic

This comment has been minimized.

Copy link
Contributor

zstanecic commented Jan 15, 2019

@josephsl

This comment has been minimized.

Copy link
Collaborator

josephsl commented Jan 15, 2019

@Novalis7747

This comment has been minimized.

Copy link

Novalis7747 commented Jan 15, 2019

@leonardder

This comment has been minimized.

Copy link
Collaborator

leonardder commented Feb 11, 2019

@leonardder commented on 15 Jan 2019, 15:20 CET:

I"m afraid this regressed for me when #8919 was merged.
Alpha 16557, Windows 10 1809, Outlook 16.0.11126.20196

I just tested this on another system with Windows 10 insider preview build 18334, Outlook 365 version 16.0.11231.20130, and I couldn't reproduce it. I will try it on my work system tomorrow and will report back. In any case, it is slightly confusing.

It is important to note that I'm now testing with an English build of Outlook, whereas the work system uses a Dutch build. May be that makes a difference, I will also check that out.

@leonardder

This comment has been minimized.

Copy link
Collaborator

leonardder commented Feb 12, 2019

I'm back on the system where I could reproduce this, which has Now Office 2016 home and business build 16.0.11231.20130, indeed, exactly the same build. Yet I have this issue with and without add-ons disabled on alpha-16743.
I've installed the English language pack, but this did not make a difference.

I'm reopening this to investigate the underlying cause of this. @michaelDCurran: do you have an idea where to start tracking this down?

@leonardder

This comment has been minimized.

Copy link
Collaborator

leonardder commented Feb 12, 2019

Further testing reveals the following. To test this, I added a tones.beep call to AutoCompleteListItem.event_stateChange:

  1. When entering "a" in the to field, I"m getting multiple suggestions. Up and down arrowing through the suggestions works just fine, however when the first suggestion pops up, it isn't read, may be because there's no state change on it? Anyhow, when there's only one suggestion (such as in the case of typing "mick"), the suggestion isn't read.
  2. When I enforce MSAA (i.e. with focus.appModule.isBadUIAWindow=lambda hwnd: True, stateChange is properly triggered on AutoCompleteListItem whenever a suggestion appears.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.