Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No item names for some listviews (64-bit) #8175

Closed
Robert-J-H opened this issue Apr 13, 2018 · 40 comments · Fixed by #11745 or #13271
Closed

No item names for some listviews (64-bit) #8175

Robert-J-H opened this issue Apr 13, 2018 · 40 comments · Fixed by #11745 or #13271
Labels
bug component/nvdaHelper p3 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority
Milestone

Comments

@Robert-J-H
Copy link
Contributor

Steps to reproduce:

Example app: Tortoise SVN

  • invoke a context command such as SVN Update on a local repo.
  • A window appears with the progress made so far.
  • Tab to the list and try to navigate.
  • one or multiple error sounds are issued.

Expected behavior:

The name of the focused item should be announced.

Actual behavior:

Only the state (e.g. selected) is announced

System configuration:

NVDA version next-15028,2754a2c8
Installed/Source/portable

Windows 10 (x64), Version 1803 (OS Build 17133.73)

Name and version of other software in use when reproducing the issue:
Tortoise SVN 1.9.7.27907 x64

Other questions:

Does the issue still occur after restarting your PC?

Yes

Have you tried any other versions of NVDA?

Not relevant

@dnz3d4c
Copy link

dnz3d4c commented Apr 13, 2018

I am also very uncomfortable with this problem. This issue can be reproduced in both 2018.1 and master, next versions.

@Robert-J-H
Copy link
Contributor Author

Background

The list in question is defined in:

NVDAObjects.IAccessible.sysListView32

The problematic class is:

ListItemWithoutColumnSupport(IAccessible)

and the method involved:

_getColumnContentRaw(self, index)

General Code Flow:

  • virtual memory is allocated in the 64-bit process
  • One part is for the receiving buffer
  • the other part is for the LVITEM, a special structure that is sent with the message
  • watchdog.cancellableSendMessage is called with msg = LVM_GETITEMTEXTW and the structure mentioned above
  • the listview item now holds a pointer to the item name
  • NVDA makes an attempt to read the buffer
  • fail

Debug Result

Note that this is made with NVDA in source mode and two helper functions:

  • a Repr class added to the LVITEM structure to show its members at each step
  • Printout of the arguments that are passed to "winKernel.readProcessMemory"

The command (after selecting an item in the list view) is:

>>> focus._getColumnContentRaw( <index> )

1. The structure written to VM:

	mask 513 # set text and columns
	iItem 0
	iSubItem 0
	state 0
	stateMask 0
	pszText 57671680 # desired address of the receiving buffer
	cchTextMax 512 # length
	iImage 0
	lParam 0
	iIndent 0
	iGroupID 0
	cColumns 0
	puColumns 0
	piColFmt 0
	iGroup 0

2. the arguments passed to "ReadProcessMemory":

	2372 <type 'int'> # Process Handle
	57606144 <type 'int'> # Base Address
	<cparam 'P' (06AE9DB8)> <type 'CArgObject'> # the structure
	80 <type 'int'> # size in bytes
	None <type 'NoneType'>

3. The returned structure

	mask 1 # Text
	iItem 0
	iSubItem 0
	state 0
	stateMask 0
	pszText 669217843296 # didn't use our address!!
	cchTextMax 512 # length was accepted though
	iImage 0
	lParam 0
	iIndent 0
	iGroupID 0
	cColumns 0
	puColumns 0
	piColFmt 0
	iGroup 0

4. The arguments for reading the buffer:

	2372 <type 'int'>
	669217843296 <type 'long'> # Note that int is now long which is actually c_ulonglong
	<ctypes.c_wchar_Array_1 object at 0x06CDC490> <class 'ctypes.c_wchar_Array_1'>
	2 <type 'int'>
	None <type 'NoneType'>

5. The resulting error:

	Traceback (most recent call last):
	  File "<console>", line 1, in <module>
	  File "C:\nvda\source\NVDAObjects\IAccessible\sysListView32.py", line 341, in _getColumnContentRaw
		winKernel.readProcessMemory(processHandle,item.pszText,buffer,sizeof(buffer),None)
	  File "C:\nvda\source\winKernel.py", line 180, in readProcessMemory
		return kernel32.ReadProcessMemory(*args)
	ArgumentError: argument 2: <type 'exceptions.OverflowError'>: long int too long to convert

Conclusion:

  • The listview control doesn't accept our pointer to the buffer and replaces it with a 64-bit pointer.
  • From our 32-bit process, it is impossible to read the buffer.

Note that all other functions in this class that use the similar procedure work fine.
So can the column headers be read without problems.

There is nothing mentioned on MSDN with regard to the LVM_GETITEMTEXTW message.

However, this can be found under the similar LVM_GETITEMW "Remarks" section:

If the LVIF_TEXT flag is set in the mask member of the LVITEM structure, the pszText member must point to a valid buffer and the cchTextMax member must be set to the number of characters in that buffer.
Applications should not assume that the text will necessarily be placed in the specified buffer.
The control may instead change the pszText member of the structure to point to the new text, rather than place it in the buffer.

https://msdn.microsoft.com/en-us/library/windows/desktop/bb774953(v=vs.85).aspx

@leonardder
Copy link
Collaborator

I just closed #7874 in favour of this issue. @Robert-J-H: Thanks for providing a technical background.

@josephsl
Copy link
Collaborator

Hi,

Because we did have issues with this particular structure, and since we now have more detailed information, I recommend keeping this one in favor of others. Thanks. CC @jcsteh, @michaelDCurran

@Robert-J-H
Copy link
Contributor Author

Thanks @leonardder

I should have investigated a bit deeper and commented on the open issue.
However, the problem is more general and affects probably all 64-bit applications that use this class (but I've only encountered Tortoise).
Note that there is another bug hidden in the method.
If you look at the arguments under 4.0, there is the integer "2" right above "None".
This is the buffer length, however, this can't be since we are using 512 characters, no matter what.
Notice how the variable len (bad name by the way) is used to create the buffer within the relevant code snippet:

len = watchdog.cancellableSendMessage(self.windowHandle,LVM_GETITEMTEXTW, (self.IAccessibleChildID-1), internalItem)
				if len:
					winKernel.readProcessMemory(processHandle,internalItem,byref(item),sizeof(self.LVITEM),None)
					buffer=create_unicode_buffer(len)
					winKernel.readProcessMemory(processHandle,item.pszText,buffer,sizeof(buffer),None)

For the buffer creation "item.cchTextMax" should be used instead.

@jcsteh
Copy link
Contributor

jcsteh commented Apr 14, 2018 via email

@Robert-J-H
Copy link
Contributor Author

@jcsteh
Thanks James,
I've just realized that len is the amount of characters.
The _getHeaderRaw method does it curiously in the other way:

	res = watchdog.cancellableSendMessage(self.windowHandle,LVM_GETCOLUMNW, index, internalColumn)
	if res:
		winKernel.readProcessMemory(processHandle,internalColumn,byref(column),sizeof(self.LVCOLUMN),None)
		buffer=create_unicode_buffer(column.cchTextMax)
		winKernel.readProcessMemory(processHandle,column.pszText,buffer,sizeof(buffer),None)```

I agree with you, an external Helper function is probably needed.
I first contemplated replacing the text pointer with a callback (-1) but this would only move the problem to another place.
[Using text callbacks in ListView Controls](https://www.codeproject.com/Articles/70/Using-text-callbacks-in-ListView-Controls)

Robert-J-H added a commit to Robert-J-H/nvda that referenced this issue Apr 18, 2018
Item text in a list view can't be retrieved if the returned pointer is 64 bits long.
Therefore, return the displayText within the column boundaries instead.
@Robert-J-H
Copy link
Contributor Author

I've created pull request #8190 that addresses this issue, although not at a IPC level.
I think that it would be embarrassing if we recommended Tortoise SVN for translators without proper accessibility for the program.
I hope that some of you have other programs that exhibit the same bug and can be tested with the workaround.

@k-kolev1985
Copy link

I'm using NVDA version alpha-15693,b46b83ea and now the reading of the log list in TortoiseSVN is broken again. Was this commit one of the ones lost during the change of the testing/release process a few weeks ago?

@josephsl
Copy link
Collaborator

josephsl commented Jul 30, 2018 via email

@JulienCochuyt
Copy link
Collaborator

If I read the two above comments correctly, shouldn't the patch for PR #8190 be merged again or am I missing the point?

@ruifontes
Copy link
Contributor

I don't understand why it was lost and never merged again...

@Robert-J-H
Copy link
Contributor Author

Robert-J-H commented May 13, 2020 via email

@ScottChesworth
Copy link

Bumping this because it's also been identified as a show-stopper issue for Sibiac (an NVDA addon that provides access to DAW plugins in context of music production). Any news on that merge?

@JulienCochuyt
Copy link
Collaborator

cc @michaelDCurran, @feerrenrut

@Robert-J-H
Copy link
Contributor Author

Robert-J-H commented Jun 3, 2020 via email

@ScottChesworth
Copy link

Hey @Robert-J-H , my admittedly layman understanding of how this relates to Sibiac is that with some 64-bit plugins, this issue affects NVDA's handling of the FX chain list view, which is where Sibiac recognizes the plugin that's loaded and determines which overlay to use. CC-ing @AZSlow3, he's better placed to explain what's going on.

@Robert-J-H
Copy link
Contributor Author

Robert-J-H commented Jun 4, 2020 via email

@jcsteh
Copy link
Contributor

jcsteh commented Jun 4, 2020 via email

@AZSlow3
Copy link

AZSlow3 commented Jun 4, 2020

There is no strait way to reproduce the problem in REAPER. So far I cood hit it with Toontrack EZ Drummer and XLN Addictive Drums VSTi only, in relatively busy project. These plug-ins fill all lower 32bit address space during GUI loading. Can be somehow JUCE related, so probably affect many plug-ins.

Synthetic test is in http://www.azslow.com/index.php?action=downloads;sa=downfile&id=77
atest.exe fills all 32bit space. Both continuously monitor REAPER, when it is running. That way I have checked the reason in REAPER is the same.

If I understand that right, IAccessible works using the same approach. So many things can break once 32bit addresses are reserved in the target. That is not a bug on 64bit application side, modern programs grow in size, VirtualAlloc is not unusual call inside applications and consumes lower address space by default. Not good for 32bit accessibility applications.

@leonardder
Copy link
Collaborator

Here is a try build that hopefully fixes this issue.

@CyrilleB79
Copy link
Collaborator

Good job! Thanks! It is working on my end.
Let's hope it is merged soon, I was expecting it.

@nvaccessAuto nvaccessAuto added this to the 2020.4 milestone Oct 12, 2020
@ScottChesworth
Copy link

Hey @leonardder, just tested the try build above in Reaper X64, and I'm still getting broken list views in the FX window once memory usage of the project exceeds 4GB. Is there anything I can provide to help diagnose? NVDA log etc?

@Robert-J-H
Copy link
Contributor Author

Robert-J-H commented Oct 31, 2020 via email

@leonardder
Copy link
Collaborator

@ScottChesworth A log would be pretty helpful. However, I'm pretty sure this error has a different background and requires more porting to C++.

@leonardder leonardder reopened this Nov 2, 2020
@feerrenrut feerrenrut removed this from the 2020.4 milestone Nov 24, 2020
@feerrenrut
Copy link
Contributor

Hi @ScottChesworth could you please provide an NVDA log for this issue, or advise if it still occurs?

@ScottChesworth
Copy link

@feerrenrut, yep, still present in the public release of NVDA 2020.4. In the software I use most often (Reaper X64), any plug-in included in the "List1" control of Reaper's FX window in a project that uses over 4GB of RAM seems to be susceptible on my machine. It doesn't happen every time those conditions are met here, but it does happen often. I seem to be having a lucky day today, will pop a log in here next time I experience it. Thanks for the nudge, I should've done that ages ago.

@Robert-J-H
Copy link
Contributor Author

Robert-J-H commented Apr 28, 2021 via email

@jcsteh
Copy link
Contributor

jcsteh commented Nov 5, 2021

I've been seeing this in REAPER in all ListViews (FX chain list, FX add list, Actions list) with large projects too. I didn't grab a log when it was happening, though. :(

@ScottChesworth
Copy link

It's primarily in large projects when it happens here. Annoyingly it always seems to crop up when I'm against the clock. Will try harder to remember that I need to grab a log next time it occurs.

@jcsteh
Copy link
Contributor

jcsteh commented Nov 5, 2021

@AZSlow3's synthetic test case in #8175 (comment) makes it ridiculously easy to reproduce this. Thanks.

Log:

IO - inputCore.InputManager.executeGesture (21:02:55.303) - winInputHook (16888):
Input: kb(desktop):downArrow
ERROR - eventHandler.executeEvent (21:02:55.337) - MainThread (23152):
error executing event: gainFocus on <NVDAObjects.IAccessible.sysListView32.ListItem object at 0x06F87A90> with extra args of {}
Traceback (most recent call last):
  File "eventHandler.pyc", line 284, in executeEvent
  File "eventHandler.pyc", line 98, in __init__
  File "eventHandler.pyc", line 107, in next
  File "C:\Users\jamie\AppData\Roaming\nvda\addons\remote\globalPlugins\remoteClient\__init__.py", line 458, in event_gainFocus
    nextHandler()
  File "eventHandler.pyc", line 107, in next
  File "NVDAObjects\__init__.pyc", line 1173, in event_gainFocus
  File "NVDAObjects\behaviors.pyc", line 581, in reportFocus
  File "NVDAObjects\__init__.pyc", line 1028, in reportFocus
  File "speech\speech.pyc", line 555, in speakObject
  File "speech\speech.pyc", line 598, in getObjectSpeech
  File "speech\speech.pyc", line 450, in getObjectPropertiesSpeech
  File "baseObject.pyc", line 42, in __get__
  File "baseObject.pyc", line 146, in _getPropertyViaCache
  File "NVDAObjects\IAccessible\sysListView32.pyc", line 451, in _get_name
  File "NVDAObjects\IAccessible\sysListView32.pyc", line 357, in _getColumnLocation
  File "baseObject.pyc", line 42, in __get__
  File "baseObject.pyc", line 146, in _getPropertyViaCache
  File "NVDAObjects\IAccessible\sysListView32.pyc", line 220, in _get__columnOrderArray
  File "winKernel.pyc", line 278, in virtualAllocEx
OSError: [WinError 8] Not enough memory resources are available to process this command.

Fixing this will require moving _get__columnOrderArray to C++, as well as _getColumnLocationRaw and _getColumnHeaderRaw. You don't see exceptions here for the latter two, but that's only because the code never hits that point. I verified (with the Python console) that those throw the same exception too. There's also _getColumnImageIDRaw, but that only gets used by Outlook Express, so we can probably ignore that.

_get__columnOrderArray is going to be interesting because it needs to return an array of ints and raw RPC isn't great at handling that kind of thing. It's likely that a BSTR hack similar to displayModel_getWindowTextInRect will be necessary where we shove ints into a BSTR, casting them back and forth from wchars.

@jcsteh
Copy link
Contributor

jcsteh commented Nov 5, 2021

To further clarify, the difference between this issue and the earlier one is that with the earlier (now fixed) one, we can successfully allocate 32 bit memory in the remote process, but the control decides to give us back memory beyond 32 bit and we fail to read it. In this case, we can't allocate any 32 bit memory in the remote process at all because it's been exhausted.

@AZSlow3
Copy link

AZSlow3 commented Nov 10, 2021

I can confirm that my current workaround (own 64bit proxy) overwrites columnOrderArray for the List and getColumnLocationRaw for ListItem. That way FX List in REAPER works correctly (that list has no Header).

@leonardder
Copy link
Collaborator

feerrenrut pushed a commit that referenced this issue Feb 16, 2022
…ired (PR #13271)

Fixes #8175
Follow up of #11745
Fixes regression from #11469, #9873

Summary:
Getting list item content on 64 bit systems sometimes failed.

Description of change:
Based on #11745, fetching text content in-process fixed some but not all issues.
The change adds:
- Location fetching
- Column header text fetching
- Column order array fetching

In short, everything that required allocating memory within a process using VirtualAlloc
@nvaccessAuto nvaccessAuto added this to the 2022.1 milestone Feb 16, 2022
@seanbudd
Copy link
Member

seanbudd commented Mar 31, 2022

We've just merged new code affecting this issue to alpha and beta.
As 2022.1 release candidate is nearing, it is important that we retest this code thoroughly to ensure we haven't broken anything again.
Could users affected by this issue initially, please re-test the latest alpha build.

Based on this issue history, can anyone confirm that the latest alpha works as expected when using:

@seanbudd
Copy link
Member

@leonardder can you confirm #11468 is still fixed on alpha?

@cary-rowen
Copy link
Contributor

I installed Alpha and tested it with SVN, and the list items can be report fine.

@seanbudd
Copy link
Member

seanbudd commented Apr 1, 2022

Thanks @cary-rowen

@ScottChesworth
Copy link

I've tested the FX browser and the FX chain list in Reaper X64. List box controls seem solid here in projects that were using enough memory to consistently break before. Thanks much for the effort that went into this!
It's probably worth noting for any other Sibiac/LBL users following the thread that I don't think this fix is gonna be a magic bullet. I'm still experiencing occasional issues where I'm just hearing "window" instead of the appropriate overlay firing up, even when the list control is reading fine. So, probably some extra development needed from Sibiac/LBL. Either way, this fix is doing what it's meant to on my machine.

@seanbudd
Copy link
Member

seanbudd commented May 3, 2022

Thanks @ScottChesworth for the testing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment