No item names for some listviews (64-bit)

[Robert-J-H]

Steps to reproduce:

Example app: Tortoise SVN

• invoke a context command such as SVN Update on a local repo.
• A window appears with the progress made so far.
• Tab to the list and try to navigate.
• one or multiple error sounds are issued.

Expected behavior:

The name of the focused item should be announced.

Actual behavior:

Only the state (e.g. selected) is announced

System configuration:

NVDA version next-15028,2754a2c8
Installed/Source/portable

Windows 10 (x64), Version 1803 (OS Build 17133.73)

Name and version of other software in use when reproducing the issue:
Tortoise SVN 1.9.7.27907 x64

Other questions:

Does the issue still occur after restarting your PC?

Yes

Have you tried any other versions of NVDA?

Not relevant

[dnz3d4c]

I am also very uncomfortable with this problem. This issue can be reproduced in both 2018.1 and master, next versions.

[Robert-J-H]

Background

The list in question is defined in:

NVDAObjects.IAccessible.sysListView32

The problematic class is:

ListItemWithoutColumnSupport(IAccessible)

and the method involved:

_getColumnContentRaw(self, index)

General Code Flow:

• virtual memory is allocated in the 64-bit process
• One part is for the receiving buffer
• the other part is for the LVITEM, a special structure that is sent with the message
• watchdog.cancellableSendMessage is called with msg = LVM_GETITEMTEXTW and the structure mentioned above
• the listview item now holds a pointer to the item name
• NVDA makes an attempt to read the buffer
• fail

Debug Result

Note that this is made with NVDA in source mode and two helper functions:

• a Repr class added to the LVITEM structure to show its members at each step
• Printout of the arguments that are passed to "winKernel.readProcessMemory"

The command (after selecting an item in the list view) is:

>>> focus._getColumnContentRaw( <index> )

1. The structure written to VM:

	mask 513 # set text and columns
	iItem 0
	iSubItem 0
	state 0
	stateMask 0
	pszText 57671680 # desired address of the receiving buffer
	cchTextMax 512 # length
	iImage 0
	lParam 0
	iIndent 0
	iGroupID 0
	cColumns 0
	puColumns 0
	piColFmt 0
	iGroup 0

2. the arguments passed to "ReadProcessMemory":

	2372 <type 'int'> # Process Handle
	57606144 <type 'int'> # Base Address
	<cparam 'P' (06AE9DB8)> <type 'CArgObject'> # the structure
	80 <type 'int'> # size in bytes
	None <type 'NoneType'>

3. The returned structure

	mask 1 # Text
	iItem 0
	iSubItem 0
	state 0
	stateMask 0
	pszText 669217843296 # didn't use our address!!
	cchTextMax 512 # length was accepted though
	iImage 0
	lParam 0
	iIndent 0
	iGroupID 0
	cColumns 0
	puColumns 0
	piColFmt 0
	iGroup 0

4. The arguments for reading the buffer:

	2372 <type 'int'>
	669217843296 <type 'long'> # Note that int is now long which is actually c_ulonglong
	<ctypes.c_wchar_Array_1 object at 0x06CDC490> <class 'ctypes.c_wchar_Array_1'>
	2 <type 'int'>
	None <type 'NoneType'>

5. The resulting error:

	Traceback (most recent call last):
	  File "<console>", line 1, in <module>
	  File "C:\nvda\source\NVDAObjects\IAccessible\sysListView32.py", line 341, in _getColumnContentRaw
		winKernel.readProcessMemory(processHandle,item.pszText,buffer,sizeof(buffer),None)
	  File "C:\nvda\source\winKernel.py", line 180, in readProcessMemory
		return kernel32.ReadProcessMemory(*args)
	ArgumentError: argument 2: <type 'exceptions.OverflowError'>: long int too long to convert

Conclusion:

• The listview control doesn't accept our pointer to the buffer and replaces it with a 64-bit pointer.
• From our 32-bit process, it is impossible to read the buffer.

Note that all other functions in this class that use the similar procedure work fine.
So can the column headers be read without problems.

There is nothing mentioned on MSDN with regard to the LVM_GETITEMTEXTW message.

However, this can be found under the similar LVM_GETITEMW "Remarks" section:

If the LVIF_TEXT flag is set in the mask member of the LVITEM structure, the pszText member must point to a valid buffer and the cchTextMax member must be set to the number of characters in that buffer.
Applications should not assume that the text will necessarily be placed in the specified buffer.
The control may instead change the pszText member of the structure to point to the new text, rather than place it in the buffer.

https://msdn.microsoft.com/en-us/library/windows/desktop/bb774953(v=vs.85).aspx

[leonardder]

I just closed #7874 in favour of this issue. @Robert-J-H: Thanks for providing a technical background.

[josephsl]

Hi,

Because we did have issues with this particular structure, and since we now have more detailed information, I recommend keeping this one in favor of others. Thanks. CC @jcsteh, @michaelDCurran

[Robert-J-H]

Thanks @leonardder

I should have investigated a bit deeper and commented on the open issue.
However, the problem is more general and affects probably all 64-bit applications that use this class (but I've only encountered Tortoise).
Note that there is another bug hidden in the method.
If you look at the arguments under 4.0, there is the integer "2" right above "None".
This is the buffer length, however, this can't be since we are using 512 characters, no matter what.
Notice how the variable len (bad name by the way) is used to create the buffer within the relevant code snippet:

len = watchdog.cancellableSendMessage(self.windowHandle,LVM_GETITEMTEXTW, (self.IAccessibleChildID-1), internalItem)
				if len:
					winKernel.readProcessMemory(processHandle,internalItem,byref(item),sizeof(self.LVITEM),None)
					buffer=create_unicode_buffer(len)
					winKernel.readProcessMemory(processHandle,item.pszText,buffer,sizeof(buffer),None)

For the buffer creation "item.cchTextMax" should be used instead.

[jcsteh]

LVM_GETITEMTEXT is documented to return the length of the text, which is why we do (and it's appropriate for us to) use it as the length of the buffer. The issue with 64 bit apps is that NVDA, as a 32 bit app, can't accept remote 64 bit memory addresses. There's another issue open somewhere to deal with this using our own in-process memory reading/writing helpers.

[Robert-J-H]

@jcsteh
Thanks James,
I've just realized that len is the amount of characters.
The _getHeaderRaw method does it curiously in the other way:

	res = watchdog.cancellableSendMessage(self.windowHandle,LVM_GETCOLUMNW, index, internalColumn)
	if res:
		winKernel.readProcessMemory(processHandle,internalColumn,byref(column),sizeof(self.LVCOLUMN),None)
		buffer=create_unicode_buffer(column.cchTextMax)
		winKernel.readProcessMemory(processHandle,column.pszText,buffer,sizeof(buffer),None)```

I agree with you, an external Helper function is probably needed.
I first contemplated replacing the text pointer with a callback (-1) but this would only move the problem to another place.
[Using text callbacks in ListView Controls](https://www.codeproject.com/Articles/70/Using-text-callbacks-in-ListView-Controls)

[Robert-J-H]

I've created pull request #8190 that addresses this issue, although not at a IPC level.
I think that it would be embarrassing if we recommended Tortoise SVN for translators without proper accessibility for the program.
I hope that some of you have other programs that exhibit the same bug and can be tested with the workaround.

[k-kolev1985]

I'm using NVDA version alpha-15693,b46b83ea and now the reading of the log list in TortoiseSVN is broken again. Was this commit one of the ones lost during the change of the testing/release process a few weeks ago?

[josephsl]

Hi, yes. Thanks.

[JulienCochuyt]

If I read the two above comments correctly, shouldn't the patch for PR #8190 be merged again or am I missing the point?

[ruifontes]

I don't understand why it was lost and never merged again...

[Robert-J-H]

Sure, it could have been merged some two years ago. Seems that NV Access wants another way but I'm not sure if it will work so easily within the C++ code.

…

On 13/05/2020, ruifontes ***@***.***> wrote: I don't understand why it was lost and never merged again... -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #8175 (comment)

[ScottChesworth]

Bumping this because it's also been identified as a show-stopper issue for Sibiac (an NVDA addon that provides access to DAW plugins in context of music production). Any news on that merge?

[JulienCochuyt]

cc @michaelDCurran, @feerrenrut

[Robert-J-H]

In which way does it tangle Sibiac? I use Azlo's addon daily, although be it with Python 2. For now, the author could just overwrite the SysListView32 class with this version and work from there.

[ScottChesworth]

Hey @Robert-J-H , my admittedly layman understanding of how this relates to Sibiac is that with some 64-bit plugins, this issue affects NVDA's handling of the FX chain list view, which is where Sibiac recognizes the plugin that's loaded and determines which overlay to use. CC-ing @AZSlow3, he's better placed to explain what's going on.

[Robert-J-H]

Hm, I see some problems in the FX Chain list box. It often stops to read the actual VST Plug-in (Same in the browser) and NVDA reports some out of memory error. Which is hardly the case with 32 GB. Anyway, Reapi has a function to return the current dll associated with the focused plug-in. This makes sense as plug-in instances can have any name if the user wishes so. Apart from the memory leak that I've mentioned above, the lists work nicely in Reaper, including name and number of entries. In contrast, the lack of those would mark the issue addressed in this thread. The MRO for a list item is (ignore the column review part, it's created by an addon): Python class mro: (<class 'NVDAObjects.Dynamic_ColumnsReview32ListItemIAccessible'>, <class 'globalPlugins.columnsReview.ColumnsReview32'>, <class 'globalPlugins.columnsReview.ColumnsReview'>, <class 'NVDAObjects.IAccessible.sysListView32.ListItem'>, <class 'NVDAObjects.behaviors.RowWithFakeNavigation'>, <class 'NVDAObjects.behaviors.RowWithoutCellObjects'>, <class 'NVDAObjects.IAccessible.sysListView32.ListItemWithoutColumnSupport'>, <class 'NVDAObjects.IAccessible.IAccessible'>, <class 'NVDAObjects.window.Window'>, <class 'NVDAObjects.NVDAObject'>, <class 'documentBase.TextContainerObject'>, <class 'baseObject.ScriptableObject'>, <class 'baseObject.AutoPropertyObject'>, <type 'object'>)

[jcsteh]

I'm confused. The FX Chain list reads just fine for me in 64 bit REAPER with NVDA...?

[AZSlow3]

There is no strait way to reproduce the problem in REAPER. So far I cood hit it with Toontrack EZ Drummer and XLN Addictive Drums VSTi only, in relatively busy project. These plug-ins fill all lower 32bit address space during GUI loading. Can be somehow JUCE related, so probably affect many plug-ins.

Synthetic test is in http://www.azslow.com/index.php?action=downloads;sa=downfile&id=77
atest.exe fills all 32bit space. Both continuously monitor REAPER, when it is running. That way I have checked the reason in REAPER is the same.

If I understand that right, IAccessible works using the same approach. So many things can break once 32bit addresses are reserved in the target. That is not a bug on 64bit application side, modern programs grow in size, VirtualAlloc is not unusual call inside applications and consumes lower address space by default. Not good for 32bit accessibility applications.

[leonardder]

Here is a try build that hopefully fixes this issue.

[CyrilleB79]

Good job! Thanks! It is working on my end.
Let's hope it is merged soon, I was expecting it.

[ScottChesworth]

Hey @leonardder, just tested the try build above in Reaper X64, and I'm still getting broken list views in the FX window once memory usage of the project exceeds 4GB. Is there anything I can provide to help diagnose? NVDA log etc?

[Robert-J-H]

@scott, yes do so. Is the error the same as in the normal built?

[leonardder]

@ScottChesworth A log would be pretty helpful. However, I'm pretty sure this error has a different background and requires more porting to C++.

[feerrenrut]

Hi @ScottChesworth could you please provide an NVDA log for this issue, or advise if it still occurs?

[ScottChesworth]

@feerrenrut, yep, still present in the public release of NVDA 2020.4. In the software I use most often (Reaper X64), any plug-in included in the "List1" control of Reaper's FX window in a project that uses over 4GB of RAM seems to be susceptible on my machine. It doesn't happen every time those conditions are met here, but it does happen often. I seem to be having a lucky day today, will pop a log in here next time I experience it. Thanks for the nudge, I should've done that ages ago.

[Robert-J-H]

Confirmed here. The list in the FX Browser is affected as well.

[jcsteh]

I've been seeing this in REAPER in all ListViews (FX chain list, FX add list, Actions list) with large projects too. I didn't grab a log when it was happening, though. :(

[ScottChesworth]

It's primarily in large projects when it happens here. Annoyingly it always seems to crop up when I'm against the clock. Will try harder to remember that I need to grab a log next time it occurs.

[jcsteh]

@AZSlow3's synthetic test case in #8175 (comment) makes it ridiculously easy to reproduce this. Thanks.

Log:

IO - inputCore.InputManager.executeGesture (21:02:55.303) - winInputHook (16888):
Input: kb(desktop):downArrow
ERROR - eventHandler.executeEvent (21:02:55.337) - MainThread (23152):
error executing event: gainFocus on <NVDAObjects.IAccessible.sysListView32.ListItem object at 0x06F87A90> with extra args of {}
Traceback (most recent call last):
  File "eventHandler.pyc", line 284, in executeEvent
  File "eventHandler.pyc", line 98, in __init__
  File "eventHandler.pyc", line 107, in next
  File "C:\Users\jamie\AppData\Roaming\nvda\addons\remote\globalPlugins\remoteClient\__init__.py", line 458, in event_gainFocus
    nextHandler()
  File "eventHandler.pyc", line 107, in next
  File "NVDAObjects\__init__.pyc", line 1173, in event_gainFocus
  File "NVDAObjects\behaviors.pyc", line 581, in reportFocus
  File "NVDAObjects\__init__.pyc", line 1028, in reportFocus
  File "speech\speech.pyc", line 555, in speakObject
  File "speech\speech.pyc", line 598, in getObjectSpeech
  File "speech\speech.pyc", line 450, in getObjectPropertiesSpeech
  File "baseObject.pyc", line 42, in __get__
  File "baseObject.pyc", line 146, in _getPropertyViaCache
  File "NVDAObjects\IAccessible\sysListView32.pyc", line 451, in _get_name
  File "NVDAObjects\IAccessible\sysListView32.pyc", line 357, in _getColumnLocation
  File "baseObject.pyc", line 42, in __get__
  File "baseObject.pyc", line 146, in _getPropertyViaCache
  File "NVDAObjects\IAccessible\sysListView32.pyc", line 220, in _get__columnOrderArray
  File "winKernel.pyc", line 278, in virtualAllocEx
OSError: [WinError 8] Not enough memory resources are available to process this command.

Fixing this will require moving _get__columnOrderArray to C++, as well as _getColumnLocationRaw and _getColumnHeaderRaw. You don't see exceptions here for the latter two, but that's only because the code never hits that point. I verified (with the Python console) that those throw the same exception too. There's also _getColumnImageIDRaw, but that only gets used by Outlook Express, so we can probably ignore that.

_get__columnOrderArray is going to be interesting because it needs to return an array of ints and raw RPC isn't great at handling that kind of thing. It's likely that a BSTR hack similar to displayModel_getWindowTextInRect will be necessary where we shove ints into a BSTR, casting them back and forth from wchars.

[jcsteh]

To further clarify, the difference between this issue and the earlier one is that with the earlier (now fixed) one, we can successfully allocate 32 bit memory in the remote process, but the control decides to give us back memory beyond 32 bit and we fail to read it. In this case, we can't allocate any 32 bit memory in the remote process at all because it's been exhausted.

[AZSlow3]

I can confirm that my current workaround (own 64bit proxy) overwrites columnOrderArray for the List and getColumnLocationRaw for ListItem. That way FX List in REAPER works correctly (that list has no Header).

[leonardder]

The following try build works with @AZSlow3's test case:
https://ci.appveyor.com/api/buildjobs/fi7wksdofpb8trcv/artifacts/output%2Fnvda_snapshot_pr13271-24521%2C6ac85491.exe

[seanbudd]

We've just merged new code affecting this issue to alpha and beta.
As 2022.1 release candidate is nearing, it is important that we retest this code thoroughly to ensure we haven't broken anything again.
Could users affected by this issue initially, please re-test the latest alpha build.

Based on this issue history, can anyone confirm that the latest alpha works as expected when using:

• Tortoise SVN
• REAPER
• FX Browser
• [-] Sibiac (partially fixed, not worse)
• IIS: Internet Information Services (IIS) Manager: error when moving through listview #11468

[seanbudd]

@leonardder can you confirm #11468 is still fixed on alpha?

[cary-rowen]

I installed Alpha and tested it with SVN, and the list items can be report fine.

[seanbudd]

Thanks @cary-rowen

[ScottChesworth]

I've tested the FX browser and the FX chain list in Reaper X64. List box controls seem solid here in projects that were using enough memory to consistently break before. Thanks much for the effort that went into this!
It's probably worth noting for any other Sibiac/LBL users following the thread that I don't think this fix is gonna be a magic bullet. I'm still experiencing occasional issues where I'm just hearing "window" instead of the appropriate overlay firing up, even when the list control is reading fine. So, probably some extra development needed from Sibiac/LBL. Either way, this fix is doing what it's meant to on my machine.

[seanbudd]

Thanks @ScottChesworth for the testing