The request could not be performed because of an I/O device error (4.0rc1)

[igorrogov]

Hello,

I've just tried the new 4.0rc1 release on my laptop, but it produced an error while trying to capture RAM.

00% 0x00100000 ....Failed to Read memory.
: The request could not be performed because of an I/O device error. 

I've tried all three methods, but all of them either produced the above error or sent my computer to BSOD.

This is Windows 10 Pro x64 with 2004 update (exact version is 10.0.19041.508). Full output is attached.

method_2_output.txt

[scudette]

Thanks for reporting - can you please capture kernel debug log using https://docs.microsoft.com/en-us/sysinternals/downloads/debugview - select capture kernel messages and try the acquisition again?

[ricoulele]

Hi
i have the same message than igorrogov (Windows 10 Pro x64 with 2004 update)

[vivianezw]

This is a successfully catched but real error. Though it never occurred on my system and I have 2004.
All three cases? (Impossible, I had said but ...)

I need the precise ntstatus error to know what went wrong.
Since this is a 'real' error, it should have a Dbgprint statement associated. (Error Dbgprints should never be disabled.)

1.) Can you use Dbgview[1], to directly see the message from the driver? You must do it elevated, with all verbose kernel capure settings enabled. (in menu), and put me the debug print output here.

Current diagnosis:
The driver loaded, but it never read anything. Something bad happened and it was handled for the non-BSOD case, that means it was a known error case.

@ edit:

Did you do this on a barebone machine?
Was this on a VM? If yes, were "guest services" drivers of any sorts installed on the VM (Virtualbox, etc)?

In case of a BSOD, the exception codes would also help a lot.

[1] https://docs.microsoft.com/en-us/sysinternals/downloads/debugview

[ricoulele]

sorry but now i have a blue screen of death....

[ricoulele]

and it is not on a VM machine

[vivianezw]

There is something really strange going on. My 2004 was a VM, not a barebone.

Did you try with or without memory compression?

Does anyone of you have a 1) Dbgview (does not help against BSOD), or better, 2), WinDbg[2]?

With Windbg, you load the crashdump (I think in c:\windows\memory.dmp per default) that was made when the OS crashed and launch a !analyze -v. That will give generally be enough output to nail down the error.

In case it gets really weird, I have to organize a win10 2004 barebone and must hope to reproduce it.

@ edit: already trying that.

[2] The most easiest way to get WinDbg is via the app store of Microsoft: type "Windbg preview" (Microsoft Corporation) to find it. That should do the job. Does not need a Microsoft online account to download it. Alternatively, it's in any SDK, and it's also in the WDK10 or WDK7600. Windbg can load the post-mortem crashdump that should be "c:\windows\memory.dmp" to tell what went wrong (using !analyze -v).

[vivianezw]

I cannot reproduce it on a Win10 2004 x64. It simply worked. But then again it that was a self-compiled driver.
The iospace method currently fails with systems that have a Hyper-V layer underneath them.
But at least PTE and the physical memory worked always and I just tested it.

@edit: collegue reported to run normally on a Win10 2004 x64, using both PTE and phsyical memory methods. Everything runs completely normal. (He self-compiled & testsigned from this git-cloned source.)

[vivianezw]

reproduced it. Happens only one real barebone machines, which makes it difficult to study. That needs to wait until tomorrow.

[igorrogov]

Hi guys,

Yes, this is a real machine, not a VM. But Hyper-V service is enabled. I use VMware and Windows Sandbox sometimes. I'm trying the 2nd method (PTE remapping). Please see the logs from DebugView attached.

This is the command that I'm running:
winpmem_mini_x64 -2 laptop.mem > output_method_2.txt

It works fine on the second machine which I haven't updated to 2004 yet. I'm going to update it soon and check if it works there.

output_method_2.txt
method_2_debug_view.LOG

[scudette]

Ok I think the error is related to VSM mode. When running in this mode, winpmem driver literally can not read the protected memory range and needs to break up the buffer and read around the holes.

We did have code to do this in here
https://github.com/Velocidex/c-aff4/blob/657dc28b805a55e44696125e6b3627e64dfc653a/tools/pmem/win_pmem.cc#L438

but this has not been replicated in the very simple mini imager.

Maybe I will prepare another release of the old aff4 imager to get around this issue. It would also be nice to be able to write an elf image so we can skip all the padding regions - a machine i tested with had a couple gb of padding.

[igorrogov]

Update: I have upgraded my older desktop PC to Win10 2004 and 4.0rc1 works fine there, no issues. 3.3rc2 doesn't work on this PC because of this issue - Velocidex/c-aff4#144

So, the only issue that remains is being able to capture RAM on a PC that has both Win10 2004 and VSM mode.

Maybe I will prepare another release of the old aff4 imager to get around this issue

Thanks, that would be great!

[vivianezw]

It was also fine on my gaming PC, which has no virtualization at all. :D

Approach: I think we might be able to catch the error and leave out the VSM-failing page, and return a sufficient amount of zero bytes instead.

I also need a better securing. There is VSM interference to be taken into account.

[vivianezw]

Here is a physical read one, before I fixed it:

FAULTING_SOURCE_LINE:  c:\src\driver\WinPmem\kernel\read.c

FAULTING_SOURCE_FILE:  c:\src\driver\WinPmem\kernel\read.c

FAULTING_SOURCE_LINE_NUMBER:  74

FAULTING_SOURCE_CODE:  
    72: 
    73:     if (NT_SUCCESS(NtStatus)) {
>   74:       RtlCopyMemory(buf, mapped_buffer + page_offset, to_read);
    75:       ZwUnmapViewOfSection((HANDLE)-1, mapped_buffer);
    76: 
    77:     } else {
    78:       DbgPrint("Failed to Map page at 0x%llX\n", offset.QuadPart);
    79:       RtlZeroMemory(buf, to_read);

I think I fixed all three now. I am testing myself and then will put it on my repo & PR...
You will have to use again my testsigned ones. ;)

[scudette]

I added a slow read path when buffers can not be read in #14 . Can you please test the rc2 binary with VSM enabled? You can find the binary in the releases.

It will now fall back to a slow read where it gets read errors from the driver. This will display an x in the progress instead of . (each character corresponds to 16mb buffers).

[vivianezw]

You don't need that, just see my patch.

[scudette]

Which patch?

[vivianezw]

well, since I have no write access to your repo, here : https://github.com/vivianezw/WinPmem I did not PR yet, because I need to terminate all tests on barebone first. With all methods.

[vivianezw]

My solution does not report the errors back.That's not good.

[igorrogov]

Yes, I can confirm that 4.0rc2 works fine on my laptop now (method 2). I can see those "x" symbols in the log. I assume this is where it was failing in rc1.

Thanks a lot!

[thomasp1392]

We tested 4.0rc2 on a physical device with Win10 Build 2004. Unfortunately, we still get an BSOD (Memory management) when we try to access memory. If it is helpful, we have a mini-dump and full-dump of the crash available.

[vivianezw]

Yes, that's exactly why I pushed my update.

Currently please use my testsigned drivers in the pull request!

For convenience, here: https://github.com/Velocidex/WinPmem/files/5386510/winpmem_testsigned_15_okt_2020_2.zip
pw: betatest.

I fixed that already! (But only in the testsigned drivers.)

[scudette]

Does this crash occur with the -2 flag? Using PTE mode?

When you say "trying to access memory" - are you trying to take an image or are you trying to use the driver in your own code?

If so - the only stable access method is the PTE mode which you will need to switch to using the required ioctrl. This is the only mode that captures the hyper V page faults correctly. Vivian added those same checks for the other modes in a later PR but that is not present in the signed driver.

[vivianezw]

I'm pretty sure he used the old signed drivers.

My patched current build is pretty invisible right now. (Hanging in a closed pull request.)

The physmem method ist also safe but only in my testsigned build. The iospace method is safe if no kernel debugger is attached (only in my latest testsigned build).

@scudette: pleeease put my testsigned nice driver to a more visible place. :-)