Layerwriter: No output, debugger triggered.

[NiklasBeierl]

I was trying to run volatility on a dump from a kvm, created with virsh dump. (Size: 551mb)
Even with --parallelism processes it took somewhere around 20 minutes and it prompted me with the Layerwriter warning:
"Reads to this layer are slow, it's recommended to use the layerwriter plugin once to produce a raw file"
I couldn't really find any very explicit documentation, but I figured layerwriter would "convert" my dump into something that can be processed faster by volatility, so I ran:
vol --parallelism processes -f memdump layerwriter.LayerWriter
Which after quite some time, dropped me to the python debugger. That left my quite baffled, but I peeked into the code,
discovered these two lines, removed them and reran. (I guess someone forgot to clean that up before committing?)
This time, tho, I was left with no output at all. It ate all of my CPU for 20 minutes and then stopped. :/ As far as I can tell no file was written.

(volvenv) ~/code/bathesis vol --parallelism processes -f memdump layerwriter.LayerWriter
Volatility 3 Framework 2.0.0-beta.1
WARNING  volatility.framework.interfaces.automagic: Reads to this layer are slow, it's recommended to use the layerwriter plugin once to produce a raw file
Progress:  100.00               PDB scanning finished                             
Status

(Here I would like to note that prompting the user to run the layerwriter plugin while he is running that very plugin seems almost like mockery. 😄 )

I guess self.config['layers'] ends up being empty, which would mean that there where no mapped layers? I do not know what exactly that means and whether it is a "non-bug" edge case. If it is, it would make sense to at least let the user know that we can not write any file for him. Maybe we could also detect this case earlier and safe him a bunch of runtime?

(Edit for searches:)
Analyzing QEMU or KVM virsh dumps in volatility.

[ikelos]

So the parallelism flags have a tendency to be slightly slower, which is why they're disabled by default. You're right the pdb lines shouldn't have been there (and have been removed for the upcoming release, see commit f75e779), thanks for pointing that out. 5:)

If lists is not populated, it should look for intel layers and try to dump them, if no intel layers are found, no layers will be dumped. We could change that to dumping all layers if you think it would be useful, but I suspect it's probably not what anyone ever wnats? You should run layerwriter with --list which will list you the names of layers that can be dumped, and it can then be rerun with --layers <layername>.

It's unfortunate that qemu files take so long to process, but unfortunately scanning such layers or doing lots of reads is very slow (hence the warning). It's not really feasible to determine which plugin is being run, so you'll get the advice to use layerwriter even when trying to use layerwriter.

I've looked into telling the user which layer needs writing out in the slow warning, but unfortunately the warning occurs during the stacking phase (to avoid the warning showing up too often), and the stacking phase works by trying to create a layer and if it succeeds, only the configuration is passed back, and the layer is recreated (potentially with a different name). As such, it's not really possible to provide the information there, and ensuring the warning is only fired once on the reconstructed layer would be difficult (since it would fire during the stacking attempt, and thus confuse users). As such, I'm not entirely sure how to fix this. I'll have more of a think, but it may not be possible to reasonably solve this problem I'm afraid... 5:S

[NiklasBeierl]

Taking the user-interface / warnings stuff stuff aside, I do not (yet) know enough to argue with you on this one. 😄

I ran the linux.pslist.PsList plugin on my original dump and it gave me a reasonable list of processes. But it took quite some time.
I ran layerwriter with --list and got the following layer names:

base_layer
memory_layer
primary

Since I don't really know what I am doing I decided to write all layers. 😄 That ended well for base_layer and memory_layer, but primary exploded in size, so I aborted. (I presume it tried to "write out" the entire virtual address space? (2^64) byte?)
As I almost expected base_layer.raw and the original memory dump are identical according to cksum.
Running PsList and the volshell on memory_layer.raw is quite fast however!
So I guess vol -f mydump layerwriter.LayerWriter --layers memory_layer is the way to "convert" my virsh dump into something "more digestable" for volatility. I am happy with that. :)

But this also took a lot of trial and error, is there something we can do to make this easier for the next person? I guess I could add my insights to the docs somewhere? I am just not sure of the "canonical" way to figure out what layer needs to be written. (Passed to --layers)

It seems like here we are trying to choose a layer automatically, It just didn't work in my case? I also cant tell why. Shouldn't my memory_layer have been added to the list?

[NiklasBeierl]

I think I found the culprit!

I removed aforementioned debugger lines and reran the layerwriter plugin without any arguments and set breakpoints like so:

The first two breakpoints where never triggered! This is important!
Arriving at the third breakpoint, I verified my arguments in the debugger console:

>>> import sys
>>> sys.argv
['/home/niklas/code/volatility3/vol.py', '-f', '../thesis/memdump', 'layerwriter.LayerWriter']

And then I checked self.config["layers"]

>>> self.config["layers"]
[]

AFAIK an empty list is falsy, but its not None. Sanity check:

>>> self.config["layers"] is None
False

So even though i did not specify a --layers parameter, self.config["layers"] ended up being an empty list instead of None!
This is easy enough to fix, instead of if self.config["layers"] is None: we could just use if not self.config["layers"]: because:

>>> not self.config["layers"]
True

BUT: The requirements for the LayerWriter specify the default value for "layers" as None. So maybe the error actually lies within the argument parsing somehow?

To be 100% sure I also checked self.config["layers"] here. And it was also an empty list. So either the arguments parsing is ignoring the default for the ListRequirement, or the LayerWriter makes incorrect use of the default parameter. 😄

[ikelos]

Ah, good find. It appears as though ArgParser's list type arguments ignore the default (I assume only if it's not a list), so we need to fix up the layers code to check not just for None for but an empty list. I've merged the fix in and added it as a bugfix to the upcoming release. Thanks for figuring it out! 5:)