RawLayerRequirement / Creating a TranslationLayer without running automagic

[NiklasBeierl]

Is your feature request related to a problem? Please describe.
In my usecase I do not have symbols for the linux kernel I want to investigate, but I do know the position of the dtb (as well as aslr and kaslr offsets ==0). I want to create various plugins that leverage volatilities address translation. However, since I do not have kernel symbols, I can't use automagic to set up the translation layer. Instead, I want to add a requirement to my plugin for specifying the dtb position and let the plugin instantiate the translation layer. But since I do not add a TranslationLayerRequirement to my Plugin it does not even get a "raw layer" when it runs. I tested running vol with -f pointing to a file but context.layers remained empty.

Describe the solution you'd like
I want a RawLayerRequirement for my plugin that lets me access the memory inputs (files/snapshots whatever) that where specified when calling my plugin.

Describe alternatives you've considered
I checked other plugins, but it seems like the only "LayerRequirement" is the TranslationLayerRequirement.
I did not find a suitable volatility/framework/configuration/requirements.py.
I have a hard time figuring out how I would implement this requirement myself and have volatility understand what I need from it.

[ikelos]

Several plugins run directly on a physical layer, for example the banners plugin ( https://github.com/volatilityfoundation/volatility3/blob/develop/volatility3/framework/plugins/banners.py#L22 ), but you need to make sure not to specify a list of architectures, otherwise it will fail if it can't put an architecture layer on top. You can either write your own automagic to find the DTB automatically, or you can make an IntRequirement so that the user can specify it as part of the plugin. I don't recall whether any stacking will happen (for example, loading a file format like lime), but that can be tested and fixed if it doesn't do what it's supposed to...

[NiklasBeierl]

Heyya, thanks for the reply,
I just tried specifying a Translationlayer without architectures and voila, I got a filelayer and a Intel32e layer. I will try a virsh dump next to see if that is being handled. But:
I do neither understand how volatility choose "Intel32e" nor how it configured the layer. Is there a piece of documentation I can read / attempt to write?
From what I read in the banner plugins code, it appears that it uses the Translationlayer Requirement only as a proxy to then access the underlying "memory_layer". But that strikes me as a "hacky" solution, no?
Here is what I understand happens:
The Banner Plugin requests a Translation / Intellayer. Because we don't specify architecture, volatility creates the layer without complaints, tho the Layer is missconfigured if the defaults don't allign with the memory sources. Then Bannerplugin just grabs the underlying layers and Ignores the Translationlayer.
Here is what I think should happen:
Bannerplugin should have a "Rawlayerrequirement" that handels anything below paging. (Lime, Crash, etc.)
Do you agree? If yes: where would that be implemented? Requirement + Stacker? Do we need a new automagic?

[ikelos]

Hiya, the difficulty with a RawLayerRequirement is what you mean by it? Do you mean "the last layer before a virtual layer" and what then happens when hypervisor virtualization kicks in (and you've got intel layers on intel layers on physical layers) or file formats (and you've got an intel layer on a physical layer on a physical layer). Where do swap layers sit on that (since volatility 3 supports a layer tree allowing multiple lower layers to be used by a single higher layer).

The example you provided doesn't quite work the way you suggest. The banners plugin doesn't know whether the automagic will have given it an intel layer or not (the stacker goes as high as it can). At the moment, it is somewhat hacky in that it checks whether it's a virtual intel layer and if so goes a layer down. It should do that recursively in case there's virtual layers on virtual layers, but again, we need clear semantics for what we need. Everyone kinda just "knows" what we mean, but codifying that for all possible instances is tricky. It also wasn't really an issue (as you can see, we've got plugins that can work on raw layers, like banners) but the area could do with fleshing out properly. If you can suggest appropriate semantics, it would help us figure out how to improve that area? It's not been very high on our list of things that need to be done, but we're definitely happy to improve it once the tough questions have been answered...

[NiklasBeierl]

Those are some very good points:
One definition I could think of is:
The highest layer I can get to without involving "address translation". [1]
I would expect the "rawlayer" to have things like file formats or compression abstracted away, but nothing more.

Specifics:
I am not to familiar with the intricacies of Swap. And I also never analyzed a memdump with swap. But I imagine that when using swap there will be two sources (files), one for the memory and one for swap and then there is a layer that "unifies" them into one address space? And I suspect that SWAP is handled via pagefaults so it only works with address translation. If these assumptions are correct then my "rawlayer" can not handle the unification of memory and swap. It can however handle swap and memory separately.

Now "hypervisor virtualization": I am not 100% sure, but I guess this refers to scenarios like: A memory snapshot of the hosts memory and you want to analyze a guest os running inside a vm on said host. Or a vm inside a vm, I heard that's possible with some hypervisors. In this case the terms "raw layer" or "memory layer" or "physical layer" do indeed get blurry, but I think my definition still gets us to a clear answer: Since going to the physical layer of the guest os requires address translating the physical layer of the host, the layer targeted by the definition is the "physical layer" of the host.

Targeting the physical layer of the guest seems somewhat exotic to me, but maybe its just my lack of experience. Perhaps we can provide facilities that can extract the physical layer of the guest to a new file? One can then run any plugins that have a RawLayerRequriement on the new file. Perhaps this can be achieved using the layerwriter plugin? Tho I am afraid that the layerwriterplugin would only be able to write the entire virtual address space of the host, thus constructing an enormous file. I guess one would have to create a memory snapshot of the guest-vm process? Can volatility already do that?
Perhaps that case is indeed exotic and we should not worry about it for now?

Last but not least:
The Name RawLayerRequirement is of course up for debate. Perhaps PhysicalLayerReq is better? It is physical in the "MMU" sense. But then again its not real-world physical if we look at a memdump from a vm, in that case its just a chunk of the hosts virtual memory . Maybe "PreVirtualLayer"?

[ikelos]

So I understand what you're getting at, but it doesn't quite fit. File formats (such as Lime) translate addresses from the physical address space into the file format's layout in order to provide (rudimentary) compression. Having a whole separate requirement for translation versus non-translation, which does actually translate, doesn't feel like the right way to go. I'd be more tempted to have options on the translation layer requirement (such as architecture) that have to be met appropriately. There was some thought of providing tags like physical and so on to layers, but again preventing inheritance and dealing with layers that can stack on top of themselves is very difficult. For (a contrived) example, a lime layer inside an AFF layer, that has an intel architecture on top of it that's virtualizing an arm architecture starts to get very sketchy.

It's generally a bad idea to add a new design but put off considering known problems. It sounds as though a TranslationLayerRequirement without a specific architecture fits your bill, so I'm more inclined to put off developing anything extra until we've thought everything through and considered the best way of organizing it all. I will bring it up at the next meeting though and see if we can get the ideas more in line. It may take a while but might as well get it into people's minds to mull over...

[NiklasBeierl]

I guess you are in a way better place to judge that than me. Thanks for hearing my considerations, tho. :)
I will go with a TranslationLayerRequirement without architectures now.
I also figured out why I got the "mysterious" Intel32e when adding the TranslationLayerRequirement to my plugin: I removed my symbols profile from the wrong place and Automagic just went ahead and figured it can add some more so it just did that. 🤦🏼
With the symbols file removed, the TranslationLayerRequirement gives me nothing but the "FileLayers" and "QemuSuspendLayers" I wanted to get anyways. So actually The banners plugin did exactly what I want to do, I was just to confused to see it.
I will see if I can find some useful words that can be added to the docs to prevent buggers like me from showing up in the future. :D

[ikelos]

Also, just to join this together into a single conversation, this seems very similar to #215

[ikelos]

Not a great deal of additional commentary going on here. At the moment, the recommended mechanism is still to set no architecture list, or to create the Translation layer, then check if it's an intel layer and descend if so. Something to watch out for is hypervisor guests whose physical space may actually be a mapped space (using the intel mapping mechanism). Everyone kind of knows intrinsically what they mean by physical memory, but it's pretty hard to define programmatically...

[github-actions]

This issue is stale because it has been open for 200 days with no activity.

[ikelos]

Still interested in doing this, it just needs thinking about...

[github-actions]

This issue is stale because it has been open for 200 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 60 days since being marked as stale.