Windows XP: Symbol type not in nt_symbols1 SymbolTable: _ETHREAD`

[Wenzel]

Describe the bug
Volatility3 failed to run the SSDT plugin on a windows XP dump
Automagic exception occurred: ValueError: Symbol type not in nt_symbols1 SymbolTable: _ETHREAD

Context
Volatility Version: master
Operating System: Ubuntu 20.04
Python Version: 3.8
Suspected Operating System: Windows XP
Command: vol -f winxp-4da5322a-f55c-4124-92e0-a4520647bb36.dump windows.ssdt.SSDT

To Reproduce
Steps to reproduce the behavior:

1. Download windows xp dump file
2. run vol -f winxp-4da5322a-f55c-4124-92e0-a4520647bb36.dump windows.ssdt.SSDT

Expected behavior
SSDT table should have been printed

Screenshots

Additional information
Gist of the volatility log file

Thanks !

[ikelos]

My guess is that the volatility automagic may not have been able to find the kernel's PDB to generate the appropriate symbols, but I'll know more once the download request goes through... 5:)

[Wenzel]

Sorry, I thought the link could be shared.

Updated link to winxp dump file on google drive.

Update 2: The link can now be opened by anyone.

[ikelos]

Hmmm, that image seems to work fine for me? You might want to try clearing out your cache (/path/to/home/.cache/volatility3), and if that still doesn't work, check that you don't have any unusual symbol tables present in your symbols directories (also, running vol.py -vvvv should give you more information about what's going on and where/why it might be going wrong?)

winxp-4da5322a-f55c-4124-92e0-a4520647bb36.json.txt

I've included a file that is the symbol table to the image you linked (it can be downloaded and put at volatility/symbols/ntkrpamp.pdb/C40DD53A8D3D4AE3A24CE6BE866649C9-1.json, named that exactly). That should already have been automatically generated and created when the image is first analyzed though? Please let us know whether you have any success or not, so we know when we can close this issue... 5:)

[Wenzel]

Hi @ikelos, thanks for the feedback.

I tried removing my volatility cache, didn't work.
I tried running volatility with -vvvv, but I didn't have more debug infos

I downloaded your JSON profile in the ntkrpamp.pdb directory, and this solution worked:

DEBUG    volatility.framework.automagic.pdbscan: Using symbol library: ntkrpamp.pdb/C40DD53A8D3D4AE3A24CE6BE866649C9-1
INFO     volatility.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility.schemas: All validations will report success, even with malformed input
DEBUG    volatility.framework.automagic.pdbscan: Setting kernel_virtual_offset to 0x804d7000

Index	Address	Module	Symbol

0	0x805a3054	ntoskrnl	NtAcceptConnectPort
1	0x805ef2d8	ntoskrnl	NtAccessCheck
2	0x805f2b0e	ntoskrnl	NtAccessCheckAndAuditAlarm
3	0x805ef30a	ntoskrnl	NtAccessCheckByType

My compressed JSON profile is really different than yours:

C40DD53A8D3D4AE3A24CE6BE866649C9-1.txt

So I have no clue why the generated profile is not working for me ? maybe incomplete ?

Thanks !

[ikelos]

Hmmmm, very strange? Was it just the profile (as in, if you dropped my profile in place of yours, would it work?). Also, are you running the latest git release of volatility3, there's been a fair bit of development work since the first beta?

Hmmm, so the metadata suggests mine wasn't built by volatility, but by another tool we used to use before hand (which was used for all the profiles in windows.zip):

 "metadata": {
    "format": "6.0.0", 
    "producer": {
      "datetime": "2019-01-16T13:15:03.272000", 
      "name": "dia2json", 
      "version": "0.2.0"
    }, 
    "windows": {
      "pdb": {
        "GUID": "C40DD53A8D3D4AE3A24CE6BE866649C9", 
        "age": 1, 
        "database": "ntkrpamp.pdb", 
        "machine_type": 0
      }
    }
  },

It's for a different database and an earlier age than the one you generated:

  "metadata": {
    "format": "6.1.0",
    "producer": {
      "datetime": "2020-06-30T21:41:32.598724",
      "name": "volatility3",
      "version": "1.0.0-beta.1"
    },
    "windows": {
      "pdb": {
        "GUID": "C40DD53A8D3D4AE3A24CE6BE866649C9",
        "age": 2,
        "database": "ntkrnlmp.pdb",
        "machine_type": 332
      }
    }
  },

But essentially yours didn't generate any types (no base_types or user_types) which will be why it failed. With the image you sent I can start to do some analysis and see what's going on at least. I'll let you know if I find anything...

[ikelos]

Hmmm, so it appears as though the PDB files that Microsoft offers (even for the earlier PDB of age 1 and name ntkrpamp.pdb) is reduced and contains only symbols and not types? 5:S Which is strange that they'd change files they've previously provided? @npetroni Have we seen examples of this happening before? It might mean that windows.zip archive is more useful than we expected?

[Wenzel]

@ikelos I created a Dockerfile for you to repro the bug;

FROM ubuntu:20.04

RUN apt-get update && apt-get install -y git python3 python3-dev python3-venv
RUN git clone https://github.com/volatilityfoundation/volatility3
RUN python3 -m venv venv
RUN . venv/bin/activate && pip install wheel && pip install /volatility3
ENTRYPOINT . venv/bin/activate && vol -f /image -vvvv windows.ssdt.SSDT

docker build -t test_volatility .
docker run -ti --rm=true -v ~/Projets/oswatcher/winxp-4da5322a-f55c-4124-92e0-a4520647bb36.dump:/image test_volatility

[ikelos]

For reference, here is the most recent PDB file available from Microsoft:

issue242.zip

[npetroni]

@npetroni Have we seen examples of this happening before? It might mean that windows.zip archive is more useful than we expected?

IIRC, we saw this once before for a similarly old PDB.

I verified that the DIA library does not seem to find types in the PDB in the attachment.

[thall63]

For reference, here is the most recent PDB file available from Microsoft:

issue242.zip

What file will this replace?

[ikelos]

@thall63 It would need to be converted to the appropriate JSON, it's in PDB format at the moment. The file was more for us to debug what was happening. The PDB can be processed into a JSON file if you're interested in seeing what it would look like, but it would be much more helpful to keep discussions like this which are more interactive on a medium like slack (which you can join by going to https://www.volatilityfoundation.org/slack) so please contact me there if you'd like to do that...

[github-actions]

This issue is stale because it has been open for 200 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 60 days since being marked as stale.