Bug in the artifact filter (using groups)

[kiddinn]

Description of problem:

Tried to extract only browser history from an image using the artifact group filters.... instead of ending up with a plaso file filled with browser history I got an error and the tool died before producing any output.

Command line and arguments:

(using the XP tdungan image from SANS as an example)

$ log2timeline.py --artifact_filters BrowserHistory hist.plaso xp-tdungan-c-drive.E01

2018-12-28 02:41:07,407 [INFO] (MainProcess) PID:224349 <data_location> Determined data location: /usr/lib/python2.7/dist-packages
2018-12-28 02:41:07,407 [INFO] (MainProcess) PID:224349 <artifact_definitions> Determined artifact definitions path: /usr/share/artifacts
2018-12-28 02:41:08,863 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.userprofile%%\Local Settings\Application Data\Microsoft\Feeds Cache\index.dat"
2018-12-28 02:41:08,863 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Microsoft\Windows\WebCache\WebCacheV*.dat"
2018-12-28 02:41:08,863 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.appdata%%\Microsoft\Windows\IEDownloadHistory\index.dat"
2018-12-28 02:41:08,863 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Microsoft\Windows\Temporary Internet Files\Low\Content.IE5\index.dat"
2018-12-28 02:41:08,863 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Microsoft\Windows\History\History.IE5\index.dat"
2018-12-28 02:41:08,864 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Microsoft\Windows\History\Low\History.IE5\*\index.dat"
2018-12-28 02:41:08,864 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Microsoft\Windows\History\History.IE5\*\index.dat"
2018-12-28 02:41:08,864 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Microsoft\Windows\History\Low\History.IE5\index.dat"
2018-12-28 02:41:08,864 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Microsoft\Windows\Temporary Internet Files\Content.IE5\index.dat"
2018-12-28 02:41:08,864 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Chromium\User Data\*\History"
2018-12-28 02:41:08,864 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Google\Chrome\User Data\*\Archived History"
2018-12-28 02:41:08,864 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Google\Chrome SxS\User Data\*\Archived History"
2018-12-28 02:41:08,864 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Google\Chrome SxS\User Data\*\History"
2018-12-28 02:41:08,864 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Chromium\User Data\*\Archived History"
2018-12-28 02:41:08,865 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Google\Chrome\User Data\*\History"
2018-12-28 02:41:08,865 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Apple Computer\Safari\History.plist"
2018-12-28 02:41:08,865 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.appdata%%\Apple Computer\Safari\History.plist"
2018-12-28 02:41:08,866 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Mozilla\Firefox\Profiles\*\places.sqlite"
2018-12-28 02:41:08,866 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.appdata%%\Mozilla\Firefox\Profiles\*\places.sqlite"
2018-12-28 02:41:08,866 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Apple Computer\Safari\Downloads.plist"
2018-12-28 02:41:08,866 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.appdata%%\Apple Computer\Safari\Downloads.plist"
2018-12-28 02:41:08,866 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.appdata%%\Opera Software\Opera Stable\History"
2018-12-28 02:41:08,866 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.appdata%%\Opera\Opera\global_history.dat"
2018-12-28 02:41:08,866 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Chromium\User Data\*\History"
2018-12-28 02:41:08,867 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Google\Chrome\User Data\*\Archived History"
2018-12-28 02:41:08,867 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Google\Chrome SxS\User Data\*\Archived History"
2018-12-28 02:41:08,867 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Google\Chrome SxS\User Data\*\History"
2018-12-28 02:41:08,867 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Chromium\User Data\*\Archived History"
2018-12-28 02:41:08,867 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Google\Chrome\User Data\*\History"
2018-12-28 02:41:08,868 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Apple Computer\Safari\History.plist"
2018-12-28 02:41:08,868 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.appdata%%\Apple Computer\Safari\History.plist"
2018-12-28 02:41:08,868 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Mozilla\Firefox\Profiles\*\places.sqlite"
2018-12-28 02:41:08,868 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.appdata%%\Mozilla\Firefox\Profiles\*\places.sqlite"
2018-12-28 02:41:08,868 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.localappdata%%\Apple Computer\Safari\Downloads.plist"
2018-12-28 02:41:08,868 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.appdata%%\Apple Computer\Safari\Downloads.plist"
2018-12-28 02:41:08,868 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.appdata%%\Opera Software\Opera Stable\History"
2018-12-28 02:41:08,869 [WARNING] (MainProcess) PID:224349 <artifact_filters> The path filter must be defined as an absolute path: "%%users.appdata%%\Opera\Opera\global_history.dat"
Checking availability and versions of dependencies.
[OK]
Source path     : <REDACTED>/xp-tdungan-c-drive.E01
Source type     : single file
Artifact filters        : [u'BrowserHistory']
Processing time : 00:00:00

Processing started.
Traceback (most recent call last):
  File "/usr/bin/log2timeline.py", line 68, in <module>
    if not Main():
  File "/usr/bin/log2timeline.py", line 54, in Main
    tool.ExtractEventsFromSources()
  File "/usr/lib/python2.7/dist-packages/plaso/cli/log2timeline_tool.py", line 426, in ExtractEventsFromSources
    self._filter_file)
  File "/usr/lib/python2.7/dist-packages/plaso/engine/engine.py", line 333, in BuildFilterFindSpecs
    'Error processing filters, no valid specifications built.')
RuntimeError: Error processing filters, no valid specifications built.
Please provide the command with arguments you ran when you experienced
the problem.

Source data:

The XP tdungan image provided by SANS for the 508 class.

Plaso version:

Latest released version (not head)... 20181219

Operating system Plaso is running on:

Linux, good ol' fashion Debian GNU/Linux

Installation method:

Using apt-get from GIFT PPA

[jnettesheim]

I'll take a look at this one.

[Onager]

I'm taking a look at this as well.

[Onager]

self._artifacts is modifying while iterating over it here:

plaso/plaso/engine/artifact_filters.py

Line 126 in e7fcf76

self._artifacts.remove(name)

and here:

plaso/plaso/engine/artifact_filters.py

Line 128 in e7fcf76

self._artifacts.append(name_entry)

Which is one part of the problem.

Also, lists are converted to sets in artifact_filters.py, so the processing order is non-deterministic.

[Onager]

Unfortunately, BrowserHistory on Windows relies on Plaso being able to expand the %%users.appdata%% variable, which it currently can't do. This in turn relies on dfwinreg supporting the HKEY_USERS key: which is also a work in progress.

#2310 will clean up the error message, and address some other issues in artifact group support, but not completely resolve this issue.

[joachimmetz]

@Onager I assume this issue has been addressed, reopen if not the case

[Onager]

Still not addressed, log2timeline/dfwinreg#73 is blocking full resolution of this.

[joachimmetz]

@Onager can you be a bit more detailed about what still needs to be addressed for the artifact groups?

For other related artifacts support issues I've created:

• add support for HKEY_USERS in artifact definitions #2325
• add support for %%users.variables%% in artifact definitions #2326
• Profile paths that contain %SystemDrive% require special handling for artifact filters #2327 strip %SystemDrive% from profile paths and bug fix in _BuildFindSpecsFromArtifact

[Onager]

This looks to be resolved now that the other artifacts changes are merged.