New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collection Filters Enhancement #103

Open
dnides opened this Issue Jan 31, 2015 · 12 comments

Comments

9 participants
@dnides
Copy link
Contributor

dnides commented Jan 31, 2015

Related to my inquiry:

Do you have any examples of plaso filters for specific files in the recycle bin? We've tried something like: /$Recycle.Bin/.+/$I30 with no success.Also have tried something like /$Recycle.Bin/.+/.+ etc to try to get everything x dirs down (this same type of pattern works in other cases)...also did not work.

Requesting support for the following two items in the collection filters:

  • blacklisting entries in the filter files
  • adding recursive support
  • having the ability to define "negative" filter files, eg. a separate filter file of files to exclude
  • Add a "startswith" and "endswith" into the filter criteria
  • add ADS support
  • use "native" OS path notation

My understanding is the current support is limited, since it does not have recursive behavior....

@dnides dnides added the enhancement label Jan 31, 2015

@joachimmetz joachimmetz added the core label Jul 28, 2015

@joachimmetz joachimmetz referenced this issue Sep 1, 2015

Open

Clean up: filters #324

3 of 9 tasks complete

@kiddinn kiddinn added this to the 1.5 release milestone Oct 18, 2015

@rodgermoore

This comment has been minimized.

Copy link

rodgermoore commented Mar 17, 2016

👍 for this

@joachimmetz

This comment has been minimized.

Copy link
Member

joachimmetz commented Mar 17, 2016

The realization of this enhance will depend on: #646

@joachimmetz joachimmetz removed this from the 1.5.0 release milestone Mar 17, 2016

@rgayon

This comment has been minimized.

Copy link
Contributor

rgayon commented Aug 5, 2016

I would also like to see the "exclude" filter be implemented.

S'il vous plaît.

@joachimmetz joachimmetz added the idea label Aug 9, 2016

@ilyaglow

This comment has been minimized.

Copy link

ilyaglow commented Jun 7, 2017

Any updates here?

@ctmayhew

This comment has been minimized.

Copy link

ctmayhew commented Aug 30, 2017

This would be super handy for all those pointless files in C:\Windows\WinSxS

@joachimmetz

This comment has been minimized.

Copy link
Member

joachimmetz commented Aug 30, 2017

Added to next milestone for consideration

@joachimmetz joachimmetz added this to the 2017 Q4 release milestone Aug 30, 2017

@joachimmetz joachimmetz added this to To do in Filter support Dec 1, 2017

@joachimmetz

This comment has been minimized.

Copy link
Member

joachimmetz commented Dec 1, 2017

Update: filters in plaso are in desperate need of some love, but this will take some time.

I'll start with the filter file. Github project to follow progress: https://github.com/log2timeline/plaso/projects/6

@joachimmetz joachimmetz removed this from the 2017 December release milestone Dec 1, 2017

@MariasStory

This comment has been minimized.

Copy link

MariasStory commented Aug 17, 2018

Cool tool to get ideas from: https://github.com/EricZimmerman/MFTECmd

@joachimmetz

This comment has been minimized.

Copy link
Member

joachimmetz commented Aug 17, 2018

@MariasStory please explain your comment we cannot assess every tool out there nor can we guess what you mean.

What element(s) of MFTECmd regarding collection filtering to do you think plaso needs to adopt?

@MariasStory

This comment has been minimized.

Copy link

MariasStory commented Aug 17, 2018

Hi @joachimmetz and sorry for not being precise. Now, I am not sure if this comment is in the right place/issue.

I was referring to "ADS support" from the check list above. The MFTECmd does a good job there with "Zone.Identifier", as explained in: https://binaryforay.blogspot.com/2018/06/introducing-mftecmd.html

The ads info is extracted in: https://github.com/EricZimmerman/MFT/blob/96af1eb2124824bd41b505a3ecd29cd9362978e9/MFT/Other/ExtensionMethods.cs

public static bool HasAds(this FileRecord record)
record.Attributes.Where(t =>
                    t.AttributeType == AttributeType.Data && t.NameSize > 0).ToList();
-----------------
public static List<AdsInfo> GetAlternateDataStreams(this FileRecord record)
...

There are also another MFT atributes to look for, like flags. Also, the function public string GetFullParentPath(string recordKey) in https://github.com/EricZimmerman/MFT/blob/f16feea02a1d68965e68f0d3fb7bf5892d822cdd/MFT/Mft.cs

@joachimmetz

This comment has been minimized.

Copy link
Member

joachimmetz commented Aug 18, 2018

@MariasStory still unclear to me what you mean. This issue is about enhancing the collection filters. The ADS support mentioned above is to update the Collection Filter specification to allow to specify the name of an ADS. The gotcha here is the Windows ADS naming convention allows for ambiguity (when applied cross-platform). Such as how to distinguish between a filename with : or an ADS.

There are also another MFT atributes to look for, like flags.

I'm unfamiliar with a "flags" MFT attribute (Also see: https://github.com/libyal/libfsntfs/blob/master/documentation/New%20Technologies%20File%20System%20(NTFS).asciidoc#61-the-attribute-types). So please be more specific what you mean.

GetFullParentPath

I'm skeptical this takes all edge cases into account. E.g. file A is part of directory B, file A gets deleted, directory B gets renamed to. Should the full path be "/B/A" or "/C/A" ?

@predictiple

This comment has been minimized.

Copy link

predictiple commented Dec 2, 2018

This would be super handy for all those pointless files in C:\Windows\WinSxS

I too suffer from the pain of winsxs and other "low value" files.

My workaround leverages some existing file filter features:

  • patterns can be regex, which means we can use a negative lookahead to exclude files/folders existing at a certain level (and above) on a certain path.
  • The file filter is cumulative, so the matches for each path are added to the collection list. Another way to think of this is that once a location is added for collection it can't be removed, with removal being the essence of what a future blacklist/negative filter feature should probably do. So we add things that we want in layers while excluding the parts of each layer that we don't want.

Some other things to bear in mind:

  • Once you apply a file filter you are implicitly excluding everything else. A future blacklist/negative filter feature would need to reverse that implicit exclusion logic, but for now we need to make sure that we also explicitly collect the other things we want.
  • The file filter path elements (between each pair of / separators) are handled separately, so your regex can't span multiple path segments.
  • The path elements are also handled case-insensitively, which makes life a lot easier.

The following is my hacky workaround filter that excludes a lot of Windows stuff that I regard as time-wasting junk. Your view of what "junk" is will obviously be different from mine.

# Include anything from root-level that doesn't start with "windows"
/(?!windows).+
# Now include the Windows dir but exclude several of it's 2nd-level+ dirs, with the famously annoying "winsxs" being one of them. At this time specifically exclude System32 because it is handled in the subsequent filter line.
/windows/(?!(assembly|boot|branding|ccmcache|ccm_old|ccmsetup|ccmsetup_old|csc|cursors|debug|diagnostics|ehome|en-us|fonts|globalization|help|ime|inf|installer|l2schemas|media|microsoft.net|migration|performance|pla|policydefinitions|rescache|resources|schemas|servicing|softwaredistribution|speech|system32|twain_32|web|winsxs)).*
# Now include Windows/System32 but exclude various 3rd-level+ dirs inside System32
/windows/system32/(?!(boot|backupfiles|catroot|dism|driverstore|en-us|ime|migwiz|manifeststore|oobe|speech|spool|spp|wbem|wdi|drivers|softwaredistribution)).*

A cautionary note: the above doesn't distinguish between file or folders at a particular level. If you are knowingly excluding a folder /Windows/Help you could unknowingly exclude a file /Windows/help.chm, for example, which you maybe didn't mean to do. Folders like "/Windows_backup" will need some extra care.

Hope this helps and feel free to suggest improvements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment