Refactored setupapi parser #2717 #2718

Onager · 2019-07-14T21:14:09Z

Also changed the EncodedTextReader to not strip every line and updated parsers that relied on this behavior.

There's also more changes that I'd like make to the parser, but I've tried to keep changes to the tests minimal in this first round.

codecov · 2019-07-15T06:44:19Z

Codecov Report

Merging #2718 into master will increase coverage by 0.07%.
The diff coverage is 92.42%.

@@            Coverage Diff             @@
##           master    #2718      +/-   ##
==========================================
+ Coverage   86.21%   86.28%   +0.07%     
==========================================
  Files         481      487       +6     
  Lines       32788    33062     +274     
==========================================
+ Hits        28268    28529     +261     
- Misses       4520     4533      +13

Impacted Files	Coverage Δ
plaso/formatters/setupapi.py	`100% <ø> (ø)`	⬆️
plaso/parsers/text_parser.py	`81.53% <100%> (ø)`	⬆️
plaso/parsers/mac_wifi.py	`90.55% <100%> (+0.07%)`	⬆️
plaso/parsers/winfirewall.py	`94.79% <100%> (+0.05%)`	⬆️
plaso/parsers/iis.py	`93.65% <100%> (+0.05%)`	⬆️
plaso/parsers/setupapi.py	`93.54% <91.37%> (+9.76%)`	⬆️
plaso/output/shared_4n6time.py	`83.33% <0%> (-1.48%)`	⬇️
plaso/parsers/winreg_plugins/interface.py	`96.29% <0%> (-0.93%)`	⬇️
plaso/storage/fake/writer.py	`84.7% <0%> (-0.55%)`	⬇️
plaso/lib/definitions.py	`100% <0%> (ø)`	⬆️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d390dfe...e5808cb. Read the comment docs.

joachimmetz · 2019-07-15T07:06:53Z

plaso/parsers/setupapi.py

-      structure (pyparsing.ParseResults): structure of tokens derived from
-          log entry.
+      time_structure (pyparsing.ParseResults): structure of tokens derived from
+          a setupapi time stamp.


time stamp => date and time value

The setupapi documents call this field a timestamp: https://docs.microsoft.com/en-us/windows-hardware/drivers/install/format-of-a-text-log-section-header

Clarified this.

plaso/parsers/setupapi.py

joachimmetz · 2019-07-15T07:08:49Z

plaso/parsers/setupapi.py

+
+    Returns:
+      dfdatetime.TimeElements: date and time extracted from the value or None
+          if the value does not represent a valid string.


a valid string => a valid date and time value

joachimmetz · 2019-07-15T07:09:47Z

plaso/parsers/setupapi.py


    Returns:
      bool: True if this is the correct parser, False otherwise.
    """
    try:
-      structure = self._SETUPAPI_LINE.parseString(lines)
+      _ = self._LOG_HEADER_START.parseString(line)


remove '_ =', it is not needed

joachimmetz · 2019-07-15T07:10:37Z

plaso/parsers/winfirewall.py

@@ -224,7 +224,7 @@ def VerifyStructure(self, parser_mediator, line):
    """
    # TODO: Examine other versions of the file format and if this parser should
    # support them.
-    return line == '#Version: 1.5'
+    return '#Version: 1.5' in line


Why change this?

Because of the whitespace matching. Changed this to just have the newline.

joachimmetz · 2019-07-15T07:11:07Z

tests/parsers/setupapi.py

@@ -101,8 +101,8 @@ def testParseSetupLog(self):

    self.CheckTimestamp(event.timestamp, '2015-11-22 17:53:29.305000')

-    expected_message = ('Setup Plug and Play Device Install - START')
-    expected_short_message = ('START - Setup Plug and Play Device Install')
+    expected_message = ('Setup Plug and Play Device Install')


remove bounding parenthesis

joachimmetz

please address comments

dfjxs

LGTM
Should we add this to the win_gen preset?

Onager · 2019-07-16T12:47:53Z

@js-forensic - good idea, I thought this had already been done

plaso/parsers/setupapi.py

joachimmetz · 2019-07-17T04:49:58Z

plaso/parsers/winfirewall.py

@@ -224,7 +224,8 @@ def VerifyStructure(self, parser_mediator, line):
    """
    # TODO: Examine other versions of the file format and if this parser should
    # support them.
-    return line == '#Version: 1.5'
+    stripped_line = line.strip()


strip() => rstrip()

joachimmetz

please make sure that the parser does not use state from a previous file it parsed

joachimmetz · 2019-07-31T15:05:18Z

tox tests are failing:

testExamineEventAndCompileReportOnSystemFile (analysis.windows_services.WindowsServicesTest)
1614Tests the ExamineEvent and CompileReport functions on a SYSTEM file. ... free(): invalid next size (fast)
1615ERROR: InvocationError: '/plaso/.tox/py27/bin/coverage run --source=plaso --omit=*_test*,*__init__*,*test_lib* run_tests.py'

I'll restart them, no luck, seems to be an issue with a dependency #2727

Onager · 2019-08-19T18:28:42Z

Ping for @joachimmetz

plaso/parsers/mac_wifi.py

joachimmetz · 2019-08-21T20:08:22Z

plaso/parsers/setupapi.py


-  def _ParseRecordLogline(self, parser_mediator, structure):
-    """Parses a logline record structure and produces events.
+  def _BuildDatetime(self, time_structure):


Change name of this method to _GetDateTime or equiv to keep it consistent with other parsers

joachimmetz

Some small remaining questions

joachimmetz

LGTM

Onager added 3 commits July 14, 2019 23:12

Refactored setupapi parser log2timeline#2717

209f89c

Removed unused buffer size constant

f9bd002

Changes for linter

28762e4

Onager requested a review from joachimmetz July 15, 2019 06:48

Onager assigned Onager and joachimmetz and unassigned Onager Jul 15, 2019

joachimmetz reviewed Jul 15, 2019

View reviewed changes

plaso/parsers/setupapi.py Show resolved Hide resolved

joachimmetz reviewed Jul 15, 2019

View reviewed changes

joachimmetz requested changes Jul 15, 2019

View reviewed changes

Changes after review

0a5b24d

Onager requested a review from joachimmetz July 15, 2019 09:26

Changed iis verification

642ddc9

Onager requested a review from dfjxs July 15, 2019 16:31

dfjxs requested changes Jul 15, 2019

View reviewed changes

Added setupapi parser to Windows preset

33ec272

dfjxs mentioned this pull request Jul 16, 2019

Apt history #2710

Merged

3 tasks

joachimmetz reviewed Jul 17, 2019

View reviewed changes

plaso/parsers/setupapi.py Show resolved Hide resolved

joachimmetz reviewed Jul 17, 2019

View reviewed changes

plaso/parsers/setupapi.py Show resolved Hide resolved

joachimmetz reviewed Jul 17, 2019

View reviewed changes

joachimmetz requested changes Jul 17, 2019

View reviewed changes

dfjxs approved these changes Jul 17, 2019

View reviewed changes

Changes after review

427b24a

Onager requested a review from joachimmetz July 30, 2019 20:44

joachimmetz reviewed Aug 21, 2019

View reviewed changes

plaso/parsers/mac_wifi.py Show resolved Hide resolved

joachimmetz reviewed Aug 21, 2019

View reviewed changes

joachimmetz requested changes Aug 21, 2019

View reviewed changes

Changes after review

e5808cb

Onager requested a review from joachimmetz August 23, 2019 11:23

joachimmetz approved these changes Sep 2, 2019

View reviewed changes

joachimmetz merged commit e22155f into log2timeline:master Sep 2, 2019

Onager deleted the setupapi branch November 30, 2019 13:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactored setupapi parser #2717 #2718

Refactored setupapi parser #2717 #2718

Onager commented Jul 14, 2019 •

edited

Loading

codecov bot commented Jul 15, 2019 •

edited

Loading

joachimmetz Jul 15, 2019

Onager Jul 15, 2019

joachimmetz Jul 15, 2019

Onager Jul 15, 2019

joachimmetz Jul 15, 2019 •

edited

Loading

Onager Jul 15, 2019

joachimmetz Jul 15, 2019

Onager Jul 15, 2019

joachimmetz Jul 15, 2019

Onager Jul 15, 2019

joachimmetz left a comment

dfjxs left a comment

Onager commented Jul 16, 2019

joachimmetz Jul 17, 2019

Onager Jul 30, 2019

joachimmetz left a comment

joachimmetz commented Jul 31, 2019 •

edited

Loading

Onager commented Aug 19, 2019

joachimmetz Aug 21, 2019

Onager Aug 23, 2019

joachimmetz left a comment

joachimmetz left a comment

Refactored setupapi parser #2717 #2718

Refactored setupapi parser #2717 #2718

Conversation

Onager commented Jul 14, 2019 • edited Loading

codecov bot commented Jul 15, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joachimmetz Jul 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joachimmetz left a comment

Choose a reason for hiding this comment

dfjxs left a comment

Choose a reason for hiding this comment

Onager commented Jul 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joachimmetz left a comment

Choose a reason for hiding this comment

joachimmetz commented Jul 31, 2019 • edited Loading

Onager commented Aug 19, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joachimmetz left a comment

Choose a reason for hiding this comment

joachimmetz left a comment

Choose a reason for hiding this comment

Onager commented Jul 14, 2019 •

edited

Loading

codecov bot commented Jul 15, 2019 •

edited

Loading

joachimmetz Jul 15, 2019 •

edited

Loading

joachimmetz commented Jul 31, 2019 •

edited

Loading