Skip to content

NIFI-4735: ParseEVTX only outputs one event per chunk#2489

Closed
askene wants to merge 1 commit intoapache:masterfrom
askene:NIFI-4735
Closed

NIFI-4735: ParseEVTX only outputs one event per chunk#2489
askene wants to merge 1 commit intoapache:masterfrom
askene:NIFI-4735

Conversation

@askene
Copy link
Copy Markdown

@askene askene commented Feb 23, 2018

Updated the EVTX FileHeader class to correctly check if there are more chunks in the file. Previously this would not process the last chunk.

Updated the EVTX ChunkHeader class to correctly check if there are additional records in the chunk. Previously this would only process the first record of each chunk. It was using the fileLastRecordNumber where it should have been using the logLastRecordNumber value.

Updated the EVTX unit tests to have the correct expected number of events and use the logLastRecordNumber.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

  • Is there a JIRA ticket associated with this PR? Is it referenced
    in the commit message?

  • Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

  • Has your PR been rebased against the latest commit within the target branch (typically master)?

  • Is your initial contribution a single, squashed commit?

For code changes:

  • Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
  • Have you written or updated unit tests to verify your changes?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
  • If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
  • If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

  • Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

…e chunks in the file. Previously this would not process the last chunk.

Updated the EVTX ChunkHeader class to correctly check if there are additional records in the chunk. Previously this would only process the first record of each chunk. It was using the fileLastRecordNumber where it should have been using the logLastRecordNumber value.
Updated the EVTX unit tests to have the correct expected number of events and use the logLastRecordNumber.
Copy link
Copy Markdown
Contributor

@szaboferee szaboferee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution @askene. I have tried to validate your fix and ran into an issue that I could not determine the valid log message number from the test evtx file because it is corrupted. Could you add a valid evtx file where you know exactly how many log events should we parse and use that to test the parser? I have used the windows event log viewer to check if the new numbers are correct, it showed 1277 events, a python evtx converter showed 1300 events and the tests expect 1053 events. This makes it hard to review the fix.

szaboferee pushed a commit to szaboferee/nifi that referenced this pull request Mar 20, 2019
This change is based on apache#2489

I have reproduced the issue with some additional test cases and test files then applied the original fix.

commit message from the original change:
Updated the EVTX FileHeader class to correctly check if there are more chunks in the file. Previously this would not process the last chunk.

Updated the EVTX ChunkHeader class to correctly check if there are additional records in the chunk. Previously this would only process the first record of each chunk. It was using the fileLastRecordNumber where it should have been using the logLastRecordNumber value.

Updated the EVTX unit tests to have the correct expected number of events and use the logLastRecordNumber.
@asfgit asfgit closed this in 48a6c81 Mar 20, 2019
asfgit pushed a commit that referenced this pull request Apr 1, 2019
This change is based on #2489

I have reproduced the issue with some additional test cases and test files then applied the original fix.

commit message from the original change:
Updated the EVTX FileHeader class to correctly check if there are more chunks in the file. Previously this would not process the last chunk.

Updated the EVTX ChunkHeader class to correctly check if there are additional records in the chunk. Previously this would only process the first record of each chunk. It was using the fileLastRecordNumber where it should have been using the logLastRecordNumber value.

Updated the EVTX unit tests to have the correct expected number of events and use the logLastRecordNumber.

refactoring duplicated code and magic numbers

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2489
This closes #3379
asfgit pushed a commit that referenced this pull request Apr 3, 2019
This change is based on #2489

I have reproduced the issue with some additional test cases and test files then applied the original fix.

commit message from the original change:
Updated the EVTX FileHeader class to correctly check if there are more chunks in the file. Previously this would not process the last chunk.

Updated the EVTX ChunkHeader class to correctly check if there are additional records in the chunk. Previously this would only process the first record of each chunk. It was using the fileLastRecordNumber where it should have been using the logLastRecordNumber value.

Updated the EVTX unit tests to have the correct expected number of events and use the logLastRecordNumber.

refactoring duplicated code and magic numbers

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2489
This closes #3379
rwithers pushed a commit to rwithers/nifi that referenced this pull request May 3, 2019
This change is based on apache#2489

I have reproduced the issue with some additional test cases and test files then applied the original fix.

commit message from the original change:
Updated the EVTX FileHeader class to correctly check if there are more chunks in the file. Previously this would not process the last chunk.

Updated the EVTX ChunkHeader class to correctly check if there are additional records in the chunk. Previously this would only process the first record of each chunk. It was using the fileLastRecordNumber where it should have been using the logLastRecordNumber value.

Updated the EVTX unit tests to have the correct expected number of events and use the logLastRecordNumber.

refactoring duplicated code and magic numbers

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes apache#2489
This closes apache#3379
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants