Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactoring file templates to handle large files in chunks + deprecated methods refactor #1634

Merged
merged 32 commits into from
Mar 8, 2022
Merged

Refactoring file templates to handle large files in chunks + deprecated methods refactor #1634

merged 32 commits into from
Mar 8, 2022

Conversation

Mzack9999
Copy link
Member

@Mzack9999 Mzack9999 commented Feb 23, 2022 •
edited
Loading

Proposed changes

This PR refactors the file template logic, introducing the following improvement:

  • Large file processing in chunks
  • Removing deprecated ioutil methods

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Notes:

@Mzack9999 Mzack9999 added the Type: Enhancement Most issues will probably ask for additions or changes. label Feb 23, 2022
@Mzack9999 Mzack9999 self-assigned this Feb 23, 2022
@Mzack9999 Mzack9999 added the Status: In Progress This issue is being worked on, and has someone assigned. label Feb 23, 2022
@Mzack9999 Mzack9999 linked an issue Feb 23, 2022 that may be closed by this pull request
File protocol regex search improvement #1594
Open
@Mzack9999 Mzack9999 removed a link to an issue Feb 26, 2022
File protocol regex search improvement #1594
Open
@Mzack9999 Mzack9999 added Status: Review Needed The issue has a PR attached to it which needs to be reviewed and removed Status: In Progress This issue is being worked on, and has someone assigned. labels Feb 26, 2022
@Mzack9999 Mzack9999 requested review from Ice3man543 and removed request for Ice3man543 February 26, 2022 08:07
This was linked to issues Mar 1, 2022
Add support to traverse compressed archives for file templates #1578
Closed
File protocol regex search improvement #1594
Open
@Mzack9999 Mzack9999 removed a link to an issue Mar 2, 2022
File protocol regex search improvement #1594
Open
@Mzack9999 Mzack9999 mentioned this pull request Mar 2, 2022
Open
@Mzack9999
Copy link
Member Author

Mzack9999 commented Mar 3, 2022 •
edited by ehsandeep
Loading

Notes:

  • Stream analysis is slower than full content in-memory analysis
  • Files are read n times (with n the number of file templates selected for the scan), streaming-clustering should be implemented (a la hyperscan)
  • Stats are not working with file protocol

Ice3man543
Ice3man543 approved these changes Mar 3, 2022
View reviewed changes
Copy link
Member

@Ice3man543 Ice3man543 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@sonarcloud
Copy link

sonarcloud bot commented Mar 4, 2022

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

51.1% 51.1% Coverage
1.2% 1.2% Duplication

@Mzack9999 Mzack9999 requested a review from ehsandeep March 4, 2022 11:23
@Mzack9999 Mzack9999 mentioned this pull request Mar 7, 2022
Merged
4 tasks
@ehsandeep ehsandeep merged commit fd8b46d into dev Mar 8, 2022
@ehsandeep ehsandeep deleted the issue-1594-regexp branch March 8, 2022 04:56
@ehsandeep ehsandeep added Status: Completed Nothing further to be done with this issue. Awaiting to be closed. and removed Status: Review Needed The issue has a PR attached to it which needs to be reviewed labels Mar 8, 2022
@ehsandeep ehsandeep mentioned this pull request Mar 10, 2022
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ice3man543 Ice3man543 Ice3man543 approved these changes

@ehsandeep ehsandeep ehsandeep approved these changes

Assignees

@Mzack9999 Mzack9999

Labels
Status: Completed Nothing further to be done with this issue. Awaiting to be closed. Type: Enhancement Most issues will probably ask for additions or changes.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add support to traverse compressed archives for file templates
3 participants
@Mzack9999 @ehsandeep @Ice3man543