Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve keyword scans #53

Merged
merged 7 commits into from
Nov 1, 2023
Merged

Improve keyword scans #53

merged 7 commits into from
Nov 1, 2023

Conversation

bra1ncramp
Copy link
Contributor

@bra1ncramp bra1ncramp commented Nov 1, 2023

πŸ—£ Description

This PR makes two changes:

  • Outputs the hostname of each instance
  • Uses zgrep search command. NOTE: --recursive is removed since it is not a valid flag for zgrep

πŸ’­ Motivation and context

  • Providing the instance hostname gives more clarity in the log output, making it more useful in rolled-over logs
  • Using grep to search for key words on instances will only reveal possible IOCs if the incident happened within the last rollover date for the log files. Switching to zgrep enables us to search inside rolled-over (gzipped) log files increases the thoroughness of the scans, thus improving the utility of this script.

βœ… Pre-approval checklist

  • This PR has an informative and human-readable title.
  • Changes are limited to a single goal - eschew scope creep!
  • All relevant type-of-change labels have been added.
  • I have read the CONTRIBUTING document.
  • These code changes follow cisagov code standards.
  • All new and existing tests pass.

Use zgrep to search in rolled-over logs
Provide hostname of instance for more clarity in log output
@bra1ncramp bra1ncramp changed the title Improve log output Improve keyword scans Nov 1, 2023
@dav3r dav3r self-assigned this Nov 1, 2023
@bra1ncramp bra1ncramp added the improvement This issue or pull request will add or improve functionality, maintainability, or ease of use label Nov 1, 2023
Copy link
Member

@mcdonnnj mcdonnnj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems reasonable and LGTM. I have one suggestion just to update a comment to reflect the change in functionality.

extras/ioc_scan_by_host.sh Outdated Show resolved Hide resolved
bra1ncramp and others added 2 commits November 1, 2023 12:17
Co-authored-by: Nick <50747025+mcdonnnj@users.noreply.github.com>
Instead of ignoring `--ingnore-match --recursive` look for `grep` command
Copy link
Member

@dav3r dav3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a couple of small requests:

extras/ioc_scan_by_host.sh Outdated Show resolved Hide resolved
extras/ioc_scan_by_host.sh Outdated Show resolved Hide resolved
extras/ioc_scan_by_host.sh Outdated Show resolved Hide resolved
Copy link
Member

@dav3r dav3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

πŸ‘ Good stuff!

Copy link
Member

@mcdonnnj mcdonnnj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cisagov/team-ois I'm a bit hesitant about the version bump given that this PR does not change the Python package functionality at all. Thoughts?

@dav3r
Copy link
Member

dav3r commented Nov 1, 2023

@cisagov/team-ois I'm a bit hesitant about the version bump given that this PR does not change the Python package functionality at all. Thoughts?

I was on the fence about that also, but I told @bra1ncramp to do it. Since we are changing the way an included script works with this PR, I felt it was warranted, but honestly, I'm fine either way if you and @jsf9k want to overrule that.

@jsf9k
Copy link
Member

jsf9k commented Nov 1, 2023

@cisagov/team-ois I'm a bit hesitant about the version bump given that this PR does not change the Python package functionality at all. Thoughts?

I was on the fence about that also, but I told @bra1ncramp to do it. Since we are changing the way an included script works with this PR, I felt it was warranted, but honestly, I'm fine either way if you and @jsf9k want to overrule that.

I think anytime we make meaningful changes we are potentially breaking functionality (in this case, that of the script in extras/). Certainly we'd want users to be able to say, "Hey, this script worked in 1.5.3 but not in 1.5.4" so I think a version bump makes sense here.

@mcdonnnj
Copy link
Member

mcdonnnj commented Nov 1, 2023

@cisagov/team-ois I'm a bit hesitant about the version bump given that this PR does not change the Python package functionality at all. Thoughts?

I was on the fence about that also, but I told @bra1ncramp to do it. Since we are changing the way an included script works with this PR, I felt it was warranted, but honestly, I'm fine either way if you and @jsf9k want to overrule that.

I think anytime we make meaningful changes we are potentially breaking functionality (in this case, that of the script in extras/). Certainly we'd want users to be able to say, "Hey, this script worked in 1.5.3 but not in 1.5.4" so I think a version bump makes sense here.

I get that but my core rationale is that the file only exists if you clone the repository or download it manually. The extras/ directory is not included in the wheel when you build the package for distribution which is, in my mind, what the version represents: the Python package.

@jsf9k
Copy link
Member

jsf9k commented Nov 1, 2023

@cisagov/team-ois I'm a bit hesitant about the version bump given that this PR does not change the Python package functionality at all. Thoughts?

I was on the fence about that also, but I told @bra1ncramp to do it. Since we are changing the way an included script works with this PR, I felt it was warranted, but honestly, I'm fine either way if you and @jsf9k want to overrule that.

I think anytime we make meaningful changes we are potentially breaking functionality (in this case, that of the script in extras/). Certainly we'd want users to be able to say, "Hey, this script worked in 1.5.3 but not in 1.5.4" so I think a version bump makes sense here.

I get that but my core rationale is that the file only exists if you clone the repository or download it manually. The extras/ directory is not included in the wheel when you build the package for distribution which is, in my mind, what the version represents: the Python package.

OK, that is a valid argument. I'm fine either way, so I'm happy to defer to @mcdonnnj here. @mcdonnnj - do you mind removing that version bump commit and force-pushing?

bra1ncramp and others added 2 commits November 1, 2023 14:41
Co-authored-by: dav3r <david.redmin@gwe.cisa.dhs.gov>
Co-authored-by: dav3r <david.redmin@gwe.cisa.dhs.gov>
@dav3r
Copy link
Member

dav3r commented Nov 1, 2023

Since the version bump was removed, I went ahead and deleted the pre-merge ("Finalize version") and post-merge ("Create a release") checklist items from the PR description now that they are no longer applicable.

@dav3r
Copy link
Member

dav3r commented Nov 1, 2023

@mcdonnnj Just waiting on your final approval and then I will merge this.

Copy link
Member

@jsf9k jsf9k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval intensifies!!

@jsf9k
Copy link
Member

jsf9k commented Nov 1, 2023

@mcdonnnj Just waiting on your final approval and then I will merge this.

Just turn on auto-merge! It will merge once @mcdonnnj approves.

@dav3r dav3r enabled auto-merge November 1, 2023 19:29
Copy link
Member

@mcdonnnj mcdonnnj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops I knew I forgot to do something.

@dav3r dav3r merged commit 1eac84e into develop Nov 1, 2023
49 checks passed
@dav3r dav3r deleted the Improve-Keyword-Scans branch November 1, 2023 20:32
@dav3r dav3r mentioned this pull request Nov 3, 2023
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement This issue or pull request will add or improve functionality, maintainability, or ease of use
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants