Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate hash scans #54

Merged
merged 45 commits into from
Dec 5, 2023
Merged

Automate hash scans #54

merged 45 commits into from
Dec 5, 2023

Conversation

bra1ncramp
Copy link
Contributor

@bra1ncramp bra1ncramp commented Nov 1, 2023

πŸ—£ Description

Create extra file to automate the IOC Hash scans.

  • This script currently is only valid for Debian and Fedora systems.
  • Requires AWS_CREDENTIALS_FILE and AWS_REGION environmental variables to already be set.
  • Assumes it will be run from ioc-scanner/extras
  • Takes a file with a list of instance id's in separate lines
  • Requires declaring the AWS_PROFILE as an argument
  • Starts port forwarding on the local system. Then it verifies that netcat is installed in the instance and starts netcat listening on port 6666. It uploads the latest copy of ioc_scanner.py with the latest hashes to the instance. Then it executes ioc_scanner.py and directs the output to a local log file.

πŸ’­ Motivation and context

The current process requires manually uploading the ioc_scanner.py file to each instance one at a time. This is a laborious method and not scaleable.

πŸ§ͺ Testing

This was tested using the default test blobs in ioc_scanner.py.

βœ… Pre-approval checklist

  • This PR has an informative and human-readable title.
  • Changes are limited to a single goal - eschew scope creep!
  • All relevant type-of-change labels have been added.
  • I have read the CONTRIBUTING document.
  • These code changes follow cisagov code standards.
  • All new and existing tests pass.

@bra1ncramp bra1ncramp added the improvement This issue or pull request will add or improve functionality, maintainability, or ease of use label Nov 1, 2023
@dav3r
Copy link
Member

dav3r commented Nov 3, 2023

For the same reasons as discussed in #53, this PR will not require a version change to this repo, so I deleted the "Pre-merge" and "Post-merge" checklist sections in the PR description.

extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Show resolved Hide resolved
extras/ioc_hash_scan.sh Show resolved Hide resolved
extras/ioc_hash_scan.sh Show resolved Hide resolved
Copy link
Member

@dav3r dav3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for building out this script- it will be an excellent improvement from our current manual process.

Please take a look at my first round of suggestions.

extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Show resolved Hide resolved
extras/ioc_hash_scan.sh Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
@dav3r
Copy link
Member

dav3r commented Nov 14, 2023

@bra1ncramp Just pinging on this PR so that it doesn't get forgotten. It's good stuff- we should try to get it across the finish line soon.

bra1ncramp and others added 8 commits November 14, 2023 13:59
Co-authored-by: Shane Frasier <jeremy.frasier@gwe.cisa.dhs.gov>
Co-authored-by: Shane Frasier <jeremy.frasier@gwe.cisa.dhs.gov>
Co-authored-by: Shane Frasier <jeremy.frasier@gwe.cisa.dhs.gov>
Co-authored-by: Shane Frasier <jeremy.frasier@gwe.cisa.dhs.gov>
Co-authored-by: Shane Frasier <jeremy.frasier@gwe.cisa.dhs.gov>
Co-authored-by: Shane Frasier <jeremy.frasier@gwe.cisa.dhs.gov>
Co-authored-by: dav3r <david.redmin@gwe.cisa.dhs.gov>
Co-authored-by: dav3r <david.redmin@gwe.cisa.dhs.gov>
Copy link
Member

@jsf9k jsf9k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a suggestion.

extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
bra1ncramp and others added 4 commits November 15, 2023 11:01
Co-authored-by: dav3r <david.redmin@gwe.cisa.dhs.gov>
This may go away - but committing the suggestion for now until we improve the language overall.

Co-authored-by: dav3r <david.redmin@gwe.cisa.dhs.gov>
Simply running `hostname` should work fine with `AWS-StartNonInteractiveCommand`

Co-authored-by: dav3r <david.redmin@gwe.cisa.dhs.gov>
Co-authored-by: dav3r <david.redmin@gwe.cisa.dhs.gov>
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
@jsf9k
Copy link
Member

jsf9k commented Nov 20, 2023

@bra1ncramp - Are you running pre-commit locally? It should have automatically caught the correction I made in commit 93e1b32.

Copy link
Member

@dav3r dav3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are a couple more small thangs to clean up.

extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
extras/ioc_hash_scan.sh Outdated Show resolved Hide resolved
bra1ncramp and others added 5 commits December 4, 2023 10:41
@jsf9k
Copy link
Member

jsf9k commented Dec 4, 2023

@bra1ncramp - Are you running pre-commit locally? It should have automatically caught the correction I made in commit 93e1b32.

@bra1ncramp - Are you running pre-commit locally? It should have automatically caught the linting error you committed in e21a303.

@bra1ncramp
Copy link
Contributor Author

bra1ncramp commented Dec 4, 2023

@bra1ncramp - Are you running pre-commit locally? It should have automatically caught the correction I made in commit 93e1b32.

@bra1ncramp - Are you running pre-commit locally? It should have automatically caught the linting error you committed in e21a303.

What's the error? pre-commit isn't showing me any error.

I was running it wrong. I should have run pre-commit run --all-files instead of just pre-commit

@jsf9k
Copy link
Member

jsf9k commented Dec 4, 2023

@bra1ncramp - Are you running pre-commit locally? It should have automatically caught the correction I made in commit 93e1b32.

@bra1ncramp - Are you running pre-commit locally? It should have automatically caught the linting error you committed in e21a303.

What's the error? pre-commit isn't showing me any error.

I was running it wrong. I should have run pre-commit run --all-files instead of just pre-commit

You should run pre-commit install to install the git hook. Then pre-commit will check your changes before you are allowed to commit them. See the instructions here, for example.

@jsf9k jsf9k requested a review from dav3r December 5, 2023 19:15
Copy link
Member

@dav3r dav3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks solid - strong work! πŸ’ͺ πŸ’Ό

@jsf9k jsf9k merged commit 05a2181 into develop Dec 5, 2023
49 checks passed
@jsf9k jsf9k deleted the automate-hash-scans branch December 5, 2023 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement This issue or pull request will add or improve functionality, maintainability, or ease of use
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants