Tooling for automated detection of malware #7377
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add admin interface to view and enable checks - Implement list, detail and change_state views (#7133) - Add unit tests for check admin view * Add comprehensive test coverage for check admin
* Add initial hook-based check execution mechanism * scratch/poc * Add initial hook-based check execution mechanism * Use sqlalchemy event hooks for malware checks * Fix unit tests * Add enum for MalwareCheckObjectType * Add unit tests for init. * Add tests for tasks, services, and utils. Also, some small bugfixes in MalwareCheckFactory and the get_enabled_checks method. * Fix spurious task test. * Add missing drop enum to downgrade function. * Added TODO to dev/environment * Be more explicit in check lookup Co-authored-by: Ernest W. Durbin III <ewdurbin@gmail.com>
* Add malware check syncing mechanism * Code review changes.
* Add wipe-out functionality Related: #7133 * Call list explicitly
* Add rudimentary verdicts view. Progress on #6062. Also, add some better testing logic for wiped_out condition. * Code review changes. - Conditionally show fields that are populated - JSON pretty formatting * Fix unit test bug. - Use `get` instead of `filter` to look up verdict by pkey. * simplify unit tests for verdicts view
* introduce malware queue * correct syntax, apparently list of tuples documented doesn't work.
* Add backfill functionality to check admin #7094 - Add backfill task - Change lookup of checks to check_name instead of id - Load checks that are also in "evaluation" state * Add unit tests for backfill. - Log number of runs executed by backfill - Perform basic validation on sample_rate input - Clean up other testing logic. * Remove superfluous 'all()' * Code review changes. - Set backfill size to a fix number, not configurable via web ui. - Backfill task enqueues run_check tasks - Only retry if `check.run` fails, not if loading the check fails. - Use exponential backoff for retries. * Update warehouse/admin/templates/admin/malware/checks/detail.html Co-Authored-By: Ernest W. Durbin III <ewdurbin@gmail.com> Co-authored-by: Ernest W. Durbin III <ewdurbin@gmail.com>
* requirements: Introduce yara
* [WIP] malware/check: SetupPatternCheck
In progress.
Introduces SetupPatternCheck, an implementation of an event-based
check that scans the `setup.py`s of release files for suspicious
patterns.
* malware/checks: Give MalwareCheckBase.run/scan args, kwargs
* malware: Add check preparation
Fiddle with the check/run signature a bit more.
* malware/checks: Unpack file path correctly
* docker-compose: Override FILES_BACKEND for worker
The worker needs to be able to see the "files" virtual host
during development so that malware checks can fetch their underlying
release files.
* [WIP] malware/checks: setup.py extraction
* malware/checks: setup_patterns: Fix enum, seek
* malware/checks: setup_patterns: Apply YARA rules
Each rule match becomes a verdict.
* malware/checks: setup_patterns: Prefer get over filter
* warehouse/{admin,malware}: Consistent enum names
Also enforce uniqueness for enum values.
* warehouse/{admin,malware}: More enum changes
* tests: Update admin, malware tests
* tests: Fix enum, more test fixes
* tests: Add prepare tests
* malware/changes: base: Unpack id correctly
* tests: Begin adding SetupPatternCheck tests
* malware/checks: setup_patterns: Fix enum
* tests: More SetupPatternCheck tests
* warehouse/malware: setup_patterns: Fix enums
* tests: More SetupPatternCheck tests
* tests: Add license header
* malware/checks: setup_patterns: Add TODO
* tests: More SetupPatternCheck tests
* tests: More SetupPatternCheck tests
* tests: Complete extraction tests for SetupPatternCheck
* tests: Fix test
* malware/checks: Add docstring for prepare
* malware/checks: blacken
* malware/checks: Document, expand YARA rules
* tests, warehouse: Restructure utilities
* malware: Order some enums, reduce SetupPatternCheck verdicts
* malware/models: Add missing __lt__
* malware/checks: Always embed the model object in the prepared arguments
Use it instead of performing a DB request in the check itself.
* malware/checks: Avoid raw bytes
* malware/changes: Remove unused import
* tests: Fixup malware tests
* warehouse/malware: blacken
* tests: Fill in malware coverage
* tests, warehouse: Add a benign verdict for SetupPatternCheck
* tests: blacken
* Implement scheduled checks #7093 - Rename `run_backfill` to `run_evaluation` in admin malware view - Modify `run` and `scan` method signatures to accept `**kwargs` - Extend `run_check` to accomodate scheduled check functionality * Reduce unit test flakiness * Code review changes. Also replace `check.hooked_object` with `check.hooked_object.value` in check detail template. * tests, warehouse: enum fixes * Fix lint error Co-authored-by: William Woodruff <william@yossarian.net>
* Add verdict administrator review. Fixes #6062. - Add new `admin.verdicts.review` endpoint - Change layout of verdict list and detail view and add forms - Change sort order of the MalwareChecks, and update the tests * Code review changes. - Rename MalwareVerdict field `administrator_verdict` to `reviewer_verdict`. - Change verdict review permission from `admin` to `moderator`.
* Misc cleanup and TODOs on malware checks.
- Change backfill function to invoke `IMalwareCheckService` interface
- Add support for `kwargs to `IMalwareCheckService` interface
- Rename variable from reserved word `file` to `release_file`
- Add `FatalCheckException` for non-retryable exceptions
- Replace `MALWARE_CHECK_BACKEND` in dev/environment
* Make `IMalwareService` the entrypoint for `run_check`
- Add `run_scheduled_check` task that invokes this interface.
- Remove useless utility method
- Move `FatalCheckException` into warehouse/malware/errors.py.
* malware/checks: PackageTurnover skeleton * malware/checks: PackageTurnover: Add NOTE * malware/checks: PackageTurnoverCheck: more work * tests: blacken * malware/checks: More PackageTurnoverCheck work * malware/checks: Blacken * malware/checks: Blacken * package_turnover: Promote from indeterminate to threat * tests: Begin adding package_turnover tests * tests: Add remaining package_turnover tests * tests: Drop unused imports * warehouse: Drop (ww) from NOTE * checks/package_turnover: Drop NOTE
ewdurbin
force-pushed
the
malware-detection
branch
from
f3357b9
to
f614bad
Compare
This was referenced Feb 18, 2020
Closed
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Design and implementation work by @xmunoz, example checks by @woodruffw.