Playbooks (solves #628) #1123

0x0elliot · 2022-07-23T03:48:49Z

Description

Playbooks are simple JSON configs that help scan observables/files in mass properly.

New feature (non-breaking change which adds functionality).

Checklist

I have read and understood the rules about how to Contribute to this project
The pull request is for the branch develop
If external libraries/packages with restrictive licenses were used, they were added in the Legal Notice section.
The tests gave 0 errors.
Linters (Black, Flake, Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check CONTRIBUTE.md).

Important Rules

If your changes decrease the overall tests coverage (you will know after the Codecov CI job is done), you should add the required tests to fix the problem
Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review

Real World Example

0x0elliot · 2022-10-04T23:46:03Z

ready for a merge?

mlodic · 2022-10-05T16:43:36Z

I have been busy again, I'll check this tomorrow afternoon

mlodic · 2022-10-06T12:47:14Z

encountered weird errors for ClamAV. you might need to test that one. couldn't myself because the container keeps dying on my current machine. the error is on this line, explaining how the reports variable is a dictionary that can't be split. then why is it being split in the first place? ClamAV isn't covered in the file analyzers test cases as well.

do you have the triggered error stack trace or something similar more detailed?

mlodic · 2022-10-06T13:22:14Z

can you please fix the lines like I pushed here? (second one is missing and the first one is short) should be a fast change

EDIT:

Tags selection for playbooks still do not work:

Tags selection in playbooks do not work but this is caused to the problem we already talked about (we need to use the same validation/serializers that we use for classic analysis)

even if they fail, I would re-add the analyzers that you removed from the free-to-use playbook. They will be fixed. We can maintain them here so we don't forget that they exist as free analyzers. Another option could be to maintain both cases. A free-to-use playbook like the original one you made with all the free analyzers plus an "optimized" one with only the ones that are fast and more reliable. That would make sense and we would have 2 playbooks instead of 1 by default.
could you please add in the doc of the contributors in the section regardin new analyzers that when a contributor creates a new analyzer and that is free, he should add it in the free-to-use playbook too? A reminder in the pull request template would help too

tests/playbooks_manager/test_controller.py

Bumps [django](https://github.com/django/django) from 3.2.14 to 3.2.15. - [Release notes](https://github.com/django/django/releases) - [Commits](django/django@3.2.14...3.2.15) --- updated-dependencies: - dependency-name: django dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

frontend/package.json

…alyzers playbooks

0x0elliot · 2022-10-06T18:33:16Z

please review once again.

mlodic · 2022-10-07T12:53:25Z

please squash migrations and I'll merge! OMG I cant't believe it :) I am so excited to close this!

mlodic · 2022-10-07T12:55:15Z

.github/ISSUE_TEMPLATE/new_analyzer.md

@@ -14,6 +14,8 @@ assignees: ''
 ## Type of analyzer
 **this can be observable, file, and docker**

+**Please Note:** If the analyzer is free, Please add it in the `FREE_TO_USE_ANALYZERS` playbook in `playbook_config.json`


ah one last thing, this should be put here: https://github.com/intelowlproject/IntelOwl/blob/master/.github/release_template.md. Can you move that please?

yessir let me do that quickly

0x0elliot · 2022-10-08T10:23:25Z

🎉 🎉 🎉 🎉 🎉 and we are done with Playbooks now!

mlodic · 2022-10-10T07:33:56Z

squashing was not done properly, I'll adjust it and merge

0x0elliot added 30 commits May 30, 2022 02:23

Initialising work on Playbooks

98b4ed4

Setting up playbook_config.json

ead6745

Setting up playbook_config.json

1ecb583

Debugging playbook dataclass file

5006db3

Debugging playbook serializer file

f773eeb

Debugging playbook serializer file

40b2390

Debugging playbook serializer file

e276dd2

Debugging core serializer file for playbooks

bbf2da3

Debugging core serializer file for playbooks

722f300

Setting up test cases, urls and views for playbooks

167412f

Fixing playbook tests

34fc9e2

Adding playbooks urls

326e4da

Fixing playbooks urls

fc7047f

Debugging playbooks python_module

5cdcbad

Saving progress

5097914

Analyze request update.

dada182

Updated Job models

24b0e16

Cleaning playbooks_manager

2922df4

Cleaning playbooks_manager

b0687f7

Adding test playbook values

cfc99c9

Cleaning playbooks dataclass

d59b94d

Updating playbook_config.json for testin

47fe138

Debugging controller file

6344144

Debugging controller file

e04d42a

Debugging controller file

4d28516

Debugging controller file

96750fb

Adding playbook specific responses

9d7a9be

Debugging backend

7f25656

Optimising job reports for playbooks

cc9d67d

Adding frontend scanning support for IntelOwl.

9987bf5

0x0elliot added 3 commits October 5, 2022 04:42

Removing useless f strings

6d56dea

Removing analyzers which took too long from free playbook provided

ef3c153

Pushing playbooks down in github workflows

ada825d

Fixing frontend warnings

9e08b88

0x0elliot requested a review from mlodic October 5, 2022 00:02

mlodic mentioned this pull request Oct 6, 2022

Playbooks availability check #1225

Closed

mlodic reviewed Oct 6, 2022

View reviewed changes

tests/playbooks_manager/test_controller.py Outdated Show resolved Hide resolved

mlodic reviewed Oct 6, 2022

View reviewed changes

frontend/package.json Outdated Show resolved Hide resolved

0x0elliot added 7 commits October 6, 2022 21:41

Adding PR suggested changes

153857b

Adding instructions for contributors to add free analyzers in free an…

8b839f2

…alyzers playbooks

Adding instructions for contributors to add free analyzers in free an…

f2a3f0f

…alyzers playbooks

Letting analyzers fail in playbook tests

a13d4ae

Fixing linting

5d15bff

fixing playbook tests

9055e56

fixing linting errors

411bcf9

mlodic reviewed Oct 7, 2022

View reviewed changes

0x0elliot added 2 commits October 8, 2022 00:37

Squashing migrations together

ace29b3

Adding instructions in PR templates

a8f098b

adjusted migrations

fc77fe1

mlodic merged commit d4756f1 into develop Oct 10, 2022

mlodic deleted the Playbooks branch December 27, 2022 23:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Playbooks (solves #628) #1123

Playbooks (solves #628) #1123

0x0elliot commented Jul 23, 2022 •

edited

0x0elliot commented Oct 4, 2022

mlodic commented Oct 5, 2022

mlodic commented Oct 6, 2022 •

edited

mlodic commented Oct 6, 2022 •

edited by 0x0elliot

0x0elliot commented Oct 6, 2022

mlodic commented Oct 7, 2022

mlodic Oct 7, 2022

0x0elliot Oct 8, 2022

0x0elliot commented Oct 8, 2022

mlodic commented Oct 10, 2022

Playbooks (solves #628) #1123

Playbooks (solves #628) #1123

Conversation

0x0elliot commented Jul 23, 2022 • edited

Description

Checklist

Important Rules

Real World Example

0x0elliot commented Oct 4, 2022

mlodic commented Oct 5, 2022

mlodic commented Oct 6, 2022 • edited

mlodic commented Oct 6, 2022 • edited by 0x0elliot

0x0elliot commented Oct 6, 2022

mlodic commented Oct 7, 2022

mlodic Oct 7, 2022

Choose a reason for hiding this comment

0x0elliot Oct 8, 2022

Choose a reason for hiding this comment

0x0elliot commented Oct 8, 2022

mlodic commented Oct 10, 2022

0x0elliot commented Jul 23, 2022 •

edited

mlodic commented Oct 6, 2022 •

edited

mlodic commented Oct 6, 2022 •

edited by 0x0elliot