Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Plugin, Connector, controllers, dataclasses, tons of refactoring #524

Merged
merged 46 commits into from Jul 13, 2021
Merged

Plugin, Connector, controllers, dataclasses, tons of refactoring #524

merged 46 commits into from Jul 13, 2021

Conversation

sp35
Copy link
Member

@sp35 sp35 commented Jul 6, 2021 •
edited by m0mosenpai

Description

  • Refactor BaseAnalyzerMixin for reuse
  • Add base class BaseConnectorMixin and Connector
  • Add celery tasks to trigger connectors
  • Fix in ConnectorReport model
  • Add no of connectors called and connectors process time
  • Address issues and refactor
  • Move runtime_configuration field
  • Fix Analyzer Subclasses

Related issues

Fixes #512

Type of change

  • Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

  • The pull request is for the branch develop
  • I have added tests in the Tests folder.
  • The tests gave 0 errors.
  • Black gave 0 errors.
  • Flake gave 0 errors.
  • I squashed the commits into a single one. (optional, they will be squashed anyway by the maintainer)

@sp35 sp35 self-assigned this Jul 6, 2021
@sp35 sp35 linked an issue Jul 6, 2021 that may be closed by this pull request
[Connectors Manager] - Add base class and trigger connectors #512
Closed
@sp35 sp35 mentioned this pull request Jul 6, 2021
Merged
2 tasks
@sp35 sp35 requested a review from eshaan7 July 7, 2021 08:27
@eshaan7
Copy link
Member

eshaan7 commented Jul 7, 2021

Making changes to this.

@m0mosenpai m0mosenpai self-assigned this Jul 10, 2021
@m0mosenpai m0mosenpai added this to In progress in GSoC '21 - IntelOwl Improvements via automation Jul 10, 2021
sp35 and others added 18 commits July 11, 2021 21:04
@sp35 @m0mosenpai
Refactor BaseAnalyzerMixin to add connectors base class
a1b3b59
@sp35 @m0mosenpai
Fix: remove unique constraint from connector name
8dd57cf
@sp35 @m0mosenpai
Add tasks for triggering connectors
f1b8849 
On job success and individual connector runs
@sp35 @m0mosenpai
Add start connectors method
9a40b78
@sp35 @m0mosenpai
Add dummy misp connector
e723dad
@eshaan7 @m0mosenpai
core: Plugin class, _read_secrets in serializer
064812a
@eshaan7 @m0mosenpai
analyzers_manager: controller.py, modify classes, use Plugin class
90262b0
@eshaan7 @m0mosenpai
connectors_manager: controller.py, modify classes.py, use Plugin class
b781b37
@eshaan7 @m0mosenpai
intelowl.tasks: use controller modules, refactor
bd5d484
@eshaan7 @m0mosenpai
api_app: use controller module, refactor
c9e0e14
@eshaan7 @m0mosenpai
test_files: small adjust
b8e08d8
@m0mosenpai
moved runtime_config field to AnalyzerReport, updated migrations
7eb3fea
@m0mosenpai
misc fixes and refactoring (moved filter_analyzers() to controller.py)
94f60eb
@m0mosenpai
misc fixes
9b7bd6e
@m0mosenpai
removed extra general.py file
e33afa0
@m0mosenpai
fixes for observable analyzers
450d46b
@m0mosenpai
fixes for file analyzers
3f404fe
@m0mosenpai
misc fixes and refactoring
ec540fe
@intelowlproject intelowlproject deleted a comment from lgtm-com bot Jul 11, 2021
@intelowlproject intelowlproject deleted a comment from lgtm-com bot Jul 11, 2021
@intelowlproject intelowlproject deleted a comment from lgtm-com bot Jul 11, 2021
sp35
sp35 commented Jul 12, 2021
View reviewed changes
intel_owl/tasks.py Outdated Show resolved Hide resolved
@sp35
Fix: invalidate cache before status check
8aad3c2 
remove duplicate args for run_connector
@intelowlproject intelowlproject deleted a comment from lgtm-com bot Jul 13, 2021
@intelowlproject intelowlproject deleted a comment from lgtm-com bot Jul 13, 2021
@intelowlproject intelowlproject deleted a comment from lgtm-com bot Jul 13, 2021
@intelowlproject intelowlproject deleted a comment from lgtm-com bot Jul 13, 2021
@intelowlproject intelowlproject deleted a comment from lgtm-com bot Jul 13, 2021
@intelowlproject intelowlproject deleted a comment from lgtm-com bot Jul 13, 2021
@eshaan7 eshaan7 changed the title Plugin, Connector, controllers, refactoring Plugin, Connector, controllers, dataclasses, tons of refactoring Jul 13, 2021
@eshaan7 eshaan7 marked this pull request as ready for review July 13, 2021 17:49
code-review-doctor[bot]
code-review-doctor bot suggested changes Jul 13, 2021
View reviewed changes
Copy link
Contributor

@code-review-doctor code-review-doctor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Worth considering though

api_app/connectors_manager/models.py Show resolved Hide resolved
api_app/models.py Show resolved Hide resolved
@eshaan7 eshaan7 merged commit b3fda41 into develop-2 Jul 13, 2021
GSoC '21 - IntelOwl Improvements automation moved this from In progress to Done Jul 13, 2021
@eshaan7 eshaan7 deleted the trigger_connectors branch July 13, 2021 17:50
@lgtm-com
Copy link

lgtm-com bot commented Jul 13, 2021

This pull request introduces 12 alerts and fixes 1 when merging b388e58 into 88c51ad - view on LGTM.com

new alerts:

  • 5 for Potentially uninitialized local variable
  • 3 for Unhashable object hashed
  • 1 for Unused local variable
  • 1 for Unused import
  • 1 for Wrong number of arguments in a call
  • 1 for `__init__` method calls overridden method

fixed alerts:

  • 1 for `__init__` method calls overridden method

@eshaan7 eshaan7 linked an issue Jul 13, 2021 that may be closed by this pull request
Don't hardcode observable types in analyzer scripts #502
Closed
@sp35 sp35 moved this from In progress to Done in Connectors Manager/ connectors Jul 15, 2021
@eshaan7 eshaan7 mentioned this pull request Jul 30, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eshaan7 eshaan7 eshaan7 left review comments

@code-review-doctor code-review-doctor[bot] code-review-doctor requested changes

Assignees

@m0mosenpai m0mosenpai

@sp35 sp35

Labels
None yet
Projects
Connectors Manager/ connectors
Done
GSoC '21 - IntelOwl Improvements
Done
Milestone
No milestone
3 participants
@sp35 @eshaan7 @m0mosenpai