Merge pull request #2041 from intelowlproject/develop

* added book icon * Fix md Signed-off-by: 0ssigeno <s.berni@certego.net> * Adjusted toasts (#2015) * adjusted toast * fix test --------- Co-authored-by: Daniele Rosetti <d.rosetti@certego.net> * updated flare-floss to v3 * updated flare-floss to v3.0.1 * fixed the doc about contribute * fixed unresponsive scan page form with large files * changed social button popover trigger * fixed order problem in visualizer tabs * fixed PR template * fixed PR template * Frontend - HealthCheck and Pull buttons (#2025) * added healthcheck and pull button * test draft * frontend tests * adjusted toasts * added rule to short variable names (#2026) * added rule to short variable names * fix * fixed rules * fixed rules * added todo comment * fix * added note * fixed short var names * fixes * fix * fix * fix * comment * #1990 DNS0 /names Analyzer (#2024) * First not tested prototype * Added _monkeypatch() skeleton * Fixed a wrong condition in parameter validation * Added config() function to prepare data for request * Added hasattr() check for attributes * First successful run - Added check on existence of API parameter - Added authorization header - Added checks on parameters existence * Fix _api_name parameter check * Fixed parameter configuration * Removed unused if statement * Added support for UNIX timestamps and relative dates * Renamed dns0 service to separate services * Completed checks for not set parameters * Added checks of existence for all parameters * Added default Accept format header * Simplified regex to match single relative date * Fixed date check by forcing format * Modified _api_key to be non required * Made _api_key optional to support 10 free queries/day * Added /rrsets API * Moved files into separate dns0/ folder * Created module structure and base file * Extracted a common Mixin and applied to subclasses * Resolved performance issue flagged by deepsource * Added absolute import path * Substituted dateutils with dateparser and updated requirements * Updated Usage.md * Added explanation in Advanced-Usage.md * Fix dump Signed-off-by: 0ssigeno <s.berni@certego.net> * Black Signed-off-by: 0ssigeno <s.berni@certego.net> * Added analyzer config migrations * Added new analyzers to FREE_TO_USE_ANALYZERS playbook * Removed dns query type check * Removed unused variable 'query' * Added explicit default value in 'limit' parameter * Added DNS0 api docs reference * Solved deepsource warnings * Added custom user agent * Removed DNS0_rrsets_data * Solved deepsource warning * Added example JSON data * Generalization of ids Signed-off-by: 0ssigeno <s.berni@certego.net> * Fix dumpplugin migration generation Signed-off-by: 0ssigeno <s.berni@certego.net> * Excluding id Signed-off-by: 0ssigeno <s.berni@certego.net> * Excluding id in migration Signed-off-by: 0ssigeno <s.berni@certego.net> * Added default parameter in test * Added link to the DNS0 doc in plugin description * Fix pipeline test #1 * Fix pipeline test n2 * Fix pipeline test n3 * Fix pipeline test n4 * Fix pipeline test n5 * Fix pipeline test n6 * Fix pipeline test n7 * Fix pipeline test n8 * Fix pipeline test n9 * Fix pipeline test n10 * Fix pipeline test n11 * Fix pipeline test n12 * Fix pipeline test n13 * Fix pipeline test n14 * Fix pipeline test n15 * Fix pipeline test n16 * Fix pipeline test n17 * Removed unused q parameter --------- Signed-off-by: 0ssigeno <s.berni@certego.net> Co-authored-by: 0ssigeno <s.berni@certego.net> * Bump actions/setup-python from 4.7.0 to 5.0.0 (#2022) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4.7.0 to 5.0.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v4.7.0...v5.0.0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump google-cloud-webrisk from 1.12.0 to 1.13.0 in /requirements (#2018) Bumps [google-cloud-webrisk](https://github.com/googleapis/google-cloud-python) from 1.12.0 to 1.13.0. - [Release notes](https://github.com/googleapis/google-cloud-python/releases) - [Changelog](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-documentai/CHANGELOG.md) - [Commits](googleapis/google-cloud-python@google-cloud-webrisk-v1.12.0...google-cloud-webrisk-v1.13.0) --- updated-dependencies: - dependency-name: google-cloud-webrisk dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Corner case migration Signed-off-by: 0ssigeno <s.berni@certego.net> * More logs Signed-off-by: 0ssigeno <s.berni@certego.net> * Fix Signed-off-by: 0ssigeno <s.berni@certego.net> * incremented logs in most used apis * Elastic bi (#2036) * Elastic bi Signed-off-by: 0ssigeno <s.berni@certego.net> * Elastic bi + fix elastic package Signed-off-by: 0ssigeno <s.berni@certego.net> * Update api_app/queryset.py Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com> * Fix Signed-off-by: 0ssigeno <s.berni@certego.net> * Index template Signed-off-by: 0ssigeno <s.berni@certego.net> * Fixes Signed-off-by: 0ssigeno <s.berni@certego.net> --------- Signed-off-by: 0ssigeno <s.berni@certego.net> Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com> * changelog + docs + bump + django doctor + deepsource * Bump github/codeql-action from 2 to 3 (#2033) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3. - [Release notes](https://github.com/github/codeql-action/releases) - [Commits](github/codeql-action@v2...v3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump authlib from 1.2.0 to 1.3.0 in /requirements (#2031) Bumps [authlib](https://github.com/lepture/authlib) from 1.2.0 to 1.3.0. - [Release notes](https://github.com/lepture/authlib/releases) - [Changelog](https://github.com/lepture/authlib/blob/master/docs/changelog.rst) - [Commits](lepture/authlib@v1.2.0...v1.3.0) --- updated-dependencies: - dependency-name: authlib dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fixed permissions and log files for ClamAV * Added IP and subdomain support for DNS0_rrsets_data (#2042) * Added IP support for DNS0_rrsets_data analyzer * Added include_subdomain parameter * Typo * Restore original state * Added alter migration to add a new supported type and new parameter * fix deepsource --------- Signed-off-by: 0ssigeno <s.berni@certego.net> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Daniele Rosetti <d.rosetti@certego.net> Co-authored-by: 0ssigeno <s.berni@certego.net> Co-authored-by: Martina Carella <m.carella@certego.net> Co-authored-by: Daniele Rosetti <55402684+drosetti@users.noreply.github.com> Co-authored-by: fgibertoni <152909479+fgibertoni@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Simone Berni <simone.berni2@studio.unibo.it> Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
intelowlproject · Dec 27, 2023 · 872b113 · 872b113
2 parents abad9cc + aefb8bb
commit 872b113
Show file tree

Hide file tree

Showing 125 changed files with 2,941 additions and 1,022 deletions.
diff --git a/.github/CHANGELOG.md b/.github/CHANGELOG.md
@@ -2,8 +2,24 @@
 
 [**Upgrade Guide**](https://intelowl.readthedocs.io/en/latest/Installation.md#update-to-the-most-recent-version)
 
+## [v5.2.2](https://github.com/intelowlproject/IntelOwl/releases/tag/v5.2.2)
+
+This release has been done mainly to adjusts a broken database migration introduced in the previous release.
+
+**Main Improvements**
+* Added new analyzers for [DNS0](https://docs.dns0.eu/) PassiveDNS data
+* Added the chance to collect metrics ([Business Intelligence](https://intelowl.readthedocs.io/en/develop/Advanced-Configuration.html#business-intelligence) regarding Plugins Usage and send it to an ElasticSearch instance.
+* Added new buttons to test ["Healthcheck" and "Pull" operations](https://intelowl.readthedocs.io/en/latest/Usage.html#special-plugins-operations) for each Plugin (A feature introduced in the previous version)
+
+**Other improvements**
+* Various generic fixes and adjustments in the GUI
+* dependencies upgrades
+* adjusted contribution guides
+
 ## [v5.2.1](https://github.com/intelowlproject/IntelOwl/releases/tag/v5.2.1)
 
+!!! This release has been found with a broken database migration !!! Please upgrade to v5.2.2 to fix the problem.
+
 **General improvements**
 * Incremented wait time of containers' healthchecks to avoid to break clean installations
 * Improvements to the "Scan page":

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -19,17 +19,17 @@ Please delete options that are not relevant.
 - [ ] A new plugin (analyzer, connector, visualizer, playbook, pivot or ingestor) was added or changed, in which case:
     - [ ] I strictly followed the documentation ["How to create a Plugin"](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-add-a-new-plugin)
     - [ ] [Usage](https://github.com/intelowlproject/IntelOwl/blob/master/docs/source/Usage.md) file was updated.
-    - [ ] [Advanced-Usage](./Advanced-Usage.md) was updated (in case the plugin provides additional optional configuration).
+    - [ ] [Advanced-Usage](https://github.com/intelowlproject/IntelOwl/blob/master/docs/source/Advanced-Usage.md) was updated (in case the plugin provides additional optional configuration).
     - [ ] If the plugin requires mocked testing, `_monkeypatch()` was used in its class to apply the necessary decorators.
-    - [ ] I have dumped the configuration from Django Admin using the `dumpplugin` command and added it in the project as a data migration. [Doc](["How to create a Plugin"](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-add-a-new-plugin))
+    - [ ] I have dumped the configuration from Django Admin using the `dumpplugin` command and added it in the project as a data migration. (["How to share a plugin with the community"](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-share-your-plugin-with-the-community))
     - [ ] If a File analyzer was added and it supports a mimetype which is not already supported, you added a sample of that type inside the archive `test_files.zip` and you added the default tests for that mimetype in [test_classes.py](https://github.com/intelowlproject/IntelOwl/blob/master/tests/analyzers_manager/test_classes.py).
-    - [ ] If you created a new analyzer and it is free (does not require API keys), please add it in the `FREE_TO_USE_ANALYZERS` playbook in `playbook_config.json`.
+    - [ ] If you created a new analyzer and it is free (does not require API keys), please add it in the `FREE_TO_USE_ANALYZERS` playbook by following [this guide](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-modify-a-plugin).
     - [ ] Check if it could make sense to add that analyzer/connector to other [freely available playbooks](https://intelowl.readthedocs.io/en/develop/Usage.html#list-of-pre-built-playbooks).
     - [ ] I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
 - [ ] If external libraries/packages with restrictive licenses were used, they were added in the [Legal Notice](https://github.com/certego/IntelOwl/blob/master/.github/legal_notice.md) section.
 - [ ] Linters (`Black`, `Flake`, `Isort`) gave 0 errors. If you have correctly installed [pre-commit](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-start-setup-project-and-development-instance), it does these checks and adjustments on your behalf.
 - [ ] I have added tests for the feature/bug I solved (see `tests` folder). All the tests (new and old ones) gave 0 errors.
-- [ ] If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check [CONTRIBUTE.md](./Contribute.md)).
+- [ ] If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check [CONTRIBUTE.md](https://github.com/intelowlproject/IntelOwl/blob/master/docs/source/Contribute.md)).
 - [ ] If the GUI has been modified:
     - [ ] I have a provided a screenshot of the result in the PR.
     - [ ] I have created new frontend tests for the new component or updated existing ones.

diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml
@@ -44,7 +44,7 @@ jobs:
         fetch-depth: 2
 
     - name: Set up Python
-      uses: actions/setup-python@v4.7.0
+      uses: actions/setup-python@v5.0.0
       with:
         python-version: '3.9'
 
@@ -65,7 +65,7 @@ jobs:
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v2
+      uses: github/codeql-action/init@v3
       with:
         languages: python
         # Override the default behavior so that the action doesn't attempt
@@ -93,4 +93,4 @@ jobs:
     #   make release
 
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v2
+      uses: github/codeql-action/analyze@v3
diff --git a/.github/workflows/pull_request_automation.yml b/.github/workflows/pull_request_automation.yml
@@ -37,7 +37,7 @@ jobs:
         uses: actions/checkout@v4.1.0
 
       - name: Set up Python
-        uses: actions/setup-python@v4.7.0
+        uses: actions/setup-python@v5.0.0
         with:
           python-version: 3.9
 

diff --git a/.github/workflows/scorecard.yml b/.github/workflows/scorecard.yml
@@ -67,6 +67,6 @@ jobs:
 
       # Upload the results to GitHub's code scanning dashboard.
       - name: "Upload to code-scanning"
-        uses: github/codeql-action/upload-sarif@807578363a7869ca324a79039e6db9c843e0e100 # v2.1.27
+        uses: github/codeql-action/upload-sarif@03e7845b7bfcd5e7fb63d1ae8c61b0e791134fab # v2.22.11
         with:
           sarif_file: results.sarif
diff --git a/.github/workflows/sphinx_build_on_release.yml b/.github/workflows/sphinx_build_on_release.yml
@@ -14,7 +14,7 @@ jobs:
     steps:
       - uses: actions/checkout@v4.1.0
       - name: Set up Python
-        uses: actions/setup-python@v4.7.0
+        uses: actions/setup-python@v5.0.0
         with:
           python-version: 3.9
       - name: Install docs requirements

diff --git a/api_app/analyzers_manager/migrations/0052_analyzer_config_dns0_names.py b/api_app/analyzers_manager/migrations/0052_analyzer_config_dns0_names.py
@@ -0,0 +1,209 @@
+from django.db import migrations
+from django.db.models.fields.related_descriptors import (
+    ForwardManyToOneDescriptor,
+    ForwardOneToOneDescriptor,
+    ManyToManyDescriptor,
+)
+
+plugin = {
+    "name": "DNS0_names",
+    "python_module": {
+        "module": "dns0.dns0_names.DNS0Names",
+        "base_path": "api_app.analyzers_manager.observable_analyzers",
+    },
+    "description": "Run advanced searches on billions of current and historical domain names. [API](https://docs.dns0.eu/dns-api/names).",
+    "disabled": False,
+    "soft_time_limit": 60,
+    "routing_key": "default",
+    "health_check_status": True,
+    "type": "observable",
+    "docker_based": False,
+    "maximum_tlp": "RED",
+    "observable_supported": ["url", "domain", "generic"],
+    "supported_filetypes": [],
+    "run_hash": False,
+    "run_hash_type": "",
+    "not_supported_filetypes": [],
+    "health_check_task": None,
+    "model": "analyzers_manager.AnalyzerConfig",
+}
+
+params = [
+    {
+        "name": "api_key",
+        "type": "str",
+        "description": "",
+        "is_secret": True,
+        "required": False,
+        "python_module": {
+            "module": "dns0.dns0_names.DNS0Names",
+            "base_path": "api_app.analyzers_manager.observable_analyzers",
+        },
+    },
+    {
+        "name": "root",
+        "type": "bool",
+        "description": "Limit results to root domains.",
+        "is_secret": False,
+        "required": False,
+        "python_module": {
+            "module": "dns0.dns0_names.DNS0Names",
+            "base_path": "api_app.analyzers_manager.observable_analyzers",
+        },
+    },
+    {
+        "name": "fuzzy",
+        "type": "list",
+        "description": "Apply fuzziness to q. Accepts a comma-separated list of fuzzy algorithms, or all to apply them all.",
+        "is_secret": False,
+        "required": False,
+        "python_module": {
+            "module": "dns0.dns0_names.DNS0Names",
+            "base_path": "api_app.analyzers_manager.observable_analyzers",
+        },
+    },
+    {
+        "name": "from",
+        "type": "str",
+        "description": "Limit results to names seen after this date.",
+        "is_secret": False,
+        "required": False,
+        "python_module": {
+            "module": "dns0.dns0_names.DNS0Names",
+            "base_path": "api_app.analyzers_manager.observable_analyzers",
+        },
+    },
+    {
+        "name": "to",
+        "type": "str",
+        "description": "Limit results to names seen before this date.",
+        "is_secret": False,
+        "required": False,
+        "python_module": {
+            "module": "dns0.dns0_names.DNS0Names",
+            "base_path": "api_app.analyzers_manager.observable_analyzers",
+        },
+    },
+    {
+        "name": "not_before",
+        "type": "str",
+        "description": "Limit results to names not seen before this date.",
+        "is_secret": False,
+        "required": False,
+        "python_module": {
+            "module": "dns0.dns0_names.DNS0Names",
+            "base_path": "api_app.analyzers_manager.observable_analyzers",
+        },
+    },
+    {
+        "name": "sort",
+        "type": "str",
+        "description": "Available sorts are first_seen (the default) or last_seen. Both are descending sorts (most recent first).",
+        "is_secret": False,
+        "required": False,
+        "python_module": {
+            "module": "dns0.dns0_names.DNS0Names",
+            "base_path": "api_app.analyzers_manager.observable_analyzers",
+        },
+    },
+    {
+        "name": "format",
+        "type": "str",
+        "description": "Available formats are json and dig. Default format is based on the Accept HTTP header.",
+        "is_secret": False,
+        "required": False,
+        "python_module": {
+            "module": "dns0.dns0_names.DNS0Names",
+            "base_path": "api_app.analyzers_manager.observable_analyzers",
+        },
+    },
+    {
+        "name": "limit",
+        "type": "int",
+        "description": "Limit the number of results.",
+        "is_secret": False,
+        "required": False,
+        "python_module": {
+            "module": "dns0.dns0_names.DNS0Names",
+            "base_path": "api_app.analyzers_manager.observable_analyzers",
+        },
+    },
+    {
+        "name": "offset",
+        "type": "int",
+        "description": "Used for pagination.",
+        "is_secret": False,
+        "required": False,
+        "python_module": {
+            "module": "dns0.dns0_names.DNS0Names",
+            "base_path": "api_app.analyzers_manager.observable_analyzers",
+        },
+    },
+]
+
+values = []
+
+
+def _get_real_obj(Model, field, value):
+    if (
+        type(getattr(Model, field))
+        in [ForwardManyToOneDescriptor, ForwardOneToOneDescriptor]
+        and value
+    ):
+        other_model = getattr(Model, field).get_queryset().model
+        # in case is a dictionary, we have to retrieve the object with every key
+        if isinstance(value, dict):
+            real_vals = {}
+            for key, real_val in value.items():
+                real_vals[key] = _get_real_obj(other_model, key, real_val)
+            value = other_model.objects.get_or_create(**real_vals)[0]
+        # it is just the primary key serialized
+        else:
+            value = other_model.objects.get(pk=value)
+    return value
+
+
+def _create_object(Model, data):
+    mtm, no_mtm = {}, {}
+    for field, value in data.items():
+        if type(getattr(Model, field)) is ManyToManyDescriptor:
+            mtm[field] = value
+        else:
+            value = _get_real_obj(Model, field, value)
+            no_mtm[field] = value
+    try:
+        o = Model.objects.get(**no_mtm)
+    except Model.DoesNotExist:
+        o = Model(**no_mtm)
+        o.full_clean()
+        o.save()
+        for field, value in mtm.items():
+            attribute = getattr(o, field)
+            attribute.set(value)
+
+
+def migrate(apps, schema_editor):
+    Parameter = apps.get_model("api_app", "Parameter")
+    PluginConfig = apps.get_model("api_app", "PluginConfig")
+    python_path = plugin.pop("model")
+    Model = apps.get_model(*python_path.split("."))
+    _create_object(Model, plugin)
+    for param in params:
+        _create_object(Parameter, param)
+    for value in values:
+        _create_object(PluginConfig, value)
+
+
+def reverse_migrate(apps, schema_editor):
+    python_path = plugin.pop("model")
+    Model = apps.get_model(*python_path.split("."))
+    Model.objects.get(name=plugin["name"]).delete()
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("api_app", "0051_pythonmodule_health_check_schedule_and_more"),
+        ("analyzers_manager", "0051_analyzerreport_parameters"),
+    ]
+
+    operations = [migrations.RunPython(migrate, reverse_migrate)]