Merge pull request #2321 from intelowlproject/develop

* updated yeti analyzer and connector to support new major * updated default pycti version * fixed MaxMind data extraction for the country flag * Fix pivot + file Signed-off-by: 0ssigeno <s.berni@certego.net> * healthcheck available for Plugins with `url` option by default (#2320) * healthcheck available for Plugins with `url` option * doc * fix * Bump quark-engine from 24.4.1 to 24.5.1 in /requirements (#2313) Bumps [quark-engine](https://github.com/quark-engine/quark-engine) from 24.4.1 to 24.5.1. - [Release notes](https://github.com/quark-engine/quark-engine/releases) - [Commits](quark-engine/quark-engine@v24.4.1...v24.5.1) --- updated-dependencies: - dependency-name: quark-engine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump jsonschema from 4.21.1 to 4.22.0 in /requirements (#2311) Bumps [jsonschema](https://github.com/python-jsonschema/jsonschema) from 4.21.1 to 4.22.0. - [Release notes](https://github.com/python-jsonschema/jsonschema/releases) - [Changelog](https://github.com/python-jsonschema/jsonschema/blob/main/CHANGELOG.rst) - [Commits](python-jsonschema/jsonschema@v4.21.1...v4.22.0) --- updated-dependencies: - dependency-name: jsonschema dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump docutils from 0.20.1 to 0.21.2 in /requirements (#2312) Bumps [docutils](https://docutils.sourceforge.io) from 0.20.1 to 0.21.2. --- updated-dependencies: - dependency-name: docutils dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Revert "Bump docutils from 0.20.1 to 0.21.2 in /requirements (#2312)" This reverts commit 9e5106e. * prettier * changes (#2322) * Phoneinfoga analyzer adjustment (#2324) * Phoneinfoga Signed-off-by: 0ssigeno <s.berni@certego.net> * Linters Signed-off-by: 0ssigeno <s.berni@certego.net> * adjusted phoneinfoga * Update api_app/analyzers_manager/migrations/0088_phoneinfoga_parameters.py --------- Signed-off-by: 0ssigeno <s.berni@certego.net> Co-authored-by: Matteo Lodi <30625432+mlodic@users.noreply.github.com> * Fix serializer Signed-off-by: 0ssigeno <s.berni@certego.net> * Fix sender Signed-off-by: 0ssigeno <s.berni@certego.net> * pcap_analyzers adjusts + new playbook for PCAP files + upgraded Suricata to v7 (#2325) * pcap_analyzers adjusts + new playbook for PCAP files + upgraded Suricata to v7 * adjusted hfinger * adjust test * adjust test and upgraded watchman * tests * fix custom analysis (#2323) * hudsonrock (#2327) * hudsonrock * tests * test * add params * migration * tests * migration * i always overlook this lol * tlp to amber --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * Update api_app/analyzers_manager/observable_analyzers/hudsonrock.py Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com> * black * Fixes frontend regex (#2329) * support phone numbers * moved phone number validation to E.164 format * removed dates from parsing as IP addresses * prettier * Cy cat#1479 (#2328) * cycat * cycat * cycat wrapper done * migration * docs * tests * tests --------- Co-authored-by: g4ze <bhaiyajionline@gmail.com> * updated changelog * fix loading visualizer navbar (#2335) * fix visualizer loading * changes * --- (#2332) updated-dependencies: - dependency-name: celery[redis,sqs] dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * --- (#2334) updated-dependencies: - dependency-name: intezer-sdk dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * --- (#2333) updated-dependencies: - dependency-name: docutils dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Speed up (#2336) Signed-off-by: 0ssigeno <s.berni@certego.net> * Revert "--- (#2333)" This reverts commit 12802eb. --------- Signed-off-by: 0ssigeno <s.berni@certego.net> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Daniele Rosetti <d.rosetti@certego.net> Co-authored-by: 0ssigeno <s.berni@certego.net> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Martina Carella <m.carella@certego.net> Co-authored-by: Simone Berni <simone.berni2@studio.unibo.it> Co-authored-by: Nilay Gupta <102874321+g4ze@users.noreply.github.com> Co-authored-by: g4ze <bhaiyajionline@gmail.com> Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com> Co-authored-by: Daniele Rosetti <55402684+drosetti@users.noreply.github.com>
intelowlproject · May 21, 2024 · f13a0d3 · f13a0d3
2 parents 98197f7 + 824b8f4
commit f13a0d3
Show file tree

Hide file tree

Showing 61 changed files with 1,654 additions and 301 deletions.
diff --git a/.github/CHANGELOG.md b/.github/CHANGELOG.md
@@ -10,6 +10,7 @@ We added **Pivot** buttons to enable manual Pivoting from an Observable/File ana
 As usual, we add new plugins. This release brings the following new ones:
 * a complete **TakedownRequest** playbook to automate TakeDown requests for malicious domains
 * new File Analyzers for tools like [HFinger](https://github.com/CERT-Polska/hfinger), [Permhash](https://github.com/google/permhash) and [Blint](https://github.com/owasp-dep-scan/blint)
+* new Observable Analyzers for [CyCat](https://cycat.org/) and [Hudson Rock](https://cavalier.hudsonrock.com/docs)
 * improvement of the existing Maxmind analyzer: it now downloads the ASN database too.
 
 ## [v6.0.1](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.0.1)

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -20,12 +20,13 @@ Please delete options that are not relevant.
     - [ ] I strictly followed the documentation ["How to create a Plugin"](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-add-a-new-plugin)
     - [ ] [Usage](https://github.com/intelowlproject/IntelOwl/blob/master/docs/source/Usage.md) file was updated.
     - [ ] [Advanced-Usage](https://github.com/intelowlproject/IntelOwl/blob/master/docs/source/Advanced-Usage.md) was updated (in case the plugin provides additional optional configuration).
-    - [ ] If the plugin requires mocked testing, `_monkeypatch()` was used in its class to apply the necessary decorators.
     - [ ] I have dumped the configuration from Django Admin using the `dumpplugin` command and added it in the project as a data migration. (["How to share a plugin with the community"](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-share-your-plugin-with-the-community))
     - [ ] If a File analyzer was added and it supports a mimetype which is not already supported, you added a sample of that type inside the archive `test_files.zip` and you added the default tests for that mimetype in [test_classes.py](https://github.com/intelowlproject/IntelOwl/blob/master/tests/analyzers_manager/test_classes.py).
-    - [ ] If you created a new analyzer and it is free (does not require API keys), please add it in the `FREE_TO_USE_ANALYZERS` playbook by following [this guide](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-modify-a-plugin).
+    - [ ] If you created a new analyzer and it is free (does not require any API key), please add it in the `FREE_TO_USE_ANALYZERS` playbook by following [this guide](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-modify-a-plugin).
     - [ ] Check if it could make sense to add that analyzer/connector to other [freely available playbooks](https://intelowl.readthedocs.io/en/develop/Usage.html#list-of-pre-built-playbooks).
-    - [ ] I have provided the resulting raw JSON of a finished analysis and a screenshot of the results. 
+    - [ ] I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
+    - [ ] If the plugin interacts with an external service, I have created an attribute called precisely `url` that contains this information. This is required for Health Checks. 
+    - [ ] If the plugin requires mocked testing, `_monkeypatch()` was used in its class to apply the necessary decorators.
     - [ ] I have added that raw JSON sample to the `MockUpResponse` of the `_monkeypatch()` method. This serves us to provide a valid sample for testing.
 - [ ] If external libraries/packages with restrictive licenses were used, they were added in the [Legal Notice](https://github.com/certego/IntelOwl/blob/master/.github/legal_notice.md) section.
 - [ ] Linters (`Black`, `Flake`, `Isort`) gave 0 errors. If you have correctly installed [pre-commit](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-start-setup-project-and-development-instance), it does these checks and adjustments on your behalf.

diff --git a/api_app/analyzers_manager/file_analyzers/docguard.py b/api_app/analyzers_manager/file_analyzers/docguard.py
@@ -13,7 +13,7 @@
 
 
 class DocGuardUpload(FileAnalyzer):
-    base_url = "https://api.docguard.io:8443/api"
+    url = "https://api.docguard.io:8443/api"
     _api_key_name: str
 
     def run(self):
@@ -31,7 +31,7 @@ def run(self):
         if not binary:
             raise AnalyzerRunException("File is empty")
         response = requests.post(
-            self.base_url + "/FileAnalyzing/AnalyzeFile",
+            self.url + "/FileAnalyzing/AnalyzeFile",
             headers=headers,
             files={"file": (self.filename, binary)},
         )

diff --git a/api_app/analyzers_manager/file_analyzers/filescan.py b/api_app/analyzers_manager/file_analyzers/filescan.py
@@ -18,7 +18,7 @@ class FileScanUpload(FileAnalyzer):
 
     max_tries: int = 30
     poll_distance: int = 10
-    base_url = "https://www.filescan.io/api"
+    url = "https://www.filescan.io/api"
     _api_key: str
 
     def run(self):
@@ -31,7 +31,7 @@ def __upload_file_for_scan(self) -> int:
         if not binary:
             raise AnalyzerRunException("File is empty")
         response = requests.post(
-            self.base_url + "/scan/file",
+            self.url + "/scan/file",
             files={"file": (self.filename, binary)},
             headers={"X-Api-Key": self._api_key},
         )
@@ -41,7 +41,7 @@ def __upload_file_for_scan(self) -> int:
 
     def __fetch_report(self, task_id: int) -> dict:
         report = {}
-        url = f"{self.base_url}/scan/{task_id}/report"
+        url = f"{self.url}/scan/{task_id}/report"
         params = {
             "filter": [
                 "general",

diff --git a/api_app/analyzers_manager/file_analyzers/hfinger.py b/api_app/analyzers_manager/file_analyzers/hfinger.py
@@ -16,7 +16,17 @@ class Hfinger(FileAnalyzer):
     fingerprint_report_mode: int = 2
 
     def run(self):
-        return hfinger_analyze(self.filepath, self.fingerprint_report_mode)
+        reports = dict()
+        reports["extraction"] = hfinger_analyze(
+            self.filepath, self.fingerprint_report_mode
+        )
+        fingerprints = set()
+        for report in reports["extraction"]:
+            fingerprint = report.get("fingerprint", "")
+            if fingerprint:
+                fingerprints.add(fingerprint)
+        reports["fingerprints_summary"] = list(fingerprints)
+        return reports
 
     @classmethod
     def update(cls) -> bool:

diff --git a/api_app/analyzers_manager/file_analyzers/malpedia_scan.py b/api_app/analyzers_manager/file_analyzers/malpedia_scan.py
@@ -12,8 +12,8 @@ class MalpediaScan(FileAnalyzer):
     Scan a binary against all YARA rules in Malpedia.
     """
 
-    base_url = "https://malpedia.caad.fkie.fraunhofer.de/api"
-    url = base_url + "/scan/binary"
+    url = "https://malpedia.caad.fkie.fraunhofer.de/api"
+    binary_url = url + "/scan/binary"
 
     _api_key_name: str
 
@@ -23,7 +23,7 @@ def run(self):
         # construct req
         headers = {"Authorization": f"APIToken {self._api_key_name}"}
         files = {"file": binary}
-        response = requests.post(self.url, headers=headers, files=files)
+        response = requests.post(self.binary_url, headers=headers, files=files)
         response.raise_for_status()
 
         result = response.json()

diff --git a/api_app/analyzers_manager/file_analyzers/triage_scan.py b/api_app/analyzers_manager/file_analyzers/triage_scan.py
@@ -26,7 +26,7 @@ def run(self):
         logger.info(f"triage md5 {self.md5} sending sample for analysis")
         for _try in range(self.max_tries):
             logger.info(f"triage md5 {self.md5} polling for result try #{_try + 1}")
-            self.response = self.session.post(self.base_url + "samples", files=files)
+            self.response = self.session.post(self.url + "samples", files=files)
             if self.response.status_code == 200:
                 break
             time.sleep(self.poll_distance)

diff --git a/api_app/analyzers_manager/file_analyzers/unpac_me.py b/api_app/analyzers_manager/file_analyzers/unpac_me.py
@@ -15,7 +15,7 @@
 
 
 class UnpacMe(FileAnalyzer):
-    base_url: str = "https://api.unpac.me/api/v1/"
+    url: str = "https://api.unpac.me/api/v1/"
 
     _api_key_name: str
     private: bool
@@ -60,12 +60,10 @@ def run(self):
     def _req_with_checks(self, url, files=None, post=False):
         try:
             if post:
-                r = requests.post(
-                    self.base_url + url, files=files, headers=self.headers
-                )
+                r = requests.post(self.url + url, files=files, headers=self.headers)
             else:
                 headers = self.headers if self.private == "private" else {}
-                r = requests.get(self.base_url + url, files=files, headers=headers)
+                r = requests.get(self.url + url, files=files, headers=headers)
             r.raise_for_status()
         except requests.exceptions.HTTPError as e:
             logger.error(

diff --git a/api_app/analyzers_manager/file_analyzers/virushee.py b/api_app/analyzers_manager/file_analyzers/virushee.py
@@ -19,7 +19,7 @@ class VirusheeFileUpload(FileAnalyzer):
 
     max_tries = 30
     poll_distance = 10
-    base_url = "https://api.virushee.com"
+    url = "https://api.virushee.com"
 
     _api_key_name: str
 
@@ -46,7 +46,7 @@ def run(self):
 
     def __check_report_for_hash(self) -> Optional[dict]:
         response_json = None
-        response = self.__session.get(f"{self.base_url}/file/hash/{self.md5}")
+        response = self.__session.get(f"{self.url}/file/hash/{self.md5}")
         if response.status_code == 404:  # hash not found in db
             return response_json
         response.raise_for_status()
@@ -57,13 +57,13 @@ def __check_report_for_hash(self) -> Optional[dict]:
     def __upload_file(self, binary: bytes) -> str:
         name_to_send = self.filename if self.filename else self.md5
         files = {"file": (name_to_send, binary)}
-        response = self.__session.post(f"{self.base_url}/file/upload", files=files)
+        response = self.__session.post(f"{self.url}/file/upload", files=files)
         response.raise_for_status()
         return response.json()["task"]
 
     def __poll_status_and_result(self, task_id: str) -> dict:
         response_json = None
-        url = f"{self.base_url}/file/task/{task_id}"
+        url = f"{self.url}/file/task/{task_id}"
         for chance in range(self.max_tries):
             logger.info(f"Polling try#{chance+1}")
             response = self.__session.get(url)

diff --git a/api_app/analyzers_manager/migrations/0087_alter_mmdbserver_param.py b/api_app/analyzers_manager/migrations/0087_alter_mmdbserver_param.py
@@ -0,0 +1,26 @@
+from django.db import migrations
+
+
+def migrate(apps, schema_editor):
+    PythonModule = apps.get_model("api_app", "PythonModule")
+
+    pm = PythonModule.objects.get(
+        module="mmdb_server.MmdbServer",
+        base_path="api_app.analyzers_manager.observable_analyzers",
+    )
+    param = pm.parameters.get(name="base_url")
+    param.name = "url"
+    param.save()
+
+
+def reverse_migrate(apps, schema_editor):
+    pass
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("analyzers_manager", "0086_analyzer_config_blint"),
+    ]
+    operations = [
+        migrations.RunPython(migrate, reverse_migrate),
+    ]
diff --git a/api_app/analyzers_manager/migrations/0088_phoneinfoga_parameters.py b/api_app/analyzers_manager/migrations/0088_phoneinfoga_parameters.py
@@ -0,0 +1,53 @@
+from django.db import migrations
+
+
+def migrate(apps, schema_editor):
+    Parameter = apps.get_model("api_app", "Parameter")
+    PluginConfig = apps.get_model("api_app", "PluginConfig")
+    PythonModule = apps.get_model("api_app", "PythonModule")
+    pm = PythonModule.objects.get(
+        module="phoneinfoga_scan.Phoneinfoga",
+        base_path="api_app.analyzers_manager.observable_analyzers",
+    )
+    Parameter.objects.create(
+        name="googlecse_max_results",
+        type="int",
+        description="Number of Google results for [Phoneinfoga](https://sundowndev.github.io/phoneinfoga/)",
+        is_secret=False,
+        required=False,
+        python_module=pm,
+    )
+    p2 = Parameter.objects.create(
+        name="scanners",
+        type="list",
+        description="List of scanner names for [Phoneinfoga](https://sundowndev.github.io/phoneinfoga/). Available options are: `local,numverify,googlecse,ovh`",
+        is_secret=False,
+        required=False,
+        python_module=pm,
+    )
+    p3 = Parameter.objects.get(name="scanner_name", python_module=pm)
+    for config in pm.analyzerconfigs.all():
+        pcs = PluginConfig.objects.filter(analyzer_config=config, parameter=p3)
+        for pc in pcs:
+            pc.value = [pc.value]
+            pc.parameter = p2
+            pc.save()
+    p3.delete()
+    Parameter.objects.create(
+        name="all_scanners",
+        type="bool",
+        description="Set this to True to enable all available scanners. "
+        "If enabled, this overwrite the scanner param",
+        is_secret=False,
+        required=False,
+        python_module=pm,
+    )
+
+
+class Migration(migrations.Migration):
+    atomic = False
+    dependencies = [
+        ("analyzers_manager", "0087_alter_mmdbserver_param"),
+    ]
+
+    operations = [migrations.RunPython(migrate, migrations.RunPython.noop)]