network-traffic Observable created from a malicious-activity indicator doesn't have any dst_ref and linked object

[guillaumededrie]

Description

When using OpenCTI import feature to ingest 2023-12-22-Indicator.json, the use of "x_opencti_create_observables": true STIX additional property, doesn't properly create a associated network-traffic observable that contains the observed port and IP address.

In our TIP, we ingest a lot of this type of indicators, which result having a lot of associated observables that are created but are the same and share the same STIX ID (and that aren't merged together).

We found that problem when generating report export.
An error was raised due to presence of multiple observables having the same STIX ID:

{"timestamp": "2023-12-08T14:27:13.032383Z", "level": "INFO", "name": "pycti.entities", "message": "Reading StixCyberObservable {network-traffic--8101c72f-3fab-572b-9378-d4a8de84ebb1}."}
{"timestamp": "2023-12-08T14:27:13.069880Z", "level": "ERROR", "name": "pycti.connector", "message": "Error in message processing, reporting error to API", "exc_info": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py\", line 279, in _data_handler\n    message = self.callback(json_data[\"event\"])\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/opencti-connector-export-file-stix/export-file-stix.py\", line 33, in _process_message\n    bundle = self.helper.api_impersonate.stix2.export_entity(\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/pycti/utils/opencti_stix2.py\", line 2082, in export_entity\n    stix_objects = self.prepare_export(\n                   ^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/pycti/utils/opencti_stix2.py\", line 1926, in prepare_export\n    entity_object_data = do_read(id=entity_object[\"id\"])\n                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/pycti/entities/opencti_stix_cyber_observable.py\", line 747, in read\n    result = self.opencti.query(query, {\"id\": id})\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 368, in query\n    raise ValueError(\nValueError: {'name': 'DatabaseError', 'message': '[SEARCH] Expect only one response'}"}

Environment

1. OS (where OpenCTI server runs): Docker Image running on Debian 12 (Docker version 24.0.7, build afdd53b)
2. OpenCTI version: 5.12.9

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Launch an import of 2023-12-22-Indicator.json in OpenCTI through "Data Import" page and ImportFileStix connector.
2. Validate the associated workbench.
3. Go to the newly created Indicator and navigate to the associated observable. The STIX ID should be network-traffic--8101c72f-3fab-572b-9378-d4a8de84ebb1, and there should not be any Nested Object on the Knowlegde tab.
4. Export (simple) the object.

Actual Output

{
    "type": "bundle",
    "id": "bundle--bd3e68bd-f40b-434a-971a-5133cd5ee1e7",
    "objects": [
        {
            "type": "marking-definition",
            "spec_version": "2.1",
            "id": "marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9",
            "created": "2017-01-20T00:00:00.000Z",
            "definition_type": "tlp",
            "name": "TLP:CLEAR",
            "definition": {
                "tlp": "clear"
            }
        },
        {
            "id": "network-traffic--8101c72f-3fab-572b-9378-d4a8de84ebb1",
            "spec_version": "2.1",
            "x_opencti_description": "Simple observable of indicator {Network traffic to 127.0.0.1 on port 443}",
            "x_opencti_score": 50,
            "dst_port": 443,
            "x_opencti_id": "e1b3e270-a7cc-499c-b48e-65f3d99a4065",
            "x_opencti_type": "Network-Traffic",
            "type": "network-traffic",
            "object_marking_refs": [
                "marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9"
            ]
        }
    ]
}

Expected Output

An observable should have been created if necessary (in our example, of "type": "ipv4-addr" and "value": "127.0.0.1") and linked back to the observable, see an export example:

{
    "type": "bundle",
    "id": "bundle--dd644d2a-ef7d-4ea3-879f-ecaa1af062fc",
    "objects": [
        {
            "type": "marking-definition",
            "spec_version": "2.1",
            "id": "marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9",
            "created": "2017-01-20T00:00:00.000Z",
            "definition_type": "tlp",
            "name": "TLP:CLEAR",
            "definition": {
                "tlp": "clear"
            }
        },
        {
            "id": "network-traffic--802413d4-407e-5013-bcee-77eab422b6f2",
            "spec_version": "2.1",
            "x_opencti_description": "Simple observable of indicator {Network traffic to 127.0.0.1 on port 443}",
            "x_opencti_score": 50,
            "dst_port": 443,
            "x_opencti_id": "e1b3e270-a7cc-499c-b48e-65f3d99a4065",
            "x_opencti_type": "Network-Traffic",
            "type": "network-traffic",
            "object_marking_refs": [
                "marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9"
            ],
+           "dst_ref": "ipv4-addr--679c6c82-b4be-52e2-9c7a-198689f6f77b"
        }
    ]
}

[Jipegien]

Multiple problems to note:

• we lose the IP during the Network Traffic Observable creation from the Indicator. -> IP in dst_ref
• representative value of Network Traffic Observable is not precise enough, causing deduplication problems -> representative should be more precise, like IP:port
• when creating an indicator from a Network traffic Observable, the main_observable_type is not network-traffic
• we need to verify if we are able to import a bundle containing dst_ref or src_ref.

[Jipegien]

We cannot fix it simply. Stix-pattern is not an explicit language. It is not a language made to be parsed, it is a language made to search something. There are too many situation where we cannot decide what to create.

So, this is not a bug. We do not know how to create dst_ref, src_ref and from_ref. To be handle that, we need to change our algo massively. #5584

We must stop to import inconsistent data. So we will discard observables creation for email and network traffic when dst_ref or src_ref or from_ref is in the pattern.