create file observables with fuzzy hashes

[mangusm]

Description

Attempting to create a File observable with only fuzzy hashes results in an error

Environment

1. OS Ubuntu 20.04
2. OpenCTI version: 5.12.21
3. OpenCTI client: graphql
4. Other environment details: N/A

Reproducible Steps

Steps to create the smallest reproducible scenario:

helper = OpenCTIConnectorHelper(config)

work_id = helper.api.work.initiate_work(
    helper.connect_id,
    str(uuid.uuid4()),
)

f = stix2.File(
    name="test",
    hashes={
        "ssdeep": "12:yfNlC+jKQ/NuTD9jDuMZGo/51zNrtt+jKUXnk6lSUmTp8nnfglH/:eljRI9RZN/TzpejPXVlS3d0gF",  # hash of this file
    },
)

bundle = stix2.Bundle(objects=[f]).serialize()
helper.send_stix2_bundle(bundle=bundle, work_id=work_id)

Expected Output

A File observable with an SSDEEP hash

Actual Output

{
    "category": "APP",
    "errors": [
        {
            "attributes": {
                "genre": "BUSINESS",
                "http_status": 500,
                "key": "hashes.*",
                "operator": "wildcard",
                "values": []
            },
            "message": "Invalid filter configuration",
            "name": "UNSUPPORTED_ERROR",
            "stack": "UNSUPPORTED_ERROR: Invalid filter configuration\n    at error (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/config/errors.js:8:10)\n    at UnsupportedError (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/config/errors.js:77:51)\n    at buildLocalMustFilter (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/database/engine.js:1650:9)\n    at buildSubQueryForFilterGroup (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/database/engine.js:1669:37)\n    at buildSubQueryForFilterGroup (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/database/engine.js:1660:30)\n    at elQueryBodyBuilder (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/database/engine.js:2097:35)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at elPaginate (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/database/engine.js:2464:16)"
        }
    ],
    "inner_relation_creation": 0,
    "level": "error",
    "message": "Invalid filter configuration",
    "operation": "StixCyberObservableAdd",
    "operation_query": "mutation StixCyberObservableAdd($type:String!$stix_id:StixId$x_opencti_score:Int$x_opencti_description:String$createIndicator:Boolean$createdBy:String$objectMarking:[String]$objectLabel:[String]$objectOrganization:[String]$externalReferences:[String]$update:Boolean$AutonomousSystem:AutonomousSystemAddInput$Directory:DirectoryAddInput$DomainName:DomainNameAddInput$EmailAddr:EmailAddrAddInput$EmailMessage:EmailMessageAddInput$EmailMimePartType:EmailMimePartTypeAddInput$Artifact:ArtifactAddInput$StixFile:StixFileAddInput$X509Certificate:X509CertificateAddInput$IPv4Addr:IPv4AddrAddInput$IPv6Addr:IPv6AddrAddInput$MacAddr:MacAddrAddInput$Mutex:MutexAddInput$NetworkTraffic:NetworkTrafficAddInput$Process:ProcessAddInput$Software:SoftwareAddInput$Url:UrlAddInput$UserAccount:UserAccountAddInput$WindowsRegistryKey:WindowsRegistryKeyAddInput$WindowsRegistryValueType:WindowsRegistryValueTypeAddInput$CryptographicKey:CryptographicKeyAddInput$CryptocurrencyWallet:CryptocurrencyWalletAddInput$Hostname:HostnameAddInput$Text:TextAddInput$UserAgent:UserAgentAddInput$BankAccount:BankAccountAddInput$PhoneNumber:PhoneNumberAddInput$PaymentCard:PaymentCardAddInput$MediaContent:MediaContentAddInput){stixCyberObservableAdd(type:$type stix_id:$stix_id x_opencti_score:$x_opencti_score x_opencti_description:$x_opencti_description createIndicator:$createIndicator createdBy:$createdBy objectMarking:$objectMarking objectLabel:$objectLabel update:$update externalReferences:$externalReferences objectOrganization:$objectOrganization AutonomousSystem:$AutonomousSystem Directory:$Directory DomainName:$DomainName EmailAddr:$EmailAddr EmailMessage:$EmailMessage EmailMimePartType:$EmailMimePartType Artifact:$Artifact StixFile:$StixFile X509Certificate:$X509Certificate IPv4Addr:$IPv4Addr IPv6Addr:$IPv6Addr MacAddr:$MacAddr Mutex:$Mutex NetworkTraffic:$NetworkTraffic Process:$Process Software:$Software Url:$Url UserAccount:$UserAccount WindowsRegistryKey:$WindowsRegistryKey WindowsRegistryValueType:$WindowsRegistryValueType CryptographicKey:$CryptographicKey CryptocurrencyWallet:$CryptocurrencyWallet Hostname:$Hostname Text:$Text UserAgent:$UserAgent BankAccount:$BankAccount PhoneNumber:$PhoneNumber PaymentCard:$PaymentCard MediaContent:$MediaContent){id standard_id entity_type parent_types indicators{edges{node{id pattern pattern_type}}}}}",
    "size": 560,
    "time": 12,
    "timestamp": "2024-01-24T15:32:36.177Z",
    "type": "WRITE_ERROR",
    "user": {
        "applicant_id": "88ec0c6a-13ce-5e39-b486-354fe4a7084f",
        "call_retry_number": "1",
        "group_ids": [
            "9cc69711-0654-4c42-91f5-d13f05d514ed"
        ],
        "ip": "::ffff:127.0.0.1",
        "organization_ids": [],
        "socket": "query",
        "user_id": "88ec0c6a-13ce-5e39-b486-354fe4a7084f",
        "user_metadata": {}
    },
    "variables": {
        "StixFile": {
            "atime": null,
            "ctime": null,
            "hashes": [
                {
                    "algorithm": "SSDEEP",
                    "hash": "12:yfNlC+jKQ/NuTD9jDuMZGo/51zNrtt+jKUXnk6lSUmTp8nnfglH/:eljRI9RZN/TzpejPXVlS3d0gF"
                }
            ],
            "magic_number_hex": null,
            "mime_type": null,
            "mtime": null,
            "name": "test",
            "name_enc": null,
            "size": null,
            "x_opencti_additional_names": null
        },
        "createIndicator": false,
        "createdBy": null,
        "externalReferences": [],
        "objectLabel": [],
        "objectMarking": [],
        "objectOrganization": [],
        "stix_id": "file--4d0aa380-f62b-5df2-9dee-b2e7ea26d399",
        "type": "StixFile",
        "update": false,
        "x_opencti_description": null,
        "x_opencti_score": null
    },
    "version": "5.12.21"
}

Additional information

When creating the entity, all fuzzy hashes are removed here resulting in valuesFiltering being an empty array and an error being thrown here

It looks like this behavior is a result of addressing this issue. It makes sense why it was done, but I have a need to be able to create file observables that I only have a fuzzy hash for.

Stack trace:

UNSUPPORTED_ERROR: Invalid filter configuration
    at error (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/config/errors.js:8:10)
    at UnsupportedError (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/config/errors.js:77:51)
    at buildLocalMustFilter (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/database/engine.js:1650:9)
    at buildSubQueryForFilterGroup (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/database/engine.js:1669:37)
    at buildSubQueryForFilterGroup (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/database/engine.js:1660:30)
    at elQueryBodyBuilder (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/database/engine.js:2097:35)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at elPaginate (/home/mangus/github/opencti/opencti/opencti-platform/opencti-graphql/src/database/engine.js:2464:16)

Screenshots (optional)

[nino-filigran]

@mangusm Thanks for your request. We have indeed changed the implementation as you have mentionned .We'll fix this but also consider this request as a feature, since tackling in a quick way will not work easily. For now, the solution that we have identified are:

• be able to create an observable with a fuzzy hash
• add the hash in the UI (not existing at the moment)
• remove the deduplication mechanism for these fuzzy hashes (or maybe just on the the last 6 bits which can be the reason why 2 sames hashes exist for different files).

[SamuelHassine]

@nino-filigran @Jipegien we need this in the release of today.

[richard-julien]

In the master, covered by 2e615a0 (#5678)

In opencti-platform/opencti-graphql/src/database/middleware.js

Sorry it was a late work and i forget to split the code to have a proper commit