Potential "lock" issue causes backed up bundle queue

[securitiz]

Description

We are running the Alienvault connector and enrichment connectors.

In worker logs, we see the following error:

2023-03-02T16:18:36.565-05:00 ERROR:pycti.api:Execution timeout, too many concurrent call on the same entities

The GUI also shows a bundle queue that hovers at approximately 300k bundles.

We think the issue might be related to redis and/or locking mechanisms, as we see locks getting stuck"

We also see instances of AV creating several instances of the same report, leading us to think that there may be a lock bug (screenshot below)

Environment

1. OS (where OpenCTI server runs): Ubuntu 18.04
2. OpenCTI version: OpenCTI 5.4.4
3. OpenCTI client: python/redis
4. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Enable Alienvault connector
2. Watch worker logs, redis, bundles queued count, and reports created

Expected Output

Data processed correctly

Actual Output

Errors in worker logs that result in some objects not being created, high queue, duplicate reports

{ "category": "APP", "error": { "data": { "category": "business", "http_status": 500, "participantIds": [ "relationship--815ac158-6c1f-4442-938e-afa669bf9e61", "relationship--2056e3f1-cd94-5329-aa99-642efd7dc92d", "6f65d211-146b-4ce4-82c0-206db3027384", "0a6d9b96-bdb9-42aa-94cf-d8d27d6e8c8b" ], "reason": "Execution timeout, too many concurrent call on the same entities" }, "stacktrace": [ "LockError: Lock timeout", "at error (/opt/opencti/build/src/config/errors.js:8:10)", "at LockTimeoutError (/opt/opencti/build/src/config/errors.js:100:51)", "at createRelationRaw (/opt/opencti/build/src/database/middleware.js:2807:13)", "at processTicksAndRejections (node:internal/process/task_queues:95:5)", "at createRelation (/opt/opencti/build/src/database/middleware.js:2815:16)", "at addStixCoreRelationship (/opt/opencti/build/src/domain/stixCoreRelationship.js:223:19)" ] }, "inner_relation_creation": 1, "level": "warn", "message": "API Call", "operation": "StixCoreRelationshipAdd", "operation_query": "mutation StixCoreRelationshipAdd($input:StixCoreRelationshipAddInput!){stixCoreRelationshipAdd(input:$input){id standard_id entity_type parent_types}}", "size": 633, "time": 26043, "timestamp": "2023-03-03T23:33:42.766Z", "type": "WRITE_ERROR", "user": { "applicant_id": "6eeecee7-3470-4f27-98e7-fcf73e05277e", "call_retry_number": "25", "ip": "::ffff:10.10.5.8", "user_id": "a82be266-5a89-4766-a4d4-b97d84b06b5e" }, "version": "5.5.4" }

also errors like:

ERROR:root:{'name': 'MissingReferenceError', 'message': 'Missing reference to handle creation'}

Additional information

Screenshots (optional)

[securitiz]

Note - this may also be causing issues with how the Creator is represented. In the screenshot above, the Creator is "Worker". However, we've found that when the pipeline "unclogs", the data created by the connector is accurately displayed as "Alienvault"

[richard-julien]

These kind of errors can occured if too many workers try to work on the same entities. Its "expected". We have some future ideas to improve that but for now you can try to:

• Improve the overall responsivness of your elasticsearch.
• Limit the number of workers, having too much is not always the best solution
• Incorporate more sources of information to shuffle the creation

For the creator not aligned with the connector associated user, we never experiment that and almost impossible to investigate without looking in your platform config and data.

[securitiz]

Understood, thank you Julien

your platform config and data

What config data would you want to look at first?

[securitiz]

Note - this may also be causing issues with how the Creator is represented.

For anyone following, the Creator issue was due to the Worker user not having admin privileges.

We haven't found a solution for the original issue