Uploading from python connector stopped working

[Xehos]

Description

After upload Stix2 bundle from my Python connector, upload performs really slowly and nothing appears to be added (worker log attached below). No errors are showed in the OpenCti anyway.

Uploads from other connectors are okay.

Environment

1. Ubuntu 20.04
2. OpenCTI 4.5.4
3. OpenCTI client: python

Reproducible Steps

scenario:

1. Compose OpenCTI
2. Start Connector
3. Upload STIX2 bundle

Expected Output

Data from Stix2 should be uploaded to the ElasticSearch

Actual Output

Additional information

Uploading from any connector is working properly, but uploading from my python connectors stopped working unexpectedly yesterday.
For sending bundle I use send_stix2_bundle from OpenCTI helper library and api.work.initiate_work for work initiation.

[richard-julien]

Hi @Xehos ,

I think its due to the fact that currently with filter all errors of type "missing reference" in the UI but we should not.
When the worker try to create something, you can have missing dependencies that prevent the creation. With multiple workers it can happen because of race conditions, so we retry (max 5 times) until we ack and continue. In your case i think its not due to race condition but only real missing dependencies, so everything is retried 5 times without success.

So in fact you have 171 messages retried 5 times that fail everytimes. Its for this reason that its really slow to process and you dont see any new data in OCTI.

In case of final fail of the retry system, the UI should ne filter this error.
In your case you should have 175 errors visible in the UI.

We will improve the UI to show this kind of situation as error in the next release.

[Xehos]

Hi @Xehos ,

I think its due to the fact that currently with filter all errors of type "missing reference" in the UI but we should not.
When the worker try to create something, you can have missing dependencies that prevent the creation. With multiple workers it can happen because of race conditions, so we retry (max 5 times) until we ack and continue. In your case i think its not due to race condition but only real missing dependencies, so everything is retried 5 times without success.

So in fact you have 171 messages retried 5 times that fail everytimes. Its for this reason that its really slow to process and you dont see any new data in OCTI.

In case of final fail of the retry system, the UI should ne filter this error.
In your case you should have 175 errors visible in the UI.

We will improve the UI to show this kind of situation as error in the next release.

Actually, I don't really think that dependencies are my problem, the bundle I'm sending is for example:
{ "type": "bundle", "id": "bundle--2c6aa239-ae16-4860-839f-955b8ccb3ca9", "objects": [ { "type": "indicator", "spec_version": "2.1", "id": "indicator--9612ddd5-c7f5-4f78-8006-36ae9cb19ece", "created": "2021-06-17T15:35:35.718977Z", "modified": "2021-06-17T15:35:35.718977Z", "name": "75ee9dd7cb1cb7aafb338901dccf75cd9421b7fce4f2a51dd424f590e282e276", "pattern": "[file:hashes.sha256= '75ee9dd7cb1cb7aafb338901dccf75cd9421b7fce4f2a51dd424f590e282e276']", "pattern_type": "stix", "pattern_version": "2.1", "valid_from": "2021-06-17T15:35:35.718977Z", "labels": [ "Twitter", "Malware Hunter Team" ], "external_references": [ { "source_name": "Twitter - First Seen", "url": "2021-06-17T14:05:21.000Z" }, { "source_name": "Twitter - Last Seen", "url": "2021-06-17T14:05:33.000Z" }, { "source_name": "Twitter - URL", "url": "https://twitter.com/1334890263492370445/status/1405527038094495756" } ] }, { "type": "file", "spec_version": "2.1", "id": "file--d60228e7-ac0c-53b3-8e63-a5db887864a7", "hashes": { "SHA-256": "75ee9dd7cb1cb7aafb338901dccf75cd9421b7fce4f2a51dd424f590e282e276" } }, { "type": "relationship", "spec_version": "2.1", "id": "relationship--d9932880-41b3-4c66-aac5-c5158b241eca", "created": "2021-06-17T15:35:35.72403Z", "modified": "2021-06-17T15:35:35.72403Z", "relationship_type": "based-on", "source_ref": "indicator--9612ddd5-c7f5-4f78-8006-36ae9cb19ece", "target_ref": "file--d60228e7-ac0c-53b3-8e63-a5db887864a7" } ] }

All dependencies should be handled perfectly and I don't see any problem.
This time, I'm also struggling with RabbitMQ queues, which as I think should be handled with work_id work assignment.
(Log below)

I don't really know what should be problem, because I already tried many things (including complete OpenCTI lab reinstall) and nothing worked.

I don't really know how it could break itself without my touch.

[richard-julien]

Always difficult to mix your problem into a single question / issue.

With the first logs and screenshot you send, my explanation make sens. (Missing indicator or ipv4 addr in the relation you try to create)

Now for your second message, you can have this kind of message if you cleanup rabbitMQ without cleanup elasticsearch. When connector register themself, they create dedicated queue (rabbit) and references them in opencti (elastic). So if you clean rabbit without cleaning elastic you create inconsistency. To solve that you cleanup everything or :

• Stop all connectors
• Cleanup rabbit
• in OCTI UI, remove all connectors
• Start all connectors

[Xehos]

Always difficult to mix your problem into a single question / issue.

With the first logs and screenshot you send, my explanation make sens. (Missing indicator or ipv4 addr in the relation you try to create)

Now for your second message, you can have this kind of message if you cleanup rabbitMQ without cleanup elasticsearch. When connector register themself, they create dedicated queue (rabbit) and references them in opencti (elastic). So if you clean rabbit without cleaning elastic you create inconsistency. To solve that you cleanup everything or :

• Stop all connectors
• Cleanup rabbit
• in OCTI UI, remove all connectors
• Start all connectors

Okay, I managed to fix the ElasticSearch upload issue by cleaning the Rabbit.
Now everything seems to be fine, but the Elastic is still not updating. Now I'm stuck at:

INFO:root:Message (delivery_tag=DELIVERY_NUMBER) acknowledged
INFO:root:Processing a new message (delivery_tag=DELIVERY_NUMBER), launching a thread...
INFO:root:Report expectation for WORK_ID
INFO:root:Message processed, thread terminated

and then it obviously fails at connecting relationships because no Indicators or Observables are uploaded successfully.

That repeats again and again until it drops the work and continue.
Screenshot of log:

[Xehos]

Update
I managed to fix the problem.
The whole problem was about setup of connector scope in the config.yml file. Because in pycti documentation, connector scope wasn't mentioned as compulsory part of the way how is the connector uploading data to the Elastic through RabbitMQ.
And also I didn't find any pointers what exactly should I put in the scope, after some time I found out that are just Stix2 types.

Anyway, thank you for help with understanding why the error didn't appear in the UI for the first time!

[Xehos]

Further notice
Scopes aren't needed at all and their using is deprecated in current version of the OpenCTI.
There's no need to pass them as parameter of send_stix2_bundle.