Errors related to Elasticsearch engine pagination.(CVE Connector error)

[misohouse]

Previous question: #5999

ERR Fail to execute engine pagination | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":2,"should":[{"bool":{"minimum_should_match":3,"should":[{"bool":{"minimum_should_match":1,"should":[{"multi_match":{"fields":["event_source_id.keyword"],"query":"be6b5136-4db9-4ccd-b2fa-cb8031387f27"}}]}},{"bool":{"minimum_should_match":1,"should":[{"multi_match":{"fields":["status.keyword"],"query":"complete"}}]}},{"bool":{"minimum_should_match":1,"should":[{"range":{"completed_time":{"lte":"now-7d/d"}}}]}}]}},{"bool":{"minimum_should_match":1,"should":[{"multi_match":{"fields":["entity_type.keyword","parent_types.keyword"],"query":"Work"}}]}}]}}],"must_not":[]}},"size":500,"sort":[{"standard_id.keyword":"asc"}]},"index":["opencti_history"],"track_total_hits":true}},"message":"Fail to execute engine pagination","name":"DATABASE_ERROR","stack":"DATABASE_ERROR: Fail to execute engine pagination\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at DatabaseError (/opt/opencti/build/src/config/errors.js:58:48)\n at /opt/opencti/build/src/database/engine.js:2520:15\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at elList (/opt/opencti/build/src/database/engine.js:2543:22)\n at deleteCompletedWorks (/opt/opencti/build/src/manager/connectorManager.js:93:3)\n at connectorHandler (/opt/opencti/build/src/manager/connectorManager.js:117:7)\n at /opt/opencti/build/src/manager/connectorManager.js:137:9\n at npt.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"The content length (536932588) is bigger than the maximum allowed string (536870888)","name":"RequestAbortedError","stack":"RequestAbortedError: The content length (536932588) is bigger than the maximum allowed string (536870888)\n at aYt.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:574:34)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at k$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:89:10)\n at elList (/opt/opencti/build/src/database/engine.js:2543:22)\n at deleteCompletedWorks (/opt/opencti/build/src/manager/connectorManager.js:93:3)\n at connectorHandler (/opt/opencti/build/src/manager/connectorManager.js:117:7)\n at /opt/opencti/build/src/manager/connectorManager.js:137:9\n at npt.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}] manager=CONNECTOR_MANAGER timestamp=2024-02-16T06:34:16.696Z version=5.12.31

I read the “Cluster setting” page but it’s hard to understand that since I set up the elasticsearch by docker-compose in portainer.

• Cluster settings

Would you tell me how to configure the cluster setting of elasticsearch by docker-compose in portainer?

=============

If the error is not for the circuit breaker that @helene-nguyen mentioned in a previous question, would you suggest another solution?

[misohouse]

I verified that the "query":"be6b5136-4db9-4ccd-b2fa-cb8031387f27" in that error statement is the UUIDv4 of the CVE Connector I set up.

The connector is not in use, so I don't know why the error occurred since it was commented out in portainer.

I uncommented it and checked that it works, but I still get the same error.

The problem seems to occur when importing data.

Also, when I try to delete previously completed tasks in bulk on the [Data - Connectors] page, I get an engine pagination error. (I can delete them individually, but the amount is too large to do it one by one.)

And there is no log in CVE connector's container.

==============

The current CVE connector container state(1) and opencti container state(2) are shown in the photo below.
(1)

(2)

I don't know why I'm getting errors periodically, even though the cve import operation has already been completed.

[helene-nguyen]

Hi @misohouse,

I looked much deeper into it, and you're right, not a circuit breaker issue concerning the error above, and I found the part of the code responsible for this raised error:

This compares what the header received as response for the http request and the max_response_size allowed which is 10mb by default:

• Max Response Size Elasticsearch documentation

In order to easily access to the cluster settings, have you installed Kibana ? It's a data visualization and exploration tool from Elastic.

An example to add in your docker compose file on the link below:

• Deploy Kibana example

Once you access to your Kibana, you can open the Dev Tools and check your max_response_size by running this command:

GET /_cluster/settings?include_defaults=true

Could you check the config by your side ?

[misohouse]

@helene-nguyen

I installed kibana as you said and saw that there are max_response_size in 2 places.

And I changed the max_response_size in 1 of them with the put message below.

Below shows the result of that query.

The other one is a static value, so I get an error that it can't be changed

And in the elasticsearch container log, I could see the log that the max_response_size was changed as shown below.
updating [xpack.inference.http.max_response_size] from [10mb] to [50mb] | @timestamp=2024-02-20T00:29:28.942Z log.level=INFO ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[f60a1eb4a8a5][clusterApplierService#updateTask][T#1] log.logger=org.elasticsearch.common.settings.ClusterSettings trace.id=a5fe3ffd4a7c74e920c72ce50be384cb elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=f60a1eb4a8a5 elasticsearch.cluster.name=docker-cluster

But I'm still getting the same error in opencti container... What am I doing wrong?

[helene-nguyen]

Ok, let me check the CVE Connector side too and give you an update as soon as possible.

[helene-nguyen]

Hello @misohouse,
I cannot reproduce the error, could you send some details about your CVE configuration please ?

[misohouse]

@helene-nguyen

Currently, OpenCTI is so laggy that it's almost unusable.

Rebooting will make things temporarily better, but over time, the system will get progressively slower and slower until it becomes nearly unusable.

Looking at the CPU logs, it seems to be caused by an elasticsearch related crash.

My CVE configuration is below.
connector-cve: image: opencti/connector-cve:5.12.31 environment: - OPENCTI_URL=http://opencti:8080 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} - CONNECTOR_ID=${CONNECTOR_CVE_ID} - CONNECTOR_TYPE=EXTERNAL_IMPORT - CONNECTOR_NAME=Common Vulnerabilities and Exposures - CONNECTOR_SCOPE=identity,vulnerability - CONNECTOR_CONFIDENCE_LEVEL=75 # From 0 (Unknown) to 100 (Fully trusted) - CONNECTOR_UPDATE_EXISTING_DATA=false - CONNECTOR_RUN_AND_TERMINATE=false - CONNECTOR_LOG_LEVEL=error - CVE_BASE_URL=https://services.nvd.nist.gov/rest/json/cves - CVE_API_KEY=${CONNECTOR_CVE_API_KEY} # Required - CVE_INTERVAL=24 # Required, in hours advice min 2 - CVE_MAX_DATE_RANGE=120 # In days, max 120 - CVE_MAINTAIN_DATA=true # Required, retrieve only updated data - CVE_PULL_HISTORY=false # If true, CVE_HISTORY_START_YEAR is required - CVE_HISTORY_START_YEAR=2024 # Required if pull_history is True, min 2019 (see documentation CVE and CVSS base score V3.1 restart: always

And I send some pictures which may help to fix this situation.

1. CVE Connector status in OpenCTI

2. CPU stuck message in OpenCTI server

3. CVE Connector container logs
-> This error looks like an API-related error, but it's also happening with the AbuseIPDB connector.

`
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 385, in health_check
test = self.query(
^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 326, in query
r = self.session.post(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc6a818eb90>: Failed to establish a new connection: [Errno 111] Connection refused'))
OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...
ERROR Error pinging the API | timestamp=2024-02-26T02:45:55.041579Z name=Common Vulnerabilities and Exposures exc_info=Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 60, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Try again

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 496, in _make_request
conn.request(
File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 400, in request
self.endheaders()
File "/usr/local/lib/python3.11/http/client.py", line 1293, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.11/http/client.py", line 1052, in _send_output
self.send(msg)
File "/usr/local/lib/python3.11/http/client.py", line 990, in send
self.connect()
File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 238, in connect
self.sock = self._new_conn()
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in _new_conn
raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPConnection object at 0x7f8c4cf8d290>: Failed to resolve 'opencti' ([Errno -3] Try again)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f8c4cf8d290>: Failed to resolve 'opencti' ([Errno -3] Try again)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py", line 362, in ping
result = self.api.connector.ping(self.connector_id, initial_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_connector.py", line 63, in ping
result = self.api.query(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 326, in query
r = self.session.post(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f8c4cf8d290>: Failed to resolve 'opencti' ([Errno -3] Try again)")) attributes={"reason":"HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f8c4cf8d290>: Failed to resolve 'opencti' ([Errno -3] Try again)"))"}
ERROR Error pinging the API | timestamp=2024-02-26T17:56:07.931146Z name=Common Vulnerabilities and Exposures exc_info=Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py", line 362, in ping
result = self.api.connector.ping(self.connector_id, initial_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_connector.py", line 63, in ping
result = self.api.query(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 349, in query
raise ValueError(
ValueError: {'name': 'write ECONNRESET', 'message': 'write ECONNRESET'} attributes={"reason":"{'name': 'write ECONNRESET', 'message': 'write ECONNRESET'}"}
`

4. Elasticsearch container logs
   [gc][485115] overhead, spent [753ms] collecting in the last [1s] | @timestamp=2024-02-26T21:20:57.580Z log.level=WARN ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][scheduler][T#1] log.logger=org.elasticsearch.monitor.jvm.JvmGcMonitorService elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster [gc][young][495293][3932] duration [1.2s], collections [1]/[1.7s], total [1.2s]/[3.3m], memory [1.4gb]->[1.3gb]/[8gb], all_pools {[young] [656mb]->[0b]/[0b]}{[old] [853.3mb]->[1.2gb]/[8gb]}{[survivor] [3.7mb]->[112mb]/[0b]} | @timestamp=2024-02-27T00:10:45.828Z log.level=WARN ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][scheduler][T#1] log.logger=org.elasticsearch.monitor.jvm.JvmGcMonitorService elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster [gc][495293] overhead, spent [1.2s] collecting in the last [1.7s] | @timestamp=2024-02-27T00:10:46.182Z log.level=WARN ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][scheduler][T#1] log.logger=org.elasticsearch.monitor.jvm.JvmGcMonitorService elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster [gc][495296] overhead, spent [409ms] collecting in the last [1s] | @timestamp=2024-02-27T00:10:49.245Z log.level=INFO ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][scheduler][T#1] log.logger=org.elasticsearch.monitor.jvm.JvmGcMonitorService elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster timer thread slept for [6.8s/6874ms] on absolute clock which is above the warn threshold of [5000ms] | @timestamp=2024-02-27T00:11:24.628Z log.level=WARN ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][[timer]] log.logger=org.elasticsearch.threadpool.ThreadPool elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster timer thread slept for [6.8s/6874382826ns] on relative clock which is above the warn threshold of [5000ms] | @timestamp=2024-02-27T00:11:24.744Z log.level=WARN ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][[timer]] log.logger=org.elasticsearch.threadpool.ThreadPool elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster timer thread slept for [5.5s/5571ms] on absolute clock which is above the warn threshold of [5000ms] | @timestamp=2024-02-27T00:11:38.539Z log.level=WARN ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][[timer]] log.logger=org.elasticsearch.threadpool.ThreadPool elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster timer thread slept for [5.5s/5571765872ns] on relative clock which is above the warn threshold of [5000ms] | @timestamp=2024-02-27T00:11:38.557Z log.level=WARN ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][[timer]] log.logger=org.elasticsearch.threadpool.ThreadPool elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster timer thread slept for [6.2s/6288ms] on absolute clock which is above the warn threshold of [5000ms] | @timestamp=2024-02-27T00:12:52.180Z log.level=WARN ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][[timer]] log.logger=org.elasticsearch.threadpool.ThreadPool elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster timer thread slept for [6.2s/6287558623ns] on relative clock which is above the warn threshold of [5000ms] | @timestamp=2024-02-27T00:12:52.180Z log.level=WARN ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][[timer]] log.logger=org.elasticsearch.threadpool.ThreadPool elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster [gc][495407] overhead, spent [261ms] collecting in the last [1s] | @timestamp=2024-02-27T00:13:08.649Z log.level=INFO ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][scheduler][T#1] log.logger=org.elasticsearch.monitor.jvm.JvmGcMonitorService elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster health check of [/usr/share/elasticsearch/data] took [6306ms] which is above the warn threshold of [5s] | @timestamp=2024-02-27T00:18:58.751Z log.level=WARN ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][generic][T#16] log.logger=org.elasticsearch.monitor.fs.FsHealthService elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster [.ds-.kibana-event-log-ds-2024.02.27-000002] creating index, cause [rollover_data_stream], templates [.kibana-event-log-template], shards [1]/[1] | @timestamp=2024-02-27T00:19:11.823Z log.level=INFO ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][masterService#updateTask][T#1786] log.logger=org.elasticsearch.cluster.metadata.MetadataCreateIndexService elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster updating number_of_replicas to [0] for indices [.ds-.kibana-event-log-ds-2024.02.27-000002] | @timestamp=2024-02-27T00:19:12.045Z log.level=INFO ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][masterService#updateTask][T#1786] log.logger=org.elasticsearch.cluster.routing.allocation.AllocationService elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster Data stream lifecycle successfully rolled over datastream [.kibana-event-log-ds] due to the following met rollover conditions [[min_docs: 1], [max_age: 7d]]. The new index is [.ds-.kibana-event-log-ds-2024.02.27-000002] | @timestamp=2024-02-27T00:19:25.429Z log.level=INFO ecs.version=1.2.0 service.name=ES_ECS event.dataset=elasticsearch.server process.thread.name=elasticsearch[a7fdbd99a9b6][clusterApplierService#updateTask][T#1] log.logger=org.elasticsearch.datastreams.lifecycle.DataStreamLifecycleService elasticsearch.cluster.uuid=epnRKtHETQiPrC7pDcc9DQ elasticsearch.node.id=Uq2USWjXSBOPQbDNa4mENQ elasticsearch.node.name=a7fdbd99a9b6 elasticsearch.cluster.name=docker-cluster

[helene-nguyen]

Ok, not related to the CVE connector as AbuseIPDB connector seems to have the same error but definitively a database error, @SouadHadjiat do you have any idea about this error ?

[SouadHadjiat]

@helene-nguyen @misohouse This error seem to happen when trying to delete completed works. We try to query up to 500 completed works to be deleted and it looks like the response is too big.

I suggest for now to run this query manually on elasticsearch, it should delete all completed works older than 7 days, and see if the error stops :

POST opencti_history/_delete_by_query
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "entity_type.keyword": {
              "value": "Work"
            }
          }
        },
        {
          "term": {
            "status.keyword": {
              "value": "complete"
            }
          }
        },
        {
          "range": {
            "completed_time": {
              "lte": "now-7d/d"
            }
          }
        }
      ]
    }
  }
}

I think we should use directly this query in our code instead of querying the works to be then deleted, we won't have any issue with size limits.

[misohouse]

@SouadHadjiat @helene-nguyen

Oh! I ran the query as you said and it deleted thousands of completed tasks and no longer throws the error.

Thank you for your detailed answer! :)

[nino-filigran]

@misohouse If our support has helped you and you're no stuck anymore, I'll let you close this issue :)

[SouadHadjiat]

@nino-filigran I suggest to fix the way we delete completed tasks to avoid having this issue again