[ti_opencti] Keep expected nulls, improve error handling

[chrisberkhout]

Proposed commit message

[ti_opencti] Keep expected nulls, improve error handling (#8875)

* Set `keep_null: true` to avoid loss of fields between the CEL
  program and the ingest pipeline.
* Handle server responses with errors gracefully in the CEL program
  and in the ingest pipeline.  

Why set keep_null: true?

While ingesting from the public demo instance of OpenCTI, I observed a large number of errors, containing the message Processor "rename" with tag "" in pipeline \"logs-ti_opencti.indicator-0.3.4" failed with message "field [description] doesn't exist". The event.original value showed that description key existed and had a null value, and the pipeline did correctly handle null descriptions.

There was a pipeline test example that included a null description that was processed correctly.

The GraphQL query requested the description field and received it with a null value, but Beats removed it because the keep_null setting defaults to false. Avoiding changes between output from the CEL expression and input to the ingest pipeline seems less surprising and resolves the observed errors.

Why change error handling?

The OpenCTI server may respond with an error message. For example, for an authentication/authorization error it will respond it will HTTP status code of 200 and a body as follows:

{
  "errors": [
    {
      "message": "You must be logged in to do this.",
      "name": "AUTH_REQUIRED",
      "time_thrown": "2024-01-12T10:47:28.810Z",
      "data": {
        "http_status": 401,
        "genre": "TECHNICAL"
      }
    }
  ],
  "data": {
    "indicators": null
  }
}

The CEL expression assumed that .data.indicators is an array (as it is in a successful empty response). That led to an error message of failed eval: no such key: edges and later Processor "rename" with tag "" in pipeline "logs-ti_opencti.indicator-0.3.4" failed with message "field [created] doesn't exist".

The improved CEL expression handles this situation gracefully by detecting the error and capturing more information about it, and then the pipeline logic marks it as a failure without trying to apply processing steps that require a full, correct document. The indexed document will now include the following helpful information:

"error.message": [
  "You must be logged in to do this.",
  "Processor \"conditional\" with tag \"\" in pipeline \"logs-ti_opencti.indicator-0.3.4\" failed with message \"Error during CEL program evaluation\"\n"
],
"event.original": [
  "{\"data\":{\"indicators\":null},\"errors\":[{\"data\":{\"genre\":\"TECHNICAL\",\"http_status\":401},\"message\":\"You must be logged in to do this.\",\"name\":\"AUTH_REQUIRED\",\"time_thrown\":\"2024-01-12T11:28:01.679Z\"}]}"
],

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

• Build and install.
• Add a policy with credentials for https://demo.opencti.io/.
• Try with the wrong API key to generate an error.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Package ti_opencti - 0.3.5 containing this change is available at https://epr.elastic.co/search?package=ti_opencti