[ti_opencti] OpenCTI integration

[chrisberkhout]

Proposed commit message

[ti_opencti] OpenCTI integration

A new integration for the OpenCTI threat intelligence platform:
https://www.filigran.io/en/solutions/products/opencti/

It queries a GraphQL API for indicator data using the CEL input,
populates ECS threat fields where possible and preserves additional
indicator-related data under an `opencti` prefix.

Background

OpenCTI integration development notes has some some background information about OpenCTI and the approach taken with this integration.

OpenCTI Mappings is a spreadsheet with field-level notes.

Status

What's working

This version is ready for final review and then release.

Since the demo timestamp mapping has been changed. The current mappings of the various timestamps are described in the README. The motivations for the change were to show incoming data at recent @timestamp values for a more intuitive user experience, and to build visualizations on logs-* rather than ad-hoc data views to avoid issues with persistent visualization errors and controls not working.

Potential future enhancements

• Set up a transform to provide a view that shows the most recent non-expired data, as done elsewhere.
• Support limiting the time range of the initial fetch.
• Parse STIX rules for atomic indicators, so alert rules can work when no related observable is present, if requested.
  Infosec have said that, in general, observables will be present. For the demo.opencti.io data, only about 0.17% of indicators have a STIX pattern but no observables.

How to test this PR locally

Build and install the integration as usual:

cd packages/ti_opencti
elastic-package stack up -d
elastic-package build -v
elastic-package install -v
elastic-package test pipeline -v
elastic-package test system -v

With a free account on the OpenCTI public demo instance you can get an API key (in the UI under Profile > API access) that will work as a bearer token as long as you're logged in. The OpenCTI demo instance has about 1.2 million indicators. The API token and base URL are the only setting required when adding a policy.

Making dashboard changes

If dashboards need to be updated, it may be necessary to downgrade to version 8.7.1 of the stack and export them from there, to avoid failing with the error "packages with dashboards exported since Kibana 8.8.0 may not be installed till 8.10.0, please export the dashboard/s from a different version". To make the integration work on 8.7.1, some further changes are necessary:

Changes necessary to run on 8.7.1

diff --git a/packages/ti_opencti/data_stream/indicator/agent/stream/cel.yml.hbs b/packages/ti_opencti/data_stream/indicator/agent/stream/cel.yml.hbs
index f4d975ed9..20067bf9a 100644
--- a/packages/ti_opencti/data_stream/indicator/agent/stream/cel.yml.hbs
+++ b/packages/ti_opencti/data_stream/indicator/agent/stream/cel.yml.hbs
@@ -43,15 +43,15 @@ program: |
         {}
     )
   }).with({
-    "Body": {
-      "query": state.query,
+    "Body": '''{
+      "query": ''' + state.query.encode_json() + ''',
       "variables": {
-        "after": has(state.cursor) && has(state.cursor.value) ? state.cursor.value : null,
-        "first": state.page_size,
+        "after": ''' + (has(state.cursor) && has(state.cursor.value) ? state.cursor.value.encode_json() : 'null') + ''',
+        "first": ''' + string(state.page_size) + ''',
         "orderBy": "modified",
-        "orderMode": "asc",
+        "orderMode": "asc"
       }
-    }.encode_json()
+    }'''
   }).do_request().as(resp,
     bytes(resp.Body).decode_json().as(body, state.with({
       "events": body.data.indicators.edges.map(e, e.node.with(
diff --git a/packages/ti_opencti/manifest.yml b/packages/ti_opencti/manifest.yml
index 02d8ca449..881eb2eff 100644
--- a/packages/ti_opencti/manifest.yml
+++ b/packages/ti_opencti/manifest.yml
@@ -11,7 +11,7 @@ categories:
   - threat_intel
 conditions:
   kibana:
-    version: "^8.9.0"
+    version: "^8.7.1"
 screenshots:
   - src: /img/screenshot1.png
     title: "Dashboard: OpenCTI Overview"

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Closes OpenCTI #6186

Screenshots

Please see the screenshots in the integration.

[elasticmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-10-12T14:52:14.782+0000

• Duration: 115 min 43 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5027
Skipped	6
Total	5033

Steps errors

Expand to view the steps failures

Test integration: aws

• Took 3 min 33 sec . View more details here
• Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

Google Storage Download

• Took 0 min 0 sec . View more details here

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (464/464)	💚
Files	96.126% (794/826)	👎 -3.874
Classes	96.126% (794/826)	👎 -3.874
Methods	92.455% (7695/8323)	👎 -7.545
Lines	88.365% (175204/198274)	👎 -7.158
Conditionals	100.0% (0/0)	💚

[ebeahan]

Great work! I added some nits as a scanned your progress so far.

For the list of remaining items under To Do, let's talk to @jamiehynds if there are items/features we can descope in this initial version. For example, maybe we don't need to support additional auth methods, likemTLS certificate and OAuth2, up front.

[chrisberkhout]

/test

[chrisberkhout]

/test

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

@jsoriano I suspect the "run all the things" testing here is because of the change to links_table.yml. This is a similar situation to what I saw when I changed the pr template. Should we be doing a full test of all packages when this file changes? (I can see that we should do a check to make sure the change has not broken any generated docs).

[jsoriano]

Should we be doing a full test of all packages when this file changes? (I can see that we should do a check to make sure the change has not broken any generated docs).

Yes, it looks like a check would be enough, or we could try to make the pipeline smarter and detect what packages are using each link. Could you please open an issue to improve the situation? Though this file doesn't change frequently.

[jsoriano]

Change in CODEOWNERS and links table looks good to me.

[taylor-swanson]

Just a couple of typos to fix, but otherwise LGTM

[ebeahan]

@chrisberkhout anything else you need here or is this ready to merge and release?

[chrisberkhout]

@ebeahan All feedback addressed and tests for this integration are passing. I think this is good to go.

It looks like I can't override and merge without tests passing, but if someone else can that would be good.

[elasticmachine]

Package ti_opencti - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=ti_opencti

[colin-stubbs]

a pleasant surprise to see this pop up, nice work @chrisberkhout !