[TI CIF3] Add IOC Expiration #9550

kcreddy · 2024-04-08T19:03:33Z

Proposed commit message

Adds IOC expiration as per discussion at Expiration of Indicators of Compromise (IoCs) csirtgadgets/cif-v5#43 (comment)

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

elastic-package stack down && elastic-package build && elastic-package stack up -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test system --generate -v

--- Test results for package: ti_cif3 - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                        │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ ti_cif3 │             │ asset     │ dashboard ti_cif3-6005a190-0aba-11ed-bcc0-01c79f2670f3 is loaded │ PASS   │        500ns │
│ ti_cif3 │             │ asset     │ dashboard ti_cif3-63a0e470-0a30-11ed-bcc0-01c79f2670f3 is loaded │ PASS   │         83ns │
│ ti_cif3 │             │ asset     │ dashboard ti_cif3-aedada10-0ab5-11ed-bcc0-01c79f2670f3 is loaded │ PASS   │         42ns │
│ ti_cif3 │             │ asset     │ dashboard ti_cif3-b4d9d9b0-0a2f-11ed-bcc0-01c79f2670f3 is loaded │ PASS   │         83ns │
│ ti_cif3 │             │ asset     │ dashboard ti_cif3-bda23600-0abb-11ed-bcc0-01c79f2670f3 is loaded │ PASS   │         42ns │
│ ti_cif3 │             │ asset     │ dashboard ti_cif3-fef149c0-0a2f-11ed-bcc0-01c79f2670f3 is loaded │ PASS   │         83ns │
│ ti_cif3 │ feed        │ asset     │ index_template logs-ti_cif3.feed is loaded                       │ PASS   │        125ns │
│ ti_cif3 │ feed        │ asset     │ ingest_pipeline logs-ti_cif3.feed-1.12.0 is loaded               │ PASS   │         84ns │
╰─────────┴─────────────┴───────────┴──────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_cif3 - END   ---
Done
Run pipeline tests for the package
--- Test results for package: ti_cif3 - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                        │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────────────┼────────┼──────────────┤
│ ti_cif3 │ feed        │ pipeline  │ test-cif3-no-preserve-ndjson.log │ PASS   │  21.984334ms │
│ ti_cif3 │ feed        │ pipeline  │ test-cif3-sample-ndjson.log      │ PASS   │   3.065291ms │
╰─────────┴─────────────┴───────────┴──────────────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_cif3 - END   ---
Done
Run static tests for the package
--- Test results for package: ti_cif3 - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ ti_cif3 │ feed        │ static    │ Verify sample_event.json │ PASS   │  53.611292ms │
╰─────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: ti_cif3 - END   ---
Done
Run system tests for the package
--- Test results for package: ti_cif3 - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ ti_cif3 │ feed        │ system    │ default   │ PASS   │ 39.578945416s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: ti_cif3 - END   ---
Done

Related issues

Closes [TI_CIF3] Add support for IOC expiration #7358

Screenshots

Updated with filter NOT labels.is_ioc_transform_source: true

elasticmachine · 2024-04-08T19:18:05Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2024-04-09T04:42:27Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

packages/ti_cif3/_dev/build/docs/README.md

efd6 · 2024-04-09T04:52:05Z

packages/ti_cif3/data_stream/feed/fields/ecs.yml

@@ -98,3 +100,10 @@
  name: threat.indicator.geo.region_name
 - external: ecs
  name: threat.indicator.geo.timezone
+# Manually define this as a workaround for failing tests and validation


What is the test that fails without this?

There was validation check that fails to import this ECS field if defined with external: ecs. It doesn't seem to be failing now. Looks like it might have been fixed in newer versions of elastic-package.
Modified to external: ecs

efd6 · 2024-04-09T21:39:41Z

packages/ti_cif3/data_stream/feed/elasticsearch/ingest_pipeline/default.yml

+  - date:
      field: cif3.firsttime
      target_field: threat.indicator.first_seen
-      ignore_missing: true
-  - rename:
+      formats:
+        - "ISO8601"
+      if: ctx.cif3?.firsttime != null
+  - date:
      field: cif3.lasttime
      target_field: threat.indicator.last_seen
-      ignore_missing: true
+      formats:
+        - "ISO8601"
+      if: ctx.cif3?.lasttime != null
+  - set:
+      field: threat.indicator.name
+      copy_from: cif3.indicator
+      if: ctx.cif3?.indicator != null


What happens if these fail? Do we just except that the world has stopped working? (probably reasonable)

Added on_failure clause for date processor with error.message.

elastic-sonarqube · 2024-04-10T05:02:47Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
94.7% 94.7% Coverage on New Code
0.0% 0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-04-10T05:02:47Z

💚 Build Succeeded

Buildkite Build
Commit: 6352ddc

History

💚 Build #10278 succeeded 97954b0
💚 Build #10273 succeeded 817c5b1
💚 Build #10271 succeeded 6072f92
💚 Build #10264 succeeded e8d9491

cc @kcreddy

efd6

Still LGTM

elasticmachine · 2024-04-10T05:45:25Z

Package ti_cif3 - 1.12.0 containing this change is available at https://epr.elastic.co/search?package=ti_cif3

kcreddy added 2 commits April 9, 2024 00:16

Add ioc expiration to CIF3 package

fdd7eef

add ilm and update readme

e8d9491

kcreddy added enhancement New feature or request Integration:ti_cif3 Collective Intelligence Framework v3 labels Apr 8, 2024

kcreddy marked this pull request as ready for review April 9, 2024 04:42

kcreddy requested a review from a team as a code owner April 9, 2024 04:42

kcreddy added the Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] label Apr 9, 2024

kcreddy self-assigned this Apr 9, 2024

add ilm policy to manifest

6072f92

efd6 reviewed Apr 9, 2024

View reviewed changes

address pr comment

817c5b1

kcreddy requested a review from efd6 April 9, 2024 07:46

change ecs field in transform

97954b0

efd6 approved these changes Apr 9, 2024

View reviewed changes

address pr comment

6352ddc

efd6 approved these changes Apr 10, 2024

View reviewed changes

kcreddy merged commit 09370a0 into elastic:main Apr 10, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TI CIF3] Add IOC Expiration #9550

[TI CIF3] Add IOC Expiration #9550

kcreddy commented Apr 8, 2024 •

edited

Loading

elasticmachine commented Apr 8, 2024 •

edited

Loading

elasticmachine commented Apr 9, 2024

efd6 Apr 9, 2024

kcreddy Apr 9, 2024

efd6 Apr 9, 2024

kcreddy Apr 10, 2024

elastic-sonarqube bot commented Apr 10, 2024

elasticmachine commented Apr 10, 2024

efd6 left a comment

elasticmachine commented Apr 10, 2024

[TI CIF3] Add IOC Expiration #9550

[TI CIF3] Add IOC Expiration #9550

Conversation

kcreddy commented Apr 8, 2024 • edited Loading

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Apr 8, 2024 • edited Loading

🚀 Benchmarks report

elasticmachine commented Apr 9, 2024

efd6 Apr 9, 2024

Choose a reason for hiding this comment

kcreddy Apr 9, 2024

Choose a reason for hiding this comment

efd6 Apr 9, 2024

Choose a reason for hiding this comment

kcreddy Apr 10, 2024

Choose a reason for hiding this comment

elastic-sonarqube bot commented Apr 10, 2024

Quality Gate passed

elasticmachine commented Apr 10, 2024

💚 Build Succeeded

History

efd6 left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 10, 2024

kcreddy commented Apr 8, 2024 •

edited

Loading

elasticmachine commented Apr 8, 2024 •

edited

Loading