[Suricata] Add Observer Metadata #6985

MakoWish · 2023-07-17T19:18:29Z

Type of change

Enhancement

What does this PR do?

The Suricata integration currently supports removing host.* fields if "forwarded" is in tags, but it does not currently populate the observer.* fields. This PR adds that functionality.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have incremented the version my package's manifest.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Modify default documents config to include "forwarded" tag
Verify expected results documents include new observer.* fields

Related issues

Closes [Suricata] Enable observer.* Fields for Integration #6984

…akoWish/integrations into suricata_add_observer_metadata

elasticmachine · 2023-07-17T19:21:52Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-10-03T00:49:57.963+0000
Duration: 14 min 28 sec

Test stats 🧪

Test	Results
Failed	0
Passed	14
Skipped	0
Total	14

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

packages/suricata/data_stream/eve/_dev/test/pipeline/test-common-config.yml

packages/suricata/data_stream/eve/agent/stream/log.yml.hbs

packages/suricata/data_stream/eve/elasticsearch/ingest_pipeline/default.yml

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

…akoWish/integrations into suricata_add_observer_metadata

packages/winlog/data_stream/winlog/elasticsearch/ingest_pipeline/default.yml

elasticmachine · 2023-07-18T08:40:36Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

…akoWish/integrations into suricata_add_observer_metadata

efd6 · 2023-07-30T22:47:57Z

/test

elasticmachine · 2023-07-30T23:02:50Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`5/5`)	💚
Classes	100.0% (`5/5`)	💚
Methods	100.0% (`44/44`)	💚 8.333
Lines	96.204% (`887/922`)	👎 -0.44
Conditionals	100.0% (`0/0`)	💚

efd6

minor nit and then LGTM

packages/suricata/data_stream/eve/_dev/test/pipeline/test-eve-alerts.log-config.yml

packages/suricata/data_stream/eve/_dev/test/pipeline/test-eve-small.log-config.yml

…akoWish/integrations into suricata_add_observer_metadata

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

MakoWish · 2023-08-28T19:51:13Z

@efd6 ,

Would you mind kicking off a /test?

efd6 · 2023-08-28T22:10:46Z

/test

packages/suricata/manifest.yml

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

botelastic · 2023-09-28T16:14:20Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

efd6 · 2023-09-28T20:57:35Z

/test

efd6 · 2023-09-28T21:50:54Z

You'll need to run elastic-package test pipeline -g.

…akoWish/integrations into suricata_add_observer_metadata

MakoWish · 2023-09-29T14:45:58Z

You'll need to run elastic-package test pipeline -g.

I usually do, but looks like I didn't on this one. Done.

efd6 · 2023-10-03T00:49:44Z

/test

efd6

Thanks

elasticmachine · 2023-10-03T02:01:43Z

Package suricata - 2.18.0 containing this change is available at https://epr.elastic.co/search?package=suricata

MakoWish added 4 commits July 17, 2023 12:32

Enable population of observer fields if tags contain "forwarded"

41af726

Enable population of observer fields if tags contain "forwarded"

21d8af4

Add entry to changelog and increment version in manifest

fbfecb1

Merge branch 'suricata_add_observer_metadata' of https://github.com/M…

f0a8552

…akoWish/integrations into suricata_add_observer_metadata

MakoWish marked this pull request as ready for review July 17, 2023 19:19

MakoWish requested a review from a team as a code owner July 17, 2023 19:19

efd6 reviewed Jul 17, 2023

View reviewed changes

MakoWish and others added 4 commits July 17, 2023 16:52

Suggested change from code review

1d78981

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

Suggested change from code review

0365e64

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

Set "observer" tag on only a couple sample logs

fb99600

Merge branch 'suricata_add_observer_metadata' of https://github.com/M…

6c43788

…akoWish/integrations into suricata_add_observer_metadata

efd6 reviewed Jul 17, 2023

View reviewed changes

packages/winlog/data_stream/winlog/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

jamiehynds added the Team:Security-External Integrations label Jul 18, 2023

jamiehynds added the Integration:suricata Suricata label Jul 18, 2023

MakoWish added 6 commits July 18, 2023 08:08

Remove accidental inclusion

c69dd2f

Merge branch 'suricata_add_observer_metadata' of https://github.com/M…

2c5f6f9

…akoWish/integrations into suricata_add_observer_metadata

Suggested changes from code review

1037571

Merge branch 'suricata_add_observer_metadata' of https://github.com/M…

e4f8be1

…akoWish/integrations into suricata_add_observer_metadata

Missing final newline in log.yml.hbs

5424a94

Merge branch 'suricata_add_observer_metadata' of https://github.com/M…

783d508

…akoWish/integrations into suricata_add_observer_metadata

MakoWish changed the title ~~Suricata add observer metadata~~ [Suricata] Add Observer Metadata Jul 20, 2023

efd6 reviewed Jul 31, 2023

View reviewed changes

packages/suricata/data_stream/eve/_dev/test/pipeline/test-eve-alerts.log-config.yml Show resolved Hide resolved

packages/suricata/data_stream/eve/_dev/test/pipeline/test-eve-small.log-config.yml Show resolved Hide resolved

MakoWish and others added 4 commits July 31, 2023 08:34

Merge branch 'main' into suricata_add_observer_metadata

2446475

Increment version due to conflict with recent merge

dba68a6

Merge branch 'suricata_add_observer_metadata' of https://github.com/M…

a24d99c

…akoWish/integrations into suricata_add_observer_metadata

Commit suggested change from code review

e28f329

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

MakoWish and others added 2 commits July 31, 2023 08:35

Commit suggested change from code review

6b21ef5

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

Merge branch 'main' into suricata_add_observer_metadata

ab44513

efd6 reviewed Aug 28, 2023

View reviewed changes

packages/suricata/manifest.yml Outdated Show resolved Hide resolved

Update packages/suricata/manifest.yml

cfe6d7f

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

botelastic bot added the Stalled label Sep 28, 2023

Merge branch 'main' into suricata_add_observer_metadata

5b339a3

botelastic bot removed the Stalled label Sep 28, 2023

MakoWish added 2 commits September 29, 2023 08:45

Expected log result changed

e429670

Merge branch 'suricata_add_observer_metadata' of https://github.com/M…

e5fe6b0

…akoWish/integrations into suricata_add_observer_metadata

efd6 approved these changes Oct 3, 2023

View reviewed changes

efd6 merged commit cfbe348 into elastic:main Oct 3, 2023
4 checks passed

MakoWish deleted the suricata_add_observer_metadata branch October 3, 2023 16:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Suricata] Add Observer Metadata #6985

[Suricata] Add Observer Metadata #6985

MakoWish commented Jul 17, 2023 •

edited

Loading

elasticmachine commented Jul 17, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jul 18, 2023

efd6 commented Jul 30, 2023

elasticmachine commented Jul 30, 2023 •

edited

Loading

efd6 left a comment

MakoWish commented Aug 28, 2023

efd6 commented Aug 28, 2023

botelastic bot commented Sep 28, 2023

efd6 commented Sep 28, 2023

efd6 commented Sep 28, 2023

MakoWish commented Sep 29, 2023

efd6 commented Oct 3, 2023

efd6 left a comment

elasticmachine commented Oct 3, 2023

[Suricata] Add Observer Metadata #6985

[Suricata] Add Observer Metadata #6985

Conversation

MakoWish commented Jul 17, 2023 • edited Loading

Type of change

What does this PR do?

Checklist

Author's Checklist

Related issues

elasticmachine commented Jul 17, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jul 18, 2023

efd6 commented Jul 30, 2023

elasticmachine commented Jul 30, 2023 • edited Loading

🌐 Coverage report

efd6 left a comment

Choose a reason for hiding this comment

MakoWish commented Aug 28, 2023

efd6 commented Aug 28, 2023

botelastic bot commented Sep 28, 2023

efd6 commented Sep 28, 2023

efd6 commented Sep 28, 2023

MakoWish commented Sep 29, 2023

efd6 commented Oct 3, 2023

efd6 left a comment

Choose a reason for hiding this comment

elasticmachine commented Oct 3, 2023

MakoWish commented Jul 17, 2023 •

edited

Loading

elasticmachine commented Jul 17, 2023 •

edited

Loading

elasticmachine commented Jul 30, 2023 •

edited

Loading