[AWS] Add support for Guardduty datastream

[vinit-chauhan]

What does this PR do?

• Added a data stream.
• Added data collection logic to the data stream.
• Added the ingest pipeline for the data stream.
• Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files
• Added dashboards and visualizations.
• Added test for pipeline for the data stream.
• Added system test cases for the data stream.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^8.4.0).

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/aws/data_stream directory.
• Run the following command to run tests.

elastic-package test

Related issues

• Closes AWS GuardDuty #2751

Screenshots

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-02-07T09:24:53.267+0000

• Duration: 53 min 24 sec

Test stats 🧪

Test	Results
Failed	0
Passed	188
Skipped	4
Total	192

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (15/15)	💚
Files	93.75% (15/16)	👎 -6.25
Classes	93.75% (15/16)	👎 -6.25
Methods	86.131% (236/274)	👍 0.417
Lines	85.925% (7387/8597)	👎 -14.075
Conditionals	100.0% (0/0)	💚

[jamiehynds]

@efd6 is this ok to merge or any further changes required from Crest?

[drewdaemon]

Dashboards are following the key best practices 👍

I see that you have used a non-Lens visualization: tag cloud. We will be adding tag clouds to Lens at some point but until then, we suggest using a Lens horizontal bar chart to convey the same information instead.

Of course, you can keep the legacy tag cloud here if you feel strongly. Just be aware that if you add non-Lens visualizations now, you will likely have to change them to Lens visualizations to use this dashboard on serverless.

[efd6]

Thanks

[vinit-chauhan]

Dashboards are following the key best practices 👍

I see that you have used a non-Lens visualization: tag cloud. We will be adding tag clouds to Lens at some point but until then, we suggest using a Lens horizontal bar chart to convey the same information instead.

Of course, you can keep the legacy tag cloud here if you feel strongly. Just be aware that if you add non-Lens visualizations now, you will likely have to change them to Lens visualizations to use this dashboard on serverless.

Hey @drewdaemon, Thanks for the suggestion.

The reason we opted for the tag cloud is that there could be multiple Threat Names and plotting it over a bar chart would result in clutter. If there are only a few values for threat names, I'd make sense to go for a bar chart in case of more than 10-15 names it would only show the first few names and others to be grouped under the "others" bucket and not grouping extra value would result in a visual mess.

Let me know what are your thoughts. :)

[drewdaemon]

@vinit-elastic

The reason we opted for the tag cloud is that there could be multiple Threat Names and plotting it over a bar chart would result in clutter. If there are only a few values for threat names, I'd make sense to go for a bar chart in case of more than 10-15 names it would only show the first few names and others to be grouped under the "others" bucket and not grouping extra value would result in a visual mess.

Yes, I understand your dilemma.

My first question would be: how many threat names is it actually useful to show? Is it important to show more than the top 10-20 values? Generally, there's a law of diminishing returns at play in data visualization.

Horizontal bar chart

The horizontal bar chart actually can support quite a few. As a comparable, here's an unrelated chart of about the same dimensions displaying 25 names (not necessarily the suggested number, just picking something big to illustrate).

It doesn't seem cluttered to me. The labels are readable, the bars are neatly aligned, and the user can mouse over any of them for more information. There is no "other bucket;" terms below the top 25 aren't shown.

A bar chart also makes the difference between one term and another much more obvious than does a tag cloud.

Tag cloud

For example, here's a tag cloud of the same data.

Looking just at this tag cloud, I challenge you to tell me which is the more common term between windows and pfsense. This kind of a comparison is much easier with a bar chart.

That said, I understand the visual appeal of the tag cloud. They border on info-graphics. People like them for sure.

Unfortunately, we're currently making you choose between using Lens and using a tag cloud. Ultimately, it's your job to do what's best for your users.

[vinit-chauhan]

Hey @drewdaemon - Appreciate the explanation. It makes a whole lot of sense now.
As you mentioned, a bar chart would be a better fit here. I'll update the visualisation from tag cloud to bar chart today.
Let me know if you have anything else. 😄

[drewdaemon]

LGTM

[elasticmachine]

Package aws - 1.31.0 containing this change is available at https://epr.elastic.co/search?package=aws