[Tenable IO] Add `scanner` dataset to experimental Tenable IO Integration #6113

MakoWish · 2023-05-06T02:16:26Z

Type of Change

Enhancement

What does this PR do?

This PR adds scanner and scan datasets to the Tenable.io Integration

List Scanners API Documentation
List Scans API Documentation

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

…/integrations into tenable_io_add_datasets

elasticmachine · 2023-05-06T02:19:55Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-05-12T01:47:25.046+0000
Duration: 16 min 57 sec

Test stats 🧪

Test	Results
Failed	0
Passed	30
Skipped	0
Total	30

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

efd6

Can you provide a link to the API docs for the data set in the PR description?

packages/tenable_io/data_stream/scanner/elasticsearch/ingest_pipeline/default.yml

efd6 · 2023-05-07T21:46:29Z

/test

elasticmachine · 2023-05-07T22:02:42Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`5/5`)	💚
Files	100.0% (`5/5`)	💚 6.25
Classes	100.0% (`5/5`)	💚 6.25
Methods	96.774% (`60/62`)	👍 13.729
Lines	91.697% (`2286/2493`)	👍 5.771
Conditionals	100.0% (`0/0`)	💚

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

MakoWish · 2023-05-08T19:37:41Z

Going to add a dataset for scan to the same branch, so please continue to make any suggestions, and I will be sure to apply them to the scan dataset as well.

…/integrations into tenable_io_add_datasets

MakoWish · 2023-05-08T22:55:40Z

Aside from the above question, should be good. Let me know what you think on that one.

Eric

efd6 · 2023-05-08T23:45:03Z

packages/tenable_io/_dev/build/docs/README.md

@@ -18,6 +18,8 @@ The Tenable.io integration collects logs for three types of events: Asset, Plugi

 **Scanner** is used to retrieve the current state of scanners, including licensing and activity. See more details in the API documentation [here](https://developer.tenable.com/reference/scanners-list).

+**Scan** is used to retrieve details about existing scans, including scan statuses, assigned targets, and more. See more details in the API documentation [here](https://developer.tenable.com/reference/scans-list).


Scans instead of Scan?

efd6 · 2023-05-08T23:46:36Z

packages/tenable_io/_dev/build/docs/README.md

+### scan
+
+This is the `scan` dataset.
+
+#### Example
+
+{{event "scan"}}
+
+{{fields "scan"}}


efd6 · 2023-05-08T23:47:37Z

packages/tenable_io/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "0.5.0"
+  changes:
+    - description: Added dataset for scanner and scan logs.


Suggested change

- description: Added dataset for scanner and scan logs.

- description: Added dataset for scanner and scans logs.

efd6 · 2023-05-08T23:48:08Z

packages/tenable_io/data_stream/scan/_dev/test/pipeline/test-scan.log

Rename to match scans.

packages/tenable_io/data_stream/scan/fields/fields.yml

efd6 · 2023-05-08T23:53:37Z

/test

MakoWish · 2023-05-09T19:37:58Z

I also had a question on the three existing datasets. Each of them are using the fingerprint processor in the Ingest Pipeline, but as far as I am aware, Data Streams cannot update existing documents; only create new documents. Can you think of a reason for using the fingerprint processor on those datasets?

MakoWish · 2023-05-10T15:56:38Z

@efd6,

If you think this will get merged soon, I will hold off for now to prevent conflicts with the changelog and manifest, but I just opened #6147 for the fingerprint issue. The way it currently sits, any updates to existing objects in Tenable will not be reflected in Elastic, because the updated record will fail to ingest. This is pretty much creating static/non-updatable content. New assets, plugins, or vulnerabilities will be ingested, but changes to existing objects will never be reflected in Elastic.

efd6 · 2023-05-11T01:00:11Z

I'm OK with the naming you have now given what already exists. You are right to open an issue for fongerprinting, it needs to have some thought and be handled separately from this.

efd6 · 2023-05-11T01:14:43Z

There looks to be an issue with the dashboards. Can you investigate or show screenshots with the relevant panels showing data and no errors?

MakoWish · 2023-05-11T14:42:04Z

Hi @efd6,

Sure thing! Here are some screenshots (hiding details from our production Tenable):

`[Logs Tenable IO] Scan`

`[Logs Tenable IO] Scanner`

…/integrations into tenable_io_add_datasets

efd6 · 2023-05-12T01:47:09Z

/test

efd6

Thanks

elasticmachine · 2023-05-12T02:31:59Z

Package tenable_io - 0.6.0 containing this change is available at https://epr.elastic.co/search?package=tenable_io

MakoWish added 2 commits May 5, 2023 20:10

Added scanners dataset.

ecf0c0d

Merge branch 'tenable_io_add_datasets' of https://github.com/MakoWish…

5e144c6

…/integrations into tenable_io_add_datasets

MakoWish requested a review from a team as a code owner May 6, 2023 02:16

Update changelog.yml

a3aa127

efd6 reviewed May 7, 2023

View reviewed changes

packages/tenable_io/data_stream/scanner/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

packages/tenable_io/data_stream/scanner/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

MakoWish and others added 5 commits May 8, 2023 08:06

Commit suggested change

27fa6ff

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>

Update README.md template

3c67c58

Update README.md

96de49b

Fix type-o

dd9c12a

Fix type-o

4d7b273

MakoWish added 5 commits May 8, 2023 16:39

Added scan dataset.

fc3bd65

Merge branch 'tenable_io_add_datasets' of https://github.com/MakoWish…

01e58de

…/integrations into tenable_io_add_datasets

Update changelog.yml

1aeb444

Removed unnecessary checks for variable existence.

2887989

Merge branch 'tenable_io_add_datasets' of https://github.com/MakoWish…

4f24909

…/integrations into tenable_io_add_datasets

efd6 reviewed May 8, 2023

View reviewed changes

MakoWish added 3 commits May 11, 2023 08:43

Merge branch 'main' into tenable_io_add_datasets

ab59e3a

Increment version for conflict with recent commit.

cb0f731

Made corrections to asset Ingest Pipeline.

109f713

Merge branch 'tenable_io_add_datasets' of https://github.com/MakoWish…

d8858eb

…/integrations into tenable_io_add_datasets

efd6 approved these changes May 12, 2023

View reviewed changes

efd6 merged commit ca90abb into elastic:main May 12, 2023
3 checks passed

MakoWish deleted the tenable_io_add_datasets branch May 12, 2023 14:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tenable IO] Add `scanner` dataset to experimental Tenable IO Integration #6113

[Tenable IO] Add `scanner` dataset to experimental Tenable IO Integration #6113

MakoWish commented May 6, 2023 •

edited

elasticmachine commented May 6, 2023 •

edited

Build stats

Test stats 🧪

efd6 left a comment

efd6 commented May 7, 2023

elasticmachine commented May 7, 2023 •

edited

MakoWish commented May 8, 2023 •

edited

MakoWish commented May 8, 2023

efd6 May 8, 2023

efd6 May 8, 2023

efd6 May 8, 2023

efd6 May 8, 2023

efd6 commented May 8, 2023

MakoWish commented May 9, 2023

MakoWish commented May 10, 2023

efd6 commented May 11, 2023

efd6 commented May 11, 2023

MakoWish commented May 11, 2023 •

edited

efd6 commented May 12, 2023

efd6 left a comment

elasticmachine commented May 12, 2023

		@@ -18,6 +18,8 @@ The Tenable.io integration collects logs for three types of events: Asset, Plugi

		Scanner is used to retrieve the current state of scanners, including licensing and activity. See more details in the API documentation [here](https://developer.tenable.com/reference/scanners-list).

		Scan is used to retrieve details about existing scans, including scan statuses, assigned targets, and more. See more details in the API documentation [here](https://developer.tenable.com/reference/scans-list).

	- description: Added dataset for scanner and scan logs.
	- description: Added dataset for scanner and scans logs.

[Tenable IO] Add scanner dataset to experimental Tenable IO Integration #6113

[Tenable IO] Add scanner dataset to experimental Tenable IO Integration #6113

Conversation

MakoWish commented May 6, 2023 • edited

Type of Change

What does this PR do?

Checklist

elasticmachine commented May 6, 2023 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

efd6 left a comment

Choose a reason for hiding this comment

efd6 commented May 7, 2023

elasticmachine commented May 7, 2023 • edited

🌐 Coverage report

MakoWish commented May 8, 2023 • edited

MakoWish commented May 8, 2023

efd6 May 8, 2023

Choose a reason for hiding this comment

efd6 May 8, 2023

Choose a reason for hiding this comment

efd6 May 8, 2023

Choose a reason for hiding this comment

efd6 May 8, 2023

Choose a reason for hiding this comment

efd6 commented May 8, 2023

MakoWish commented May 9, 2023

MakoWish commented May 10, 2023

efd6 commented May 11, 2023

efd6 commented May 11, 2023

MakoWish commented May 11, 2023 • edited

[Logs Tenable IO] Scan

[Logs Tenable IO] Scanner

efd6 commented May 12, 2023

efd6 left a comment

Choose a reason for hiding this comment

elasticmachine commented May 12, 2023

[Tenable IO] Add `scanner` dataset to experimental Tenable IO Integration #6113

[Tenable IO] Add `scanner` dataset to experimental Tenable IO Integration #6113

MakoWish commented May 6, 2023 •

edited

elasticmachine commented May 6, 2023 •

edited

elasticmachine commented May 7, 2023 •

edited

MakoWish commented May 8, 2023 •

edited

MakoWish commented May 11, 2023 •

edited

`[Logs Tenable IO] Scan`

`[Logs Tenable IO] Scanner`