[f5_bigip] Initial Release for the F5 BIG-IP #4242

vinit-chauhan · 2022-09-20T15:24:40Z

What does this PR do?

Generated the skeleton of the F5 BIG-IP integration package.
Added data stream.
Added data collection logic for the data stream.
Added the ingest pipeline for the data stream.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Added test for pipeline for the data stream.
Added system test cases for the data stream.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

All changes

Change follows the contributing guidelines
Supported versions of the monitoring target is documented
Supported operating systems are documented (if applicable)
Integration or System tests exist
Documentation exists
Fields follow ECS and naming conventions
At least a manual test with ES / Kibana / Agent has been performed.
Required Kibana version set to: ^7.17.0 || ^8.0.0

New Package

Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

Dashboards exists
Screenshots added or updated
Datastream filters added to visualizations

Log dataset changes

Pipeline tests exist (if applicable)
Generated output for at least 1 log file exists
Sample event (sample_event.json) exists

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/f5_bigip directory.
Run the following command to run tests.

elastic-package test

Related issues

Closes F5 ASM #1310

Screenshots

elasticmachine · 2022-09-20T15:28:11Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2022-09-20T15:35:55Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-11-10T09:08:16.532+0000
Duration: 15 min 29 sec

Test stats 🧪

Test	Results
Failed	0
Passed	16
Skipped	0
Total	16

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-09-20T15:42:46Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2022-09-20T15:43:08Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`6/6`)	💚 2.318
Classes	100.0% (`6/6`)	💚 2.318
Methods	98.039% (`50/51`)	👍 6.845
Lines	95.786% (`2682/2800`)	👍 4.082
Conditionals	100.0% (`0/0`)	💚

…at_version

efd6 · 2022-10-17T00:55:24Z

packages/f5_bigip/_dev/build/docs/README.md

+
+## Overview
+
+The [F5 BIG-IP](https://www.f5.com) integration allows users to monitor LTM, AFM, APM, ASM, and AVR activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions.


Suggested change

The [F5 BIG-IP](https://www.f5.com) integration allows users to monitor LTM, AFM, APM, ASM, and AVR activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions.

The [F5](https://www.f5.com) BIG-IP integration allows users to monitor [LTM](https://www.f5.com/products/big-ip-services/local-traffic-manager), [AFM](https://www.f5.com/products/security/advanced-firewall-manager), [APM](https://www.f5.com/products/security/access-policy-manager), [ASM](https://www.f5.com/pdf/products/big-ip-application-security-manager-overview.pdf), and [AVR](https://clouddocs.f5.com/training/community/analytics/html/class1/class1.html) activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions.

Are there better links for ASM and AVR?

Alternatively maybe just...

Suggested change

The [F5 BIG-IP](https://www.f5.com) integration allows users to monitor LTM, AFM, APM, ASM, and AVR activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions.

The [F5 BIG-IP](https://www.f5.com/products/big-ip-services) integration allows users to monitor LTM, AFM, APM, ASM, and AVR activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions.

and use any useful links in the set above in the "Data stream" section below.

efd6 · 2022-10-17T01:16:24Z

packages/f5_bigip/data_stream/log/_dev/test/pipeline/test-pipeline-bigip-afm.log-expected.json

+    "expected": [
+        {
+            "@timestamp": "2018-12-17T22:46:04.000Z",
+            "destination": {


GeoIP look-ups for destination and source in the common pipeline?

As all the event categories doesn’t have a field that contains public IP, so we haven’t use GeoIP in default pipeline.

efd6 · 2022-10-17T01:18:15Z

packages/f5_bigip/data_stream/log/_dev/test/pipeline/test-pipeline-bigip-ltm.log-expected.json

+            },
+            "related": {
+                "hosts": [
+                    "81.2.69.192:80"


Ensure that hosts have had any ports removed?

We noticed in the live log that the field value couldn't follow a specific pattern, therefore we decided it would be better to delete it from related.host in order to keep the related.host field's proper value.

efd6 · 2022-10-17T01:42:28Z

packages/f5_bigip/data_stream/log/fields/base-fields.yml

@@ -0,0 +1,20 @@
+- name: data_stream.dataset


Move ECS-defined fields into ecs.yml with external definitions where possible.

efd6 · 2022-10-17T01:44:46Z

packages/f5_bigip/data_stream/log/fields/fields.yml

@@ -0,0 +1,840 @@
+- name: f5_bigip.log


Is there documentation available for the definitions of the use of these fields in the f5 docs?

We have gone through the Telemetry Streaming doc which contains the sample events having only field names but no description.

efd6 · 2022-10-17T01:49:10Z

packages/f5_bigip/data_stream/log/sample_event.json

+            "response": {
+                "code": 0
+            },


I'm wondering how many of these nested fields with no siblings need to be nested (there are a fair few of them).

We are getting many other fields like “ResponseDuration” and we have grouped those fields under the name “response”, so we kept it as a type “group”.

packages/f5_bigip/manifest.yml

jamiehynds · 2022-10-20T09:41:29Z

@vinit-elastic we currently have an F5 integration which supports 2 BIG-IP products. Rather than shipping a new package, do you think we could upgrade the current package with the new/updated pipelines? This avoids having to deprecate the current F5 package and ensures users can easily access the improved integration, without having to move to a new integration.

vinit-chauhan · 2022-10-31T09:51:38Z

Hey Jamie - The older integration uses the F5 default input (fetch logs directly from the F5 instance) and the new integration leverages the Telematry Streaming Service. Keeping both in the same package might lead to confusion. Therefore, we thought of keeping it a separate connector. Let me know if you think otherwise.

jamiehynds · 2022-11-23T10:11:06Z

Hey Jamie - The older integration uses the F5 default input (fetch logs directly from the F5 instance) and the new integration leverages the Telematry Streaming Service. Keeping both in the same package might lead to confusion. Therefore, we thought of keeping it a separate connector. Let me know if you think otherwise.

Discussed offline, and agreed that we'll stick with a new F5 BIG-IP integration as the current experimental integration relies on syslog directly from ASM/AFM, whereas the new integration leverages the Telemetry Streaming Services. We'll be deprecating the experimental F5 module and integration once this new one is available.

P1llus

I think some overall discussion has to be done around ignore_failure, but that is not related to this single package, the rest is LGTM for now.
Great job!

elasticmachine · 2022-11-28T09:37:02Z

Package f5_bigip - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=f5_bigip

vinit-chauhan added 2 commits September 20, 2022 20:47

Initial Release for the F5 BIG-IP

34f67a9

Update changelog file

77ed38a

vinit-chauhan added enhancement New feature or request Team:Security-External Integrations New Integration labels Sep 20, 2022

vinit-chauhan marked this pull request as draft September 20, 2022 16:30

vinit-chauhan added 2 commits October 7, 2022 17:41

Updated the pipeline by adding new fields, tag and visualizations

6a6c397

Add new fields in ASM and APM category, Update field-mapping and form…

2a56520

…at_version

vinit-chauhan marked this pull request as ready for review October 14, 2022 13:34

efd6 reviewed Oct 17, 2022

View reviewed changes

Update the PR as per the review comments

dc2e560

Add base-fields file

e4417a8

vinit-chauhan added 2 commits November 10, 2022 11:10

Merge branch 'main' into package_f5_bigip

a6f7f6f

Update CODEOWNERS

1cd1b73

P1llus approved these changes Nov 28, 2022

View reviewed changes

P1llus merged commit 2fb1cba into elastic:main Nov 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[f5_bigip] Initial Release for the F5 BIG-IP #4242

[f5_bigip] Initial Release for the F5 BIG-IP #4242

vinit-chauhan commented Sep 20, 2022

elasticmachine commented Sep 20, 2022

elasticmachine commented Sep 20, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Sep 20, 2022

elasticmachine commented Sep 20, 2022 •

edited

Loading

efd6 Oct 17, 2022

efd6 Oct 17, 2022

vinit-chauhan Oct 18, 2022

efd6 Oct 17, 2022

vinit-chauhan Oct 18, 2022

efd6 Oct 17, 2022

efd6 Oct 17, 2022

vinit-chauhan Oct 18, 2022

efd6 Oct 17, 2022

vinit-chauhan Oct 18, 2022

jamiehynds commented Oct 20, 2022

vinit-chauhan commented Oct 31, 2022 •

edited

Loading

jamiehynds commented Nov 23, 2022

P1llus left a comment

elasticmachine commented Nov 28, 2022


		## Overview

		The [F5 BIG-IP](https://www.f5.com) integration allows users to monitor LTM, AFM, APM, ASM, and AVR activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions.

	The [F5 BIG-IP](https://www.f5.com) integration allows users to monitor LTM, AFM, APM, ASM, and AVR activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions.
	The [F5 BIG-IP](https://www.f5.com/products/big-ip-services) integration allows users to monitor LTM, AFM, APM, ASM, and AVR activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions.

[f5_bigip] Initial Release for the F5 BIG-IP #4242

[f5_bigip] Initial Release for the F5 BIG-IP #4242

Conversation

vinit-chauhan commented Sep 20, 2022

What does this PR do?

Integration release checklist

All changes

New Package

Dashboards changes

Log dataset changes

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Sep 20, 2022

elasticmachine commented Sep 20, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Sep 20, 2022

🚀 Benchmarks report

elasticmachine commented Sep 20, 2022 • edited Loading

🌐 Coverage report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jamiehynds commented Oct 20, 2022

vinit-chauhan commented Oct 31, 2022 • edited Loading

jamiehynds commented Nov 23, 2022

P1llus left a comment

Choose a reason for hiding this comment

elasticmachine commented Nov 28, 2022

elasticmachine commented Sep 20, 2022 •

edited

Loading

elasticmachine commented Sep 20, 2022 •

edited

Loading

vinit-chauhan commented Oct 31, 2022 •

edited

Loading