add citrix_waf package #3425

efd6 · 2022-05-25T06:57:31Z

What does this PR do?

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Dashboard will be in a follow-up PR
Native Citrix format extended data is not handled as I have been unable to find a specification for it. If this becomes available it will also be in a follow-up PR. From what I can see with other ingesters, people tell users to configure their Citrix device to send logs in CEF.

How to test this PR locally

Related issues

Updates Citrix Web App Firewall #1308

Screenshots

elasticmachine · 2022-05-25T07:02:01Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-07-20T21:22:12.115+0000
Duration: 19 min 50 sec

Test stats 🧪

Test	Results
Failed	0
Passed	9
Skipped	0
Total	9

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-05-25T09:46:07Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`3/3`)	💚 3.012
Classes	100.0% (`3/3`)	💚 3.012
Methods	100.0% (`21/21`)	💚 10.95
Lines	97.561% (`240/246`)	👍 7.082
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2022-06-21T07:08:48Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

packages/citrix_waf/_dev/build/docs/README.md

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/default.yml

packages/citrix_waf/data_stream/log/manifest.yml

legoguy1000 · 2022-07-11T01:52:56Z

packages/citrix_waf/data_stream/log/_dev/test/pipeline/citrix_waf_cef.log-expected.json

+                "message": "CEF:0|Citrix|NetScaler|NS11.0|APPFW|APPFW_STARTURL|6|src=175.16.199.1 spt=47606 method=GET request=http://aaron.stratum8.net/FFC/login.html msg=Disallow Illegal URL. cn1=1340 cn2=653 cs1=pr_ffc cs2=PPE1 cs3=EsdGd3VD0OaaURLcZnj05Y6DOmE0002 cs4=ALERT cs5=2015 act=blocked",
+                "severity": 6,
+                "timezone": "UTC",
+                "url": {


I think this is an error event.url?

legoguy1000 · 2022-07-11T01:55:05Z

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/cef.yml

+  - rename:
+      # request – request URL
+      field: citrix.extended_kv.request
+      target_field: event.url.original


probably should be url.original

legoguy1000 · 2022-07-11T01:55:56Z

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/default.yml

+  - uri_parts:
+      field: event.url.original
+      target_field: event.url
+      ignore_failure: true


same about the url not event.url??

P1llus · 2022-07-11T08:48:45Z

packages/citrix_waf/changelog.yml

@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: "0.0.1"


Any specific reason we would want to start off with experimental, compared to beta or ga?

I usually release mine as GA as long as you have functional system and pipeline tests with good test data.
If not then I release as beta, unless again it has new functionality like a new filebeat input.

I will bump to beta until I have the dashboard in place — next PR.

P1llus · 2022-07-11T08:52:24Z

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/cef.yml

+description: Pipeline for Citrix CEF messages
+processors:
+  - set:
+      field: event.cef_format


Please dont use ECS root fields for non ECS fields, better then to use citrix.*

P1llus · 2022-07-11T08:55:10Z

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/default.yml

+      override: false
+  - set:
+      field: event.timezone
+      value: '{{{_conf.tz_offset}}}'


You could also use copy_from rather than using a template value, as it is safer.

P1llus · 2022-07-11T08:55:39Z

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/default.yml

+      field: _tmp.timestamp
+      if: ctx._tmp?.timestamp != null && ctx.citrix?.event_year != null
+      value: "{{{citrix.event_year}}} {{{_tmp.timestamp}}}"
+  - date:


Date formats are quite expensive, are you sure we need all of them?

I have seen a wide variety in the documentation and it is deeply underspecified.

P1llus · 2022-07-11T08:56:21Z

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/native.yml

+description: Pipeline for Citrix Native messages
+processors:
+  - set:
+      field: event.cef_format


Same here with using a non ECS root field

P1llus · 2022-07-11T08:56:31Z

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/native.yml

+      value: false
+  - grok:
+      description: Extract native header and message.
+      field: event.message


Same here with using a non ECS root field

P1llus · 2022-07-11T08:58:27Z

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/default.yml

+      field: citrix.event_year
+      ignore_failure: true
+
+  - geoip:


We have a common set of geoip processors that we reuse on a lot of our ingest pipelines, is this something we could use instead? Feel free to modify it to fit your usecase.

# IP Geolocation Lookup - geoip: field: source.ip target_field: source.geo ignore_missing: true - geoip: field: destination.ip target_field: destination.geo ignore_missing: true # IP Autonomous System (AS) Lookup - geoip: database_file: GeoLite2-ASN.mmdb field: source.ip target_field: source.as properties: - asn - organization_name ignore_missing: true - geoip: database_file: GeoLite2-ASN.mmdb field: destination.ip target_field: destination.as properties: - asn - organization_name ignore_missing: true - rename: field: source.as.asn target_field: source.as.number ignore_missing: true - rename: field: source.as.organization_name target_field: source.as.organization.name ignore_missing: true - rename: field: destination.as.asn target_field: destination.as.number ignore_missing: true - rename: field: destination.as.organization_name target_field: destination.as.organization.name ignore_missing: true

P1llus · 2022-07-11T08:59:32Z

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/default.yml

+  - uri_parts:
+      field: url.original
+      target_field: url
+      ignore_failure: true


We usually try to not use ignore_failures, as both the CI and the end user wont tell us if there are any errors. Are there any other ways we can do this? like a IF statement?

P1llus · 2022-07-11T08:59:56Z

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/default.yml

+      field:
+        - _tmp
+        - _conf
+      ignore_failure: true


Could you change this to ignore_missing?

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/default.yml

andrewkroh · 2022-07-20T01:11:58Z

packages/citrix_waf/data_stream/log/manifest.yml

+        required: false
+        show_user: false
+        default: |
+          #max_connections: 1


One setting I recommend for both UDP and TCP is to include a max_message_size. IIRC the default value is like 20 MiB which is way too big and wastes memory.

The WAF messages are fairly small, so I've suggested 10kB and also added a udp_options field.

Sounds good. I suggest having that option uncommented so most users are using it by default when they enable this.

andrewkroh · 2022-07-20T22:19:31Z

packages/citrix_waf/data_stream/log/manifest.yml

+        show_user: false
+        default: |
+          max_message_size: 10KiB
+          read_buffer_size: 10KiB


I didn't find any evidence that this read_buffer_size parameter exists. Do you know where this is used in the code?

This is used in the UDP package. ~~Apparently it is useless there too.~~ Nope, called read_buffer in beats.

efd6 added enhancement New feature or request Team:Security-External Integrations labels May 25, 2022

efd6 self-assigned this May 25, 2022

efd6 force-pushed the 1308-citrix_waf branch 3 times, most recently from aa0aa06 to 7414d01 Compare May 25, 2022 09:30

efd6 force-pushed the 1308-citrix_waf branch from 7414d01 to 54b5672 Compare June 21, 2022 04:48

efd6 changed the title ~~wip~~ add citrix_waf package Jun 21, 2022

efd6 force-pushed the 1308-citrix_waf branch from 54b5672 to 26543d0 Compare June 21, 2022 07:01

efd6 requested a review from a team June 21, 2022 07:05

efd6 added the Integration:citrix_waf Citrix Web App Firewall label Jun 21, 2022

efd6 marked this pull request as ready for review June 21, 2022 07:08

efd6 force-pushed the 1308-citrix_waf branch 2 times, most recently from fead6ac to 700931e Compare June 21, 2022 23:34

taylor-swanson reviewed Jun 22, 2022

View reviewed changes

packages/citrix_waf/_dev/build/docs/README.md Outdated Show resolved Hide resolved

packages/citrix_waf/data_stream/log/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

packages/citrix_waf/data_stream/log/manifest.yml Show resolved Hide resolved

efd6 added 2 commits June 29, 2022 15:07

citrix_waf: add package

5923fa6

address pr comments

f32e229

efd6 force-pushed the 1308-citrix_waf branch from 700931e to f32e229 Compare June 29, 2022 06:38

update ecs version

c15ebd6

efd6 requested a review from taylor-swanson June 29, 2022 06:58

efd6 added 2 commits June 30, 2022 09:40

fix formatting

8bdd143

fix test case format

2b4f716

legoguy1000 reviewed Jul 11, 2022

View reviewed changes

address pr comments

1d17ed5

P1llus reviewed Jul 11, 2022

View reviewed changes

address pr comment

c7c0bcc

efd6 requested a review from P1llus July 12, 2022 07:10

andrewkroh approved these changes Jul 20, 2022

View reviewed changes

add max message suggestion and udp options

9f51815

efd6 requested a review from andrewkroh July 20, 2022 02:17

make 10kb message size the defaultish

b0fc7b4

efd6 merged commit 404ceb2 into elastic:main Jul 20, 2022

efd6 deleted the 1308-citrix_waf branch July 20, 2022 22:19

andrewkroh reviewed Jul 20, 2022

View reviewed changes

andrewkroh mentioned this pull request Jul 26, 2022

[production] Promote many packages from snapshot elastic/package-storage#5004

Merged

andrewkroh added the New Integration label Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add citrix_waf package #3425

add citrix_waf package #3425

efd6 commented May 25, 2022 •

edited

Loading

elasticmachine commented May 25, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented May 25, 2022 •

edited

Loading

elasticmachine commented Jun 21, 2022

legoguy1000 Jul 11, 2022

legoguy1000 Jul 11, 2022

legoguy1000 Jul 11, 2022

P1llus Jul 11, 2022

P1llus Jul 11, 2022

efd6 Jul 11, 2022

P1llus Jul 11, 2022

P1llus Jul 11, 2022

P1llus Jul 11, 2022

efd6 Jul 11, 2022

P1llus Jul 11, 2022

P1llus Jul 11, 2022

P1llus Jul 11, 2022

P1llus Jul 11, 2022

P1llus Jul 11, 2022

andrewkroh Jul 20, 2022 •

edited

Loading

efd6 Jul 20, 2022

andrewkroh Jul 20, 2022

andrewkroh Jul 20, 2022

efd6 Jul 20, 2022 •

edited

Loading

add citrix_waf package #3425

add citrix_waf package #3425

Conversation

efd6 commented May 25, 2022 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented May 25, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 25, 2022 • edited Loading

🌐 Coverage report

elasticmachine commented Jun 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh Jul 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efd6 Jul 20, 2022 • edited Loading

Choose a reason for hiding this comment

efd6 commented May 25, 2022 •

edited

Loading

elasticmachine commented May 25, 2022 •

edited

Loading

elasticmachine commented May 25, 2022 •

edited

Loading

andrewkroh Jul 20, 2022 •

edited

Loading

efd6 Jul 20, 2022 •

edited

Loading