Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stormshield/create first integration #9337

Merged
merged 92 commits into from
Jun 10, 2024
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
3524466
[stormshield] Created new integration for EVA-1 logs
jrmolin Feb 23, 2024
f48c186
add the first data stream (logs)
jrmolin Feb 23, 2024
517a397
Merge branch 'main' of github.com:jrmolin/integrations into stormshie…
jrmolin Mar 7, 2024
60e0652
stash
jrmolin Mar 11, 2024
347d534
stash
jrmolin Mar 12, 2024
93ffd52
stash
jrmolin Mar 14, 2024
6922351
add the stormshield logo
jrmolin Mar 14, 2024
41a8887
stash
jrmolin Mar 21, 2024
205cefd
Merge branch 'main' of github.com:jrmolin/integrations into stormshie…
jrmolin Mar 27, 2024
e65d818
stash
jrmolin Mar 27, 2024
7ade27b
got kibana to stop stacktracing again
jrmolin Mar 27, 2024
fbf455e
closer to getting system tests
jrmolin Mar 29, 2024
3215fa7
updates to get pipeline tests passing
jrmolin Mar 30, 2024
0b0a9a7
some things working better
jrmolin Apr 15, 2024
7654e30
another stage completed
jrmolin Apr 15, 2024
c80f82c
passing system tests now
jrmolin Apr 16, 2024
a0a354f
clean up some pipeline stuff
jrmolin Apr 16, 2024
1c4e90c
add the description text to the fields in the mapping
jrmolin Apr 17, 2024
54968f3
got all the kinks worked out with the parsing script
jrmolin Apr 18, 2024
fab7829
remove extra pipelines and replace with painless scripting
jrmolin Apr 18, 2024
21fa41e
stash
jrmolin Apr 22, 2024
3eb3362
add screenshot
jrmolin Apr 23, 2024
63da61e
clean up readme a little
jrmolin Apr 23, 2024
2302a5c
add an example log
jrmolin Apr 24, 2024
b42e83b
update the version to make PR merges better
jrmolin Apr 24, 2024
8e9f235
update CODEOWNERS to indicate who owns the stormshield integration
jrmolin Apr 24, 2024
5350d47
update owner in manifest.yml
jrmolin Apr 24, 2024
219c3a9
make the version valid
jrmolin Apr 24, 2024
5ffb354
add another dynamic field
jrmolin Apr 24, 2024
8ecb2b4
update sample event
jrmolin Apr 25, 2024
b8ad422
code review updates, and add a tcp option for the agent policy
jrmolin Apr 25, 2024
6ce723b
more docs wordsmithing
jrmolin Apr 25, 2024
0278b63
stash
jrmolin Apr 30, 2024
74685d1
add system test for tcp input
jrmolin Apr 30, 2024
f901f5a
shrinkify fields.yml by combining duplicates
jrmolin May 2, 2024
6120339
linting and readme generation
jrmolin May 2, 2024
aa99481
update the pipeline with code review suggestions
jrmolin May 2, 2024
4802890
updated fields.yml
jrmolin May 6, 2024
991d664
flatten fields.yml
jrmolin May 6, 2024
74498da
convert more fields to be ecs-compatible
jrmolin May 6, 2024
bb719aa
revert the version
jrmolin May 7, 2024
2fd62fb
add tls support
jrmolin May 8, 2024
dca32f6
passing system tests now
jrmolin May 8, 2024
efa9dfa
rename more to ecs
jrmolin May 9, 2024
c8da5ad
convert numeric fields
jrmolin May 10, 2024
6b62cf6
consolidate the test files
jrmolin May 13, 2024
1fc5776
get system tests passing again
jrmolin May 13, 2024
afa1d56
use a newer version of the stream container
jrmolin May 15, 2024
2ef69cd
add more aggregations
jrmolin May 16, 2024
da07ac6
got a dashboard passing muster
jrmolin May 20, 2024
00241c5
remove unnecessary control
jrmolin May 20, 2024
07316ec
added a mean-throughput panel
jrmolin May 20, 2024
15ea2d9
added another throughput chart
jrmolin May 20, 2024
7bb0d20
refactor for the ethernet devices to be separate
jrmolin May 24, 2024
55610d7
dashboard stash
jrmolin May 24, 2024
a38e92a
more panels
jrmolin May 24, 2024
6ab193d
stash
jrmolin May 24, 2024
b074374
update dashboards
jrmolin May 28, 2024
f7d7446
new dashboard
jrmolin May 28, 2024
c14d761
added events treemap
jrmolin May 28, 2024
97277da
add a treemap of log.level
jrmolin May 28, 2024
271decd
add a filter to the dashboard
jrmolin May 28, 2024
1a683a6
more and more
jrmolin May 28, 2024
c650492
another control added
jrmolin May 28, 2024
74842ef
update dashboard again
jrmolin May 29, 2024
90e4cbd
flatten some fields
jrmolin May 31, 2024
1f43247
fix up some dashboard stuff
jrmolin May 31, 2024
803c02f
updated dashboards with working fields and better layout and controls
jrmolin Jun 4, 2024
45019b5
now it fits on one screen without scrolling
jrmolin Jun 4, 2024
3c25190
now with a dashboard screenshot
jrmolin Jun 4, 2024
3fc9379
add some more panels
jrmolin Jun 4, 2024
0a319ad
updated fields and pipeline
jrmolin Jun 4, 2024
811ac4f
add geo fields
jrmolin Jun 4, 2024
1ae2db8
added geo fields and updated dashboard to show the geo fields
jrmolin Jun 5, 2024
1199cc4
more fixes with pipeline, fields, dashboard
jrmolin Jun 5, 2024
a738871
another slight tweak
jrmolin Jun 5, 2024
5268621
one more fix
jrmolin Jun 5, 2024
be23603
update the sample event
jrmolin Jun 6, 2024
398f0d4
update screenshots
jrmolin Jun 6, 2024
ed25d57
updated dashboard capture
jrmolin Jun 6, 2024
77b71f0
updated screenshots
jrmolin Jun 6, 2024
2dfee56
fixup the sample event and ingest pipeline
jrmolin Jun 6, 2024
946483d
code review comments
jrmolin Jun 6, 2024
49d4080
code review comments
jrmolin Jun 7, 2024
ead2592
remove dead link from docs and more pipeline cleanup
jrmolin Jun 7, 2024
aa04de5
fix a pipeline script off-by-one error
jrmolin Jun 7, 2024
9e8a8a4
add another test and update the pipeline to handle nat better
jrmolin Jun 7, 2024
c9dbbde
add a tags.yml file
jrmolin Jun 7, 2024
458377c
fix the pipeline to keep nat ip/port pairs together
jrmolin Jun 7, 2024
c1ebedd
update source ip/port pairs for nat
jrmolin Jun 7, 2024
5b638e0
add some related.ip data
jrmolin Jun 7, 2024
dbf06b0
added some related fields
jrmolin Jun 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,7 @@
/packages/squid @elastic/sec-deployment-and-devices
/packages/stan @elastic/obs-infraobs-integrations
/packages/statsd_input @elastic/obs-infraobs-integrations
/packages/stormshield @elastic/sec-deployment-and-devices
/packages/suricata @elastic/sec-deployment-and-devices
/packages/symantec_edr_cloud @elastic/security-service-integrations
/packages/symantec_endpoint @elastic/security-service-integrations
Expand Down
93 changes: 93 additions & 0 deletions packages/stormshield/LICENSE.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you don't need this, since this says packages are licensed under the Elastic license by default

Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
Elastic License 2.0

URL: https://www.elastic.co/licensing/elastic-license

## Acceptance

By using the software, you agree to all of the terms and conditions below.

## Copyright License

The licensor grants you a non-exclusive, royalty-free, worldwide,
non-sublicensable, non-transferable license to use, copy, distribute, make
available, and prepare derivative works of the software, in each case subject to
the limitations and conditions below.

## Limitations

You may not provide the software to third parties as a hosted or managed
service, where the service provides users with access to any substantial set of
the features or functionality of the software.

You may not move, change, disable, or circumvent the license key functionality
in the software, and you may not remove or obscure any functionality in the
software that is protected by the license key.

You may not alter, remove, or obscure any licensing, copyright, or other notices
of the licensor in the software. Any use of the licensor’s trademarks is subject
to applicable law.

## Patents

The licensor grants you a license, under any patent claims the licensor can
license, or becomes able to license, to make, have made, use, sell, offer for
sale, import and have imported the software, in each case subject to the
limitations and conditions in this license. This license does not cover any
patent claims that you cause to be infringed by modifications or additions to
the software. If you or your company make any written claim that the software
infringes or contributes to infringement of any patent, your patent license for
the software granted under these terms ends immediately. If your company makes
such a claim, your patent license ends immediately for work on behalf of your
company.

## Notices

You must ensure that anyone who gets a copy of any part of the software from you
also gets a copy of these terms.

If you modify the software, you must include in any modified copies of the
software prominent notices stating that you have modified the software.

## No Other Rights

These terms do not imply any licenses other than those expressly granted in
these terms.

## Termination

If you use the software in violation of these terms, such use is not licensed,
and your licenses will automatically terminate. If the licensor provides you
with a notice of your violation, and you cease all violation of this license no
later than 30 days after you receive that notice, your licenses will be
reinstated retroactively. However, if you violate these terms after such
reinstatement, any additional violation of these terms will cause your licenses
to terminate automatically and permanently.

## No Liability

*As far as the law allows, the software comes as is, without any warranty or
condition, and the licensor will not be liable to you for any damages arising
out of these terms or the use or nature of the software, under any kind of
legal claim.*

## Definitions

The **licensor** is the entity offering these terms, and the **software** is the
software the licensor makes available under these terms, including any portion
of it.

**you** refers to the individual or entity agreeing to these terms.

**your company** is any legal entity, sole proprietorship, or other kind of
organization that you work for, plus all organizations that have control over,
are under the control of, or are under common control with that
organization. **control** means ownership of substantially all the assets of an
entity, or the power to direct its management and policies by vote, contract, or
otherwise. Control can be direct or indirect.

**your licenses** are all the licenses granted to you for the software under
these terms.

**use** means anything you do with the software requiring one of your licenses.

**trademark** means trademarks, service marks, and similar rights.
3 changes: 3 additions & 0 deletions packages/stormshield/_dev/build/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
dependencies:
ecs:
reference: "git@v8.11.0"
34 changes: 34 additions & 0 deletions packages/stormshield/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Stormshield SNS

Stormshield Network Security (SNS) firewalls are a stable and efficient security solution to protect corporate networks from cyberattacks. Real-time protection (intrusion prevention and detection, application control, antivirus, etc.), control and supervision (URL filtering, IP geolocation, vulnerability detection, etc.) and content filtering (antispam, antispyware, antiphishing, etc.) all guarantee secure communications. All Stormshield Network Security firewalls are based on the same firmware, and with their core features, Stormshield Network Security firewalls give you comprehensive security and high performance network protection.

Use the Stormshield SNS integration to ingest log data into Elastic Security and leverage the data for threat detection, incident response, and visualization.


## Data streams

The Stormshield SNS integration collects audit, traffic, and connection (including NAT) logs. Available log types are available here: https://documentation.stormshield.eu/SNS/v4/en/Content/Description_of_Audit_logs/Configure_logs.htm .


**Logs** help you keep a record of events happening in your firewalls.
The SNS integration handles activity logs and firewall (filter and NAT) logs. See more details in the [Logs](#logs-reference).

## Requirements

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

## Setup

The SNS integration ingests logs via a syslog parser, so the SNS appliance needs to be configured to send syslogs to a listening Agent. This is configured in the `CONFIGURATION` tab, in the `NOTIFICATIONS` / `LOGS-SYSLOG-IPFIX` section. Please review the Stormshield documentation for details on how to configure syslog: https://documentation.stormshield.eu/SNS/v4/en/Content/Description_of_Audit_logs/Configure_logs.htm.

For step-by-step instructions on how to set up an integration, see the
[Getting started](https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html) guide.

## Log

SNS can be configured to store all its logs locally, or to send them through the syslog protocol to a configured listener, such as the Elastic Agent via policy update.

{{ event "log" }}

{{ fields "log" }}
20 changes: 20 additions & 0 deletions packages/stormshield/_dev/deploy/docker/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
version: "2.3"
services:
stormshield-udp:
image: docker.elastic.co/observability/stream:v0.9.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
image: docker.elastic.co/observability/stream:v0.9.0
image: docker.elastic.co/observability/stream:v0.16.0

v0.16.0 exists now, and has support for arm, so it would be nice to use that

volumes:
- ./sample_logs:/sample_logs:ro
entrypoint: /bin/bash
command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:5144 -p=udp /sample_logs/stormshield.log"
stormshield-tcp:
image: docker.elastic.co/observability/stream:v0.9.0
volumes:
- ./sample_logs:/sample_logs:ro
entrypoint: /bin/bash
command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:6011 -p=tcp /sample_logs/stormshield.log"
stormshield-tls:
image: docker.elastic.co/observability/stream:v0.9.0
volumes:
- ./sample_logs:/sample_logs:ro
entrypoint: /bin/bash
command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:6514 -p=tls --insecure /sample_logs/stormshield.log"
Loading