diff --git a/crowdsec-docs/docs/data_sources/http.md b/crowdsec-docs/docs/data_sources/http.md new file mode 100644 index 000000000..9ca806a89 --- /dev/null +++ b/crowdsec-docs/docs/data_sources/http.md @@ -0,0 +1,191 @@ +--- +id: http +title: HTTP +--- + +This module allows the `Security Engine` to acquire logs from an HTTP endpoint. + +## Configuration examples + +To receive logs from an HTTP endpoint with basic auth: +```yaml +source: http +listen_addr: 127.0.0.1:8080 +path: /test +auth_type: basic_auth +basic_auth: + username: test + password: test +labels: + type: mytype +``` + +To receive logs from an HTTP endpoint with headers: +```yaml +source: http +listen_addr: 127.0.0.1:8080 +path: /test +auth_type: headers +headers: + MyHeader: MyValue +labels: + type: mytype +``` + +To receive logs from an HTTP endpoint with TLS and headers: + +```yaml +source: http +listen_addr: 127.0.0.1:8080 +path: /test +auth_type: headers +headers: + MyHeader: MyValue +tls: + server_cert: server.crt + server_key: server.key +labels: + type: mytype +``` + +To receive logs from an HTTP endpoint with mTLS: + +```yaml +source: http +listen_addr: 127.0.0.1:8080 +path: /test +auth_type: mtls +tls: + server_cert: server.crt + server_key: server.key + ca_cert: ca.crt +labels: + type: mytype +``` + +:::info +If most of cases when the logs are sent in JSON format, you can use the [`transform`](https://docs.crowdsec.net/docs/next/data_sources/intro/#transform) expression to parse the logs. + +For example, if the logs are sent in the following format: +```json +{ + "Records": [ + { + "message": "test", + "timestamp": "2021-01-01T00:00:00Z" + } + ] +} +``` + +the `transform` expression can be: +```yaml +transform: map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#)) +``` + + +Look at the `configuration parameters` to view all supported options. + +## Parameters + + +### `listen_addr` + +The address to listen on (e.g., `127.0.0.1:8088`). + +Required. + +### `path` + +The endpoint path to listen on. + +:::info +The request method is always `POST`. +::: + +Optional, default is `/`. + +### `auth_type` + +The authentication type to use. + +Can be `basic_auth`, `headers`, or `mtls`. + +Required. + +### `basic_auth` + +The basic auth credentials. + +### `basic_auth.username` + +The basic auth username. + +Optional, to use when `auth_type` is `basic_auth`. + +### `basic_auth.password` + +The basic auth password. + +Optional, to use when `auth_type` is `basic_auth`. + +### `headers` + +The headers to send. + +Optional, to use when `auth_type` is `headers`. + +### `tls` + +TLS configuration. + +### `tls.server_cert` + +The server certificate path. + +Optional, to use when `auth_type` is `mtls`. + +### `tls.server_key` + +The server key path. + +Optional, to use when `auth_type` is `mtls`. + +### `tls.ca_cert` + +The CA certificate path. + +Optional, to use when `auth_type` is `mtls`. + +### `custom_status_code` + +The custom status code to return. + +Optional. + +### `custom_headers` + +The custom headers to return. + +Optional. + +### `max_body_size` + +The maximum body size to accept. + +Optional. + +### `timeout` + +The timeout to read the body. + +:::info +The timeout is in duration format, e.g., `5s`. +::: + +Optional. + +## DSN and command-line + +This datasource does not support acquisition from the command line. + diff --git a/crowdsec-docs/docs/data_sources/introduction.md b/crowdsec-docs/docs/data_sources/introduction.md index 6b9373320..2541a21a8 100644 --- a/crowdsec-docs/docs/data_sources/introduction.md +++ b/crowdsec-docs/docs/data_sources/introduction.md @@ -18,6 +18,7 @@ Name | Type | Stream | One-shot [AWS S3](/data_sources/s3.md)| read logs from a S3 bucket | yes | yes [docker](/data_sources/docker.md) | read logs from docker containers | yes | yes [file](/data_sources/file.md) | single files, glob expressions and .gz files | yes | yes +[HTTP](/data_sources/http.md) | read logs from an HTTP endpoint | yes | no [journald](/data_sources/journald.md) | journald via filter | yes | yes [Kafka](/data_sources/kafka.md)| read logs from kafka topic | yes | no [Kubernetes Audit](/data_sources/kubernetes_audit.md) | expose a webhook to receive audit logs from a Kubernetes cluster | yes | no diff --git a/crowdsec-docs/sidebars.js b/crowdsec-docs/sidebars.js index 9b70103db..4c4d8f4c5 100644 --- a/crowdsec-docs/sidebars.js +++ b/crowdsec-docs/sidebars.js @@ -131,6 +131,7 @@ "data_sources/s3", "data_sources/docker", "data_sources/file", + "data_sources/http", "data_sources/journald", "data_sources/kafka", "data_sources/kubernetes_audit", diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/http.md b/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/http.md new file mode 100644 index 000000000..9ca806a89 --- /dev/null +++ b/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/http.md @@ -0,0 +1,191 @@ +--- +id: http +title: HTTP +--- + +This module allows the `Security Engine` to acquire logs from an HTTP endpoint. + +## Configuration examples + +To receive logs from an HTTP endpoint with basic auth: +```yaml +source: http +listen_addr: 127.0.0.1:8080 +path: /test +auth_type: basic_auth +basic_auth: + username: test + password: test +labels: + type: mytype +``` + +To receive logs from an HTTP endpoint with headers: +```yaml +source: http +listen_addr: 127.0.0.1:8080 +path: /test +auth_type: headers +headers: + MyHeader: MyValue +labels: + type: mytype +``` + +To receive logs from an HTTP endpoint with TLS and headers: + +```yaml +source: http +listen_addr: 127.0.0.1:8080 +path: /test +auth_type: headers +headers: + MyHeader: MyValue +tls: + server_cert: server.crt + server_key: server.key +labels: + type: mytype +``` + +To receive logs from an HTTP endpoint with mTLS: + +```yaml +source: http +listen_addr: 127.0.0.1:8080 +path: /test +auth_type: mtls +tls: + server_cert: server.crt + server_key: server.key + ca_cert: ca.crt +labels: + type: mytype +``` + +:::info +If most of cases when the logs are sent in JSON format, you can use the [`transform`](https://docs.crowdsec.net/docs/next/data_sources/intro/#transform) expression to parse the logs. + +For example, if the logs are sent in the following format: +```json +{ + "Records": [ + { + "message": "test", + "timestamp": "2021-01-01T00:00:00Z" + } + ] +} +``` + +the `transform` expression can be: +```yaml +transform: map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#)) +``` + + +Look at the `configuration parameters` to view all supported options. + +## Parameters + + +### `listen_addr` + +The address to listen on (e.g., `127.0.0.1:8088`). + +Required. + +### `path` + +The endpoint path to listen on. + +:::info +The request method is always `POST`. +::: + +Optional, default is `/`. + +### `auth_type` + +The authentication type to use. + +Can be `basic_auth`, `headers`, or `mtls`. + +Required. + +### `basic_auth` + +The basic auth credentials. + +### `basic_auth.username` + +The basic auth username. + +Optional, to use when `auth_type` is `basic_auth`. + +### `basic_auth.password` + +The basic auth password. + +Optional, to use when `auth_type` is `basic_auth`. + +### `headers` + +The headers to send. + +Optional, to use when `auth_type` is `headers`. + +### `tls` + +TLS configuration. + +### `tls.server_cert` + +The server certificate path. + +Optional, to use when `auth_type` is `mtls`. + +### `tls.server_key` + +The server key path. + +Optional, to use when `auth_type` is `mtls`. + +### `tls.ca_cert` + +The CA certificate path. + +Optional, to use when `auth_type` is `mtls`. + +### `custom_status_code` + +The custom status code to return. + +Optional. + +### `custom_headers` + +The custom headers to return. + +Optional. + +### `max_body_size` + +The maximum body size to accept. + +Optional. + +### `timeout` + +The timeout to read the body. + +:::info +The timeout is in duration format, e.g., `5s`. +::: + +Optional. + +## DSN and command-line + +This datasource does not support acquisition from the command line. + diff --git a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/introduction.md b/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/introduction.md index 6b9373320..2541a21a8 100644 --- a/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/introduction.md +++ b/crowdsec-docs/versioned_docs/version-v1.6.0/data_sources/introduction.md @@ -18,6 +18,7 @@ Name | Type | Stream | One-shot [AWS S3](/data_sources/s3.md)| read logs from a S3 bucket | yes | yes [docker](/data_sources/docker.md) | read logs from docker containers | yes | yes [file](/data_sources/file.md) | single files, glob expressions and .gz files | yes | yes +[HTTP](/data_sources/http.md) | read logs from an HTTP endpoint | yes | no [journald](/data_sources/journald.md) | journald via filter | yes | yes [Kafka](/data_sources/kafka.md)| read logs from kafka topic | yes | no [Kubernetes Audit](/data_sources/kubernetes_audit.md) | expose a webhook to receive audit logs from a Kubernetes cluster | yes | no