Improve events ingestion pipeline #150
Assignees
Labels
Comments
AlexRuiz7
added
level/task
|
We can disallow creation of indices altogether:
References: |
|
The following disables new index creation for PUT _cluster/settings
{
"persistent": {
"action.auto_create_index": "-wazuh-alerts-*"
}
} |
|
It's possible to prevent Filebeat from creating indices using a dedicated user with the necessary permissions, instead of using the
To verify that Filebeat cannot create new indices, but can index data to existing ones, edit {
"set": {
"field": "_index",
"value": "wazuh-alerts"
}
},with any other value to use as index name (the index must not exist). For example: {
"set": {
"field": "_index",
"value": "wazuh-alerts-filebeat"
}
},Then run Now, if we revert the pipelines back to Before After Notes: I also added References: |
|
We can apply these settings by default, adding each chunk of code to the corresponding file: # roles.yml
filebeat_writer:
reserved: true
hidden: false
cluster_permissions:
- "cluster_monitor"
- "cluster_manage_pipelines"
index_permissions:
- index_patterns:
- "wazuh-alerts-4.x-*"
- "wazuh-archives-4.x-*"
allowed_actions:
- "index"
# internal_users.yml
filebeat:
reserved: true
hidden: false
hash: $2y$12$Xiobfelar.b0WAaVXz8Vx.go8Zfu..Oieh/ctlA8lX2s5uGsMeE9S
# roles_mapping.yml
filebeat_writer:
reserved: true
hidden: false
users:
- "filebeat"
# filebeat.yml
setup.ilm.check_exists: falseNote: credentials are I've tested these settings and the creation of the role and user is successful. Also, Filebeat holds the events and retries indefinitely until the index exists. Testing that is as simple as removing the |
|
Commenting out |
|
If we include this in the next stage of 4.8.0, the plan would be:
I blocked this issue for discussion with management. cc: @gdiazlo |
4 tasks
|
We are unable to verify that Filebeat retries indefinitely and doesn't drop the events. We've found several resources affirming so, however, it looks like the actual drop of events takes place in the indexer side, which rejects the event for its own reasons. If the indexer replies with any status code, Filebeat assumes it delivery as completed, as it could reach the output. As a result, the event is delivered but not indexed.
We have been able to confirm this behavior by stopping the indexer and seeing how Filebeat retries to reach it. |
|
In order to verify disabling
root@ubuntu2204:~# curl -k -u admin:admin -XPUT "https://127.0.0.1:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
"persistent": {
"action.auto_create_index": "-wazuh-alerts-*"
}
}'
{
"acknowledged" : true,
"persistent" : {
"action" : {
"auto_create_index" : "-wazuh-alerts-*"
}
},
"transient" : { }
}
root@ubuntu2204:~# curl -k -u admin:admin -XDELETE "https://127.0.0.1:9200/_ingest/pipeline/filebeat-7.10.2-wazuh-alerts-pipeline?pretty"
{
"acknowledged" : true
}
root@ubuntu2204:~# curl -k -u admin:admin -XDELETE "https://127.0.0.1:9200/wazuh-alerts-4.x-2024.02.14-000001/_alias/wazuh-alerts?pretty"
{
"acknowledged" : true
}
root@ubuntu2204:~# curl -k -u admin:admin -XGET "https://127.0.0.1:9200/_cat/aliases?pretty"
.kibana .kibana_1 - - - -
wazuh-archives wazuh-archives-4.x-2024.02.14-000001 - - - true
.opendistro-ism-managed-index-history-write .opendistro-ism-managed-index-history-2024.02.14-1 - - - -
root@ubuntu2204:~# curl -k -u admin:admin -XDELETE "https://127.0.0.1:9200/wazuh-alerts-4.x-*?pretty"
{
"acknowledged" : true
}
root@ubuntu2204:~# log='{"timestamp":"2024-02-14T13:49:15.552-0300","rule":{"level":3,"description":"sshd: authentication success.","id":"5715","mitre":{"id":["T1078","T1021"],"tactic":["Defense Evasion","Persistence","Privilege Escalation","Initial Access","Lateral Movement"],"technique":["Valid Accounts","Remote Services"]},"firedtimes":53,"mail":false,"groups":["syslog","sshd","authentication_success"],"gdpr":["IV_32.2"],"gpg13":["7.1","7.2"],"hipaa":["164.312.b"],"nist_800_53":["AU.14","AC.7"],"pci_dss":["10.2.5"],"tsc":["CC6.8","CC7.2","CC7.3"]},"agent":{"id":"000","name":"ubuntu2204"},"manager":{"name":"ubuntu2204"},"id":"9999999999.9999999","full_log":"Jan 30 11:19:16 ubuntu2204 sshd[221325]: Accepted publickey for root from 192.168.83.5 port 54094 ssh2: RSA SHA256:000000000000000/000000/00000000000000000000","predecoder":{"program_name":"sshd","timestamp":"Jan 30 11:19:16","hostname":"ubuntu2204"},"decoder":{"parent":"sshd","name":"sshd"},"data":{"srcip":"192.168.83.5","srcport":"54094","dstuser":"root"},"location":"syslog"}'
root@ubuntu2204:~# echo $log >> /var/ossec/logs/alerts/alerts.json
root@ubuntu2204:~# grep '9999999999.9999999' /var/log/filebeat/filebeat*
/var/log/filebeat/filebeat.1: "message": "{\"timestamp\":\"2024-02-14T13:49:15.552-0300\",\"rule\":{\"level\":3,\"description\":\"sshd: authentication success.\",\"id\":\"5715\",\"mitre\":{\"id\":[\"T1078\",\"T1021\"],\"tactic\":[\"Defense Evasion\",\"Persistence\",\"Privilege Escalation\",\"Initial Access\",\"Lateral Movement\"],\"technique\":[\"Valid Accounts\",\"Remote Services\"]},\"firedtimes\":53,\"mail\":false,\"groups\":[\"syslog\",\"sshd\",\"authentication_success\"],\"gdpr\":[\"IV_32.2\"],\"gpg13\":[\"7.1\",\"7.2\"],\"hipaa\":[\"164.312.b\"],\"nist_800_53\":[\"AU.14\",\"AC.7\"],\"pci_dss\":[\"10.2.5\"],\"tsc\":[\"CC6.8\",\"CC7.2\",\"CC7.3\"]},\"agent\":{\"id\":\"000\",\"name\":\"ubuntu2204\"},\"manager\":{\"name\":\"ubuntu2204\"},\"id\":\"9999999999.9999999\",\"full_log\":\"Jan 30 11:19:16 ubuntu2204 sshd[221325]: Accepted publickey for root from 192.168.83.5 port 54094 ssh2: RSA SHA256:000000000000000/000000/00000000000000000000\",\"predecoder\":{\"program_name\":\"sshd\",\"timestamp\":\"Jan 30 11:19:16\",\"hostname\":\"ubuntu2204\"},\"decoder\":{\"parent\":\"sshd\",\"name\":\"sshd\"},\"data\":{\"srcip\":\"192.168.83.5\",\"srcport\":\"54094\",\"dstuser\":\"root\"},\"location\":\"syslog\"}",
/var/log/filebeat/filebeat.1:2024-02-14T15:29:43.306-0300 WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc16b5f85906b7957, ext:3635343387376, loc:(*time.Location)(0x42417a0)}, Meta:{"pipeline":"filebeat-7.10.2-wazuh-alerts-pipeline"}, Fields:{"agent":{"ephemeral_id":"f2778bdf-53c5-462b-b68b-a392e9299298","hostname":"ubuntu2204","id":"54329818-a92c-48b1-a99b-7c91822aec19","name":"ubuntu2204","type":"filebeat","version":"7.10.2"},"ecs":{"version":"1.6.0"},"event":{"dataset":"wazuh.alerts","module":"wazuh"},"fields":{"index_prefix":"wazuh-alerts-4.x-"},"fileset":{"name":"alerts"},"host":{"name":"ubuntu2204"},"input":{"type":"log"},"log":{"file":{"path":"/var/ossec/logs/alerts/alerts.json"},"offset":877454},"message":"{\"timestamp\":\"2024-02-14T13:49:15.552-0300\",\"rule\":{\"level\":3,\"description\":\"sshd: authentication success.\",\"id\":\"5715\",\"mitre\":{\"id\":[\"T1078\",\"T1021\"],\"tactic\":[\"Defense Evasion\",\"Persistence\",\"Privilege Escalation\",\"Initial Access\",\"Lateral Movement\"],\"technique\":[\"Valid Accounts\",\"Remote Services\"]},\"firedtimes\":53,\"mail\":false,\"groups\":[\"syslog\",\"sshd\",\"authentication_success\"],\"gdpr\":[\"IV_32.2\"],\"gpg13\":[\"7.1\",\"7.2\"],\"hipaa\":[\"164.312.b\"],\"nist_800_53\":[\"AU.14\",\"AC.7\"],\"pci_dss\":[\"10.2.5\"],\"tsc\":[\"CC6.8\",\"CC7.2\",\"CC7.3\"]},\"agent\":{\"id\":\"000\",\"name\":\"ubuntu2204\"},\"manager\":{\"name\":\"ubuntu2204\"},\"id\":\"9999999999.9999999\",\"full_log\":\"Jan 30 11:19:16 ubuntu2204 sshd[221325]: Accepted publickey for root from 192.168.83.5 port 54094 ssh2: RSA SHA256:000000000000000/000000/00000000000000000000\",\"predecoder\":{\"program_name\":\"sshd\",\"timestamp\":\"Jan 30 11:19:16\",\"hostname\":\"ubuntu2204\"},\"decoder\":{\"parent\":\"sshd\",\"name\":\"sshd\"},\"data\":{\"srcip\":\"192.168.83.5\",\"srcport\":\"54094\",\"dstuser\":\"root\"},\"location\":\"syslog\"}","service":{"type":"wazuh"}}, Private:file.State{Id:"native::806327-64768", PrevId:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000400340), Source:"/var/ossec/logs/alerts/alerts.json", Offset:878479, Timestamp:time.Time{wall:0xc16b5f72cfd9e98c, ext:3560333847846, loc:(*time.Location)(0x42417a0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xc4db7, Device:0xfd00}, IdentifierName:"native"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=404): {"type":"index_not_found_exception","reason":"no such index [wazuh-alerts] and [action.auto_create_index] ([-wazuh-alerts-*]) doesn't match","index":"wazuh-alerts","index_uuid":"_na_"}
root@ubuntu2204:~# /usr/share/wazuh-indexer/bin/indexer-ism-init.sh -i localhost -p admin
Will create 'wazuh' index template
SUCC: 'wazuh' template created or updated
Will create 'ism_history_indices' index template
SUCC: 'ism_history_indices' template created or updated
Will disable replicas for 'plugins.index_state_management.history' indices
SUCC: cluster's settings saved
Will create index templates to configure the alias
SUCC: 'wazuh-alerts' template created or updated
SUCC: 'wazuh-archives' template created or updated
Will create the 'rollover_policy' policy
INFO: policy 'rollover_policy' already exists. Skipping policy creation
Will create initial indices for the aliases
SUCC: 'wazuh-alerts' write index created
INFO: 'wazuh-archives' write index already exists. Skipping write index creation
SUCC: Indexer ISM initialization finished successfully.
root@ubuntu2204:~# curl -k -u admin:admin -XGET "https://127.0.0.1:9200/_cat/aliases?pretty"
wazuh-alerts wazuh-alerts-4.x-2024.02.14-000001 - - - true
.kibana .kibana_1 - - - -
wazuh-archives wazuh-archives-4.x-2024.02.14-000001 - - - true
.opendistro-ism-managed-index-history-write .opendistro-ism-managed-index-history-2024.02.14-1 - - - -
root@ubuntu2204:~# filebeat setup --pipelines
Loaded Ingest pipelines
root@ubuntu2204:~# curl -k -u admin:admin -XGET "https://127.0.0.1:9200/_ingest/pipeline?pretty"
{
"filebeat-7.10.2-wazuh-alerts-pipeline" : {
"description" : "Wazuh alerts pipeline",
"processors" : [
{
"json" : {
"field" : "message",
"add_to_root" : true
}
},
{
"set" : {
"field" : "data.aws.region",
"value" : "{{data.aws.awsRegion}}",
"override" : false,
"ignore_failure" : true,
"ignore_empty_value" : true
}
},
{
"set" : {
"field" : "data.aws.accountId",
"value" : "{{data.aws.aws_account_id}}",
"override" : false,
"ignore_failure" : true,
"ignore_empty_value" : true
}
},
{
"geoip" : {
"field" : "data.srcip",
"target_field" : "GeoLocation",
"properties" : [
"city_name",
"country_name",
"region_name",
"location"
],
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"geoip" : {
"field" : "data.win.eventdata.ipAddress",
"target_field" : "GeoLocation",
"properties" : [
"city_name",
"country_name",
"region_name",
"location"
],
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"geoip" : {
"properties" : [
"city_name",
"country_name",
"region_name",
"location"
],
"ignore_missing" : true,
"ignore_failure" : true,
"field" : "data.aws.sourceIPAddress",
"target_field" : "GeoLocation"
}
},
{
"geoip" : {
"target_field" : "GeoLocation",
"properties" : [
"city_name",
"country_name",
"region_name",
"location"
],
"ignore_missing" : true,
"ignore_failure" : true,
"field" : "data.aws.client_ip"
}
},
{
"geoip" : {
"field" : "data.aws.service.action.networkConnectionAction.remoteIpDetails.ipAddressV4",
"target_field" : "GeoLocation",
"properties" : [
"city_name",
"country_name",
"region_name",
"location"
],
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"geoip" : {
"field" : "data.aws.httpRequest.clientIp",
"target_field" : "GeoLocation",
"properties" : [
"city_name",
"country_name",
"region_name",
"location"
],
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"geoip" : {
"properties" : [
"city_name",
"country_name",
"region_name",
"location"
],
"ignore_missing" : true,
"ignore_failure" : true,
"field" : "data.gcp.jsonPayload.sourceIP",
"target_field" : "GeoLocation"
}
},
{
"geoip" : {
"field" : "data.office365.ClientIP",
"target_field" : "GeoLocation",
"properties" : [
"city_name",
"country_name",
"region_name",
"location"
],
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"date" : {
"field" : "timestamp",
"target_field" : "@timestamp",
"formats" : [
"ISO8601"
],
"ignore_failure" : false
}
},
{
"set" : {
"field" : "_index",
"value" : "wazuh-alerts"
}
},
{
"remove" : {
"ignore_missing" : true,
"ignore_failure" : true,
"field" : "message"
}
},
{
"remove" : {
"field" : "ecs",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "beat",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"ignore_missing" : true,
"ignore_failure" : true,
"field" : "input_type"
}
},
{
"remove" : {
"field" : "tags",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "count",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"ignore_missing" : true,
"ignore_failure" : true,
"field" : "@version"
}
},
{
"remove" : {
"field" : "log",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "offset",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "type",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "host",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "fields",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "event",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"field" : "fileset",
"ignore_missing" : true,
"ignore_failure" : true
}
},
{
"remove" : {
"ignore_missing" : true,
"ignore_failure" : true,
"field" : "service"
}
}
],
"on_failure" : [
{
"drop" : { }
}
]
}
} |
|
With regards to @AlexRuiz7 considerations on Below is an debug log excerpt of 2024-02-14T14:44:18.085-0300 ERROR [elasticsearch] elasticsearch/client.go:224 failed to perform any bulk index operations: Post "https://127.0.0.1:9200/_bulk": dial tcp 127.0.0.1:9200: connect: connection refused
2024-02-14T14:44:18.085-0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2024-02-14T14:44:18.085-0300 INFO [publisher] pipeline/retry.go:223 done
2024-02-14T14:44:19.817-0300 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: Post "https://127.0.0.1:9200/_bulk": dial tcp 127.0.0.1:9200: connect: connection refused
2024-02-14T14:44:19.818-0300 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://127.0.0.1:9200/))
2024-02-14T14:44:19.818-0300 DEBUG [esclientleg] eslegclient/connection.go:290 ES Ping(url=https://127.0.0.1:9200/)
2024-02-14T14:44:19.818-0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2024-02-14T14:44:19.818-0300 INFO [publisher] pipeline/retry.go:223 done
2024-02-14T14:44:19.818-0300 DEBUG [esclientleg] eslegclient/connection.go:294 Ping request failed with: Get "https://127.0.0.1:9200/": dial tcp 127.0.0.1:9200: connect: connection refused
2024-02-14T14:44:20.085-0300 DEBUG [harvester] log/log.go:107 End of file reached: /var/ossec/logs/alerts/alerts.json; Backoff now.
2024-02-14T14:44:22.845-0300 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://127.0.0.1:9200/)): Get "https://127.0.0.1:9200/": dial tcp 127.0.0.1:9200: connect: connection refused
2024-02-14T14:44:22.846-0300 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://127.0.0.1:9200/)) with 1 reconnect attempt(s)
2024-02-14T14:44:22.846-0300 DEBUG [esclientleg] eslegclient/connection.go:290 ES Ping(url=https://127.0.0.1:9200/)
2024-02-14T14:44:22.846-0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2024-02-14T14:44:22.846-0300 INFO [publisher] pipeline/retry.go:223 done
2024-02-14T14:44:22.846-0300 DEBUG [esclientleg] eslegclient/connection.go:294 Ping request failed with: Get "https://127.0.0.1:9200/": dial tcp 127.0.0.1:9200: connect: connection refused
2024-02-14T14:44:24.085-0300 DEBUG [harvester] log/log.go:107 End of file reached: /var/ossec/logs/alerts/alerts.json; Backoff now.
2024-02-14T14:44:27.083-0300 DEBUG [input] input/input.go:139 Run input
2024-02-14T14:44:27.083-0300 DEBUG [input] log/input.go:205 Start next scan
2024-02-14T14:44:27.083-0300 DEBUG [input] log/input.go:439 Check file for harvesting: /var/ossec/logs/alerts/alerts.json
2024-02-14T14:44:27.083-0300 DEBUG [input] log/input.go:530 Update existing file for harvesting: /var/ossec/logs/alerts/alerts.json, offset: 872060
2024-02-14T14:44:27.083-0300 DEBUG [input] log/input.go:582 Harvester for file is still running: /var/ossec/logs/alerts/alerts.json
2024-02-14T14:44:27.083-0300 DEBUG [input] log/input.go:226 input states cleaned up. Before: 1, After: 1, Pending: 0
2024-02-14T14:44:28.816-0300 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://127.0.0.1:9200/)): Get "https://127.0.0.1:9200/": dial tcp 127.0.0.1:9200: connect: connection refused
2024-02-14T14:44:28.816-0300 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://127.0.0.1:9200/)) with 2 reconnect attempt(s)
2024-02-14T14:44:28.816-0300 DEBUG [esclientleg] eslegclient/connection.go:290 ES Ping(url=https://127.0.0.1:9200/)
2024-02-14T14:44:28.817-0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2024-02-14T14:44:28.817-0300 INFO [publisher] pipeline/retry.go:223 done
2024-02-14T14:44:28.817-0300 DEBUG [esclientleg] eslegclient/connection.go:294 Ping request failed with: Get "https://127.0.0.1:9200/": dial tcp 127.0.0.1:9200: connect: connection refusedHowever, when (status=404): {"type":"index_not_found_exception","reason":"no such index [wazuh-alerts] and [action.auto_create_index] ([-wazuh-alerts-*]) doesn't match","index":"wazuh-alerts","index_uuid":"_na_"} |
|
We have discussed both solutions internally with the Wazuh development directors, and we are finally retiring the rollover+alias project from the 4.8.0 release, as both solutions represent a breaking change for backwards compatibility. We'll revisit this for the next major release 5.0.0, probably in the form of a plugin. Nevertheless, we've gathered very valuable information, which will for sure be put in practice in the future. As a result, I'm closing this research issue as completed. |
This was referenced Feb 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
In 4.8.0, we are including automatic policy-managed index rotation to the indexer. This feature is set up during the indexer's initialization (
indexer-init.shand/orindexer-ism-init.shscripts), but sometimes, due to race conditions, it happens that Filebeat starts indexing before everything is properly configured, creating an index that doesn't match with any of our index templates and settings.We need to investigate whether it is possible to prevent Filebeat from indexing if the
wazuh-alertsalias is not created, so it actually indexes events where it should.A good approach is to not grant admin privileges to Filebeat, so it can't create indexes (and all other kind of operations), and grant
writeorindexprivileges. Check https://opensearch.org/docs/latest/security/access-control/default-action-groups/#index-level.This would require modifications in our documentation, but would restrict Filebeat's permissions to exactly what it needs, so we'd have a better control over it.
The text was updated successfully, but these errors were encountered: