Creating a windows detector failed and generates a lot of shards reaching the cluster limit everyday

[mareban]

Hello ,

We've tried to create a windows detector using all the rules ! It failed and ask to review the configuration !
So no detector is created, but we have a lot of .opensearch-sap-windows-detectors-queries* created each day :-(, and we don't know how to clean this and create the detector sucessfully !

How can one reproduce the bug?
create the windows detector with all rules selected on winlogbeat-* indices !

What is the expected behavior?
Create the detector and notify us on security events

What is your host/environment?
the 2.8.0 Opensearch cluster is running on Ubuntu 22.04.2

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

[mareban]

For info, monitoring jobs, we have this kind of jobs always running :

GET _cat/pending_tasks

3525231 120ms HIGH put-mapping [.opensearch-sap-windows-detectors-queries-000029/1OPPMM8SQoSHDX5WpdSHUw]
3525380 1s HIGH put-mapping [.opensearch-sap-windows-detectors-queries-000035/ITm-ofAJT1G5UXL8uBQ71w]
3525404 100ms NORMAL rollover_index source [.opensearch-sap-windows-detectors-queries-000037] to target [.opensearch-sap-windows-detectors-queries-000038]
3525666 2.2s URGENT shard-started StartedShardEntry{shardId [[.opensearch-sap-windows-detectors-queries-000055][0]], allocationId [NWs7-UrZQG2wv_ktYwvNwA], primary term [1], message [after peer recovery]}
3525667 95ms HIGH put-mapping [.opensearch-sap-windows-detectors-queries-000055/ny62morgTo-2_-r3vBAFcA]

shards example :
.opensearch-sap-windows-detectors-queries-000037 0 r STARTED

Forgot to mention that before trying to create the windows log detector, we've created an _index_template for winlogbeat but we kept the _template as we did for the packetbeat and dns logs that seems to work (notification OK) and we don't have this issue of shards creation .

Thanks for your help

[praveensameneni]

@mareban , Thank you for trying out Security Analytics. Can you share couple of sample log files (with the necessary redacted data).

[sbcd90]

hi @mareban , do you have any ism policy defined for the index-pattern winlogbeat-*? or how many concrete indices are there behind the pattern winlogbeat-*?

[sbcd90]

hi @mareban , i followed the following steps while running detectors on beats data.
i built a custom version of beats. https://github.com/sbcd90/beats/tree/main

i created the index template first by exporting it.

.\winlogbeat.exe export template

i faced 2 errors, field keys "data_stream" & "priority" not available.

i then just ran winlogbeat with config https://github.com/sbcd90/beats/blob/main/winlogbeat/winlogbeat.yml

i created a windows detector with 1580 rules & provided winlogbeat* index-pattern as input.
The detector runs fine generating findings.

[sbcd90]

hi @mareban , i can finally reproduce this issue & understand the problem. will explore options to fix it.
Thanks for creating this issue.
cc: @getsaurabh02

[mareban]

hi @mareban , do you have any ism policy defined for the index-pattern winlogbeat-*? or how many concrete indices are there behind the pattern winlogbeat-*?

Hello, Thx for your reply and sorry for the delay :

There are almost 500 indices (one by day)

[mareban]

hi @mareban , i followed the following steps while running detectors on beats data. i built a custom version of beats. https://github.com/sbcd90/beats/tree/main

i created the index template first by exporting it.

.\winlogbeat.exe export template

i faced 2 errors, field keys "data_stream" & "priority" not available.

i then just ran winlogbeat with config https://github.com/sbcd90/beats/blob/main/winlogbeat/winlogbeat.yml

i created a windows detector with 1580 rules & provided winlogbeat* index-pattern as input. The detector runs fine generating findings.

Hello and Thx again

Here is what we have when trying to create a windows detector :

For the mapping we have 2 field to map with our winlogbeat template ?

So , my understanding is that's a bug, and the fix will come soon hopefully :-) ! We are still in 2.8 , and it seems that upgrading to 2.9 will not fix our issue , correct ?

Thank you very much for your help

[amsiglan]

Fixed for 2.10 and beyond. We will backport this as part of patch release

[mareban]

Thanks for the fix :-) !

We've just upgrade to 2.10 and still cannot create a windows detector using all sigma rules for winlogbeat (+500 indicies) !

We have now +60 fields to map, in 2.8 only 2 , and if we don't map anything the detector cannot be created !!

_security_analytics/detectors, params: {}
org.opensearch.securityanalytics.util.SecurityAnalyticsException: Invalid [path] value [message_winlogbeat-__1x8H4YoBiU4cj2_sBud9] for field alias [windows.message_winlogbeat-__1x8H4YoBiU4cj2_sBud9]: an alias must refer to an existing field in the mappings.

and :

[2023-09-29T15:01:04,090][INFO ][o.o.a.t.TransportIndexMonitorAction] [opnscluster-n1] Central Percolation index .opensearch-sap-windows-detectors-queries created
[2023-09-29T15:01:04,154][INFO ][o.o.p.PluginsService ] [opnscluster-n1] PluginService:onIndexModule index:[.opensearch-sap-windows-detectors-queries-000001/XONDNquhTAyTDL_fs2p0YQ]
[2023-09-29T15:01:04,175][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opnscluster-n1] Detected cluster change event for destination migration
[2023-09-29T15:01:04,336][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opnscluster-n1] Detected cluster change event for destination migration
[2023-09-29T15:01:15,543][INFO ][o.o.j.s.JobSweeper ] [opnscluster-n1] Running full sweep
[2023-09-29T15:01:15,601][INFO ][o.o.j.s.JobSweeper ] [opnscluster-n1] Error while sweeping shard [.opendistro-reports-definitions][0], error message: all shards failed
[2023-09-29T15:01:37,555][INFO ][o.o.c.s.IndexScopedSettings] [opnscluster-n1] [.opensearch-sap-windows-detectors-queries-000001] updating [index.mapping.total_fields.limit] from [1000] to [5000008]
[2023-09-29T15:01:37,570][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opnscluster-n1] Detected cluster change event for destination migration
[2023-09-29T15:01:54,941][ERROR][o.o.a.u.AlertingException] [opnscluster-n1] Alerting error: MapperParsingException[Invalid [path] value [message_winlogbeat-__1x8H4YoBiU4cj2_sBud9] for field alias [windows.message_winlogbeat-__1x8H4YoBiU4cj2_sBud9]: an alias must refer to an existing field in the mappings.]
[2023-09-29T15:01:54,942][ERROR][o.o.a.t.TransportIndexMonitorAction] [opnscluster-n1] failed to index doc level queries monitor 1x8H4YoBiU4cj2_sBud9. deleting monitor
org.opensearch.alerting.util.AlertingException: Invalid [path] value [message_winlogbeat-__1x8H4YoBiU4cj2_sBud9] for field alias [windows.message_winlogbeat-__1x8H4YoBiU4cj2_sBud9]: an alias must refer to an e

Do we need to do all the mapping ? Is it normal to extend the total field limit to this size ?

Thanks for your help.

[sbcd90]

hi @mareban , the issue actually is,

Invalid [path] value [message_winlogbeat-__1x8H4YoBiU4cj2_sBud9] for field alias [windows.message_winlogbeat-__1x8H4YoBiU4cj2_sBud9]

your index do not have a field called message but somehow in field mappings the field windows.message is mapped to message.
it is completely optional to specify any mappings.
also, yes, the total field limit can exceed to this size but for winlogbeat it should not go this much.
i would recommend delete your existing detector, clean up all .opensearch-sap* & .opensearch-alerting* & .opendistro-alerting* & then deploy the detector.

[mareban]

Thanks a lot for your help

So we've tried to delete the detectors and do the cleanup and now, if we try to create a DNS detector we've got an error : detection creation failed, double check the mapping , but the DNS detector is created !

So we've tried to create a windows detector, but it's still not work for US :-( !

We don't know what we can do now, we've upgraded hoping it will fix the problem, no more lot of sap queries indicies Thx, but still cannot create a windows detector on our cluster !

Any other suggestions are welcome ?

Thanks for your help.

[2023-10-01T15:11:34,224][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opnscluster-n1] Detected cluster change event for destination migration
[2023-10-01T15:11:34,322][INFO ][o.o.p.PluginsService ] [opnscluster-n1] PluginService:onIndexModule index:[.opensearch-sap-windows-detectors-queries-000001/zwRkaIhLTwOwGtNBKB7lvA]
[2023-10-01T15:11:34,343][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opnscluster-n1] Detected cluster change event for destination migration
[2023-10-01T15:11:34,481][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opnscluster-n1] Detected cluster change event for destination migration
[2023-10-01T15:11:45,628][INFO ][o.o.c.s.IndexScopedSettings] [opnscluster-n1] [.opensearch-sap-windows-detectors-queries-000001] updating [index.mapping.total_fields.limit] from [1000] to [5020008]
[2023-10-01T15:11:45,635][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opnscluster-n1] Detected cluster change event for destination migration
[2023-10-01T15:12:25,424][INFO ][o.o.a.a.AlertIndices ] [opnscluster-n1] Index mapping of .opensearch-sap-dns-alerts is updated
[2023-10-01T15:12:25,443][INFO ][o.o.a.a.AlertIndices ] [opnscluster-n1] Index mapping of .opensearch-sap-dns-alerts-history-2023.10.01-1 is updated
[2023-10-01T15:12:25,454][INFO ][o.o.a.a.AlertIndices ] [opnscluster-n1] Index mapping of .opensearch-sap-dns-findings-2023.10.01-1 is updated

[2023-10-01T15:34:49,147][INFO ][o.o.c.s.IndexScopedSettings] [opnscluster-n1] [.opensearch-sap-windows-detectors-queries-000001] updating [index.mapping.total_fields.limit] from [1000] to [5020008]
[2023-10-01T15:34:49,172][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [opnscluster-n1] Detected cluster change event for destination migration
[2023-10-01T15:36:16,167][INFO ][o.o.j.s.JobSweeper ] [opnscluster-n1] Running full sweep

[sbcd90]

hi @mareban , can you please specify the error you get while creating windows detector?
the above logs are only server logs.
do you get this error? detection creation failed, double check the mapping if this is the error, plz check your alias field mappings in winlgbeat-* indices. you may need to clean them up too. i would recommend start with a new index pattern say winlogbeat-new-* & create the windows detector.
Also, plz check these new mappings for 2.10. https://github.com/opensearch-project/security-analytics/blob/main/src/main/resources/OSMapping/windows_logtype.json

[amsiglan]

Closing the issue for now as it seems to be fixed. @mareban Feel free to reopen it if you have more concerns.