Repository navigation
ILM Policy best way forward #16309
Replies: 1 comment 7 replies
|
At the moment it looks like you would hit the cluster shard limit of 1000 shards per elasticsearch node before you reached your storage limit. You might consider adding a dedicated searchnode, that new searchnode would give you an additional 1000 shards (2000 total for cluster). On top of adding additional storage space to the cluster. Regardless if you can add a new searchnode or not something I would suggest is to review your shard sizes. If you have many shards that are created and they fall well below 50 G you can adjust the ILM policies for the given data stream(s), increasing the You can find a bit more information about shards here https://docs.securityonion.net/en/3/main/elasticsearch/?h=shard#shards |
|
Quick update on above - I enabled Show advanced settings and now I can see the max_age setting in the so-INDEX-NAME but what value do I put there? and do I need to change the Cisco Index or the others? Also I cannot find the Cisco index in the list of so-INDEX-NAME |
You should add "cisco" to the managed_integrations configuration (or any other integration name) and then you can force it to apply by running For the indices you have that are consistently hitting 50gb per shard those would not be ones you want to lower the max_age on. You want to increase the max_age setting on the indices you have that are consistently well below 50gb. For example if you had fortinet data coming in and those indices are getting to say only 15gb, that would be a data set you could try increasing the max_age on. |
|
Hi thanks for your response. I am still trying to understand this and make sure I get it correct. Along with Cisco FTD, I have an integration for Cisco ISE too. In the managed_integrations setting do I add to it "cisco" by itself or add both "cisco" and "cisco_ise" If we take for example the below entries of Zeek for the command sudo so-elasticsearch-indices-list , they are well below 50gb. health status index pri rep docs.count docs.deleted store.size pri.store.size dataset.size Then when I go to elasticsearch > index_settings > so-zeek > policy > phases > hot > actions > rollover > max_age [adv] it has a value of 30d, what is the max value this setting can have so that they consistently get up to 50gb? Also, with the cluster shards reaching 100% imminently, is there a console interface where its percentage can be lowered in another way? What will happen when the cluster shards reach 100%? |
You should be able to add "cisco" and it will cover both cisco_ise and cisco_ftd integrations. Adding both to the SOC configuration screen so you can see their individual configuration sections. With those Zeek indices, it looks like you aren't ingesting much live network traffic at all. So the first question you might ask is of the data you are collecting via Zeek, how much of it do you need? At the moment you have about 9 months worth of low volume Zeek data. If you don't need to retain that long of Zeek data, you can update the delete value for Zeek to 60d. That would effectively give you a totale of 90d retention for Zeek. Zeek is rolling over every 30d + 60d of delete phase = 90d You can repeat that process for any other data sources you have and determine what retention you need for each, that helps you free up space + shards for data you care about. Then the original suggestion of increasing the max_age setting still applies. For example if Zeek is only taking up 1-2gb per month, you could increase that max_age to say 180d and instead of Zeek creating 1-2gb shards it would wait longer and create closer to 6-12gb shards. So instead of 180d of Zeek data consuming 12 shards it would use 2 shards. |
|
Thank you I will try that as well but, another question I had was - with the cluster shards reaching 100% imminently, is there a console interface where its percentage can be lowered in another way? What will happen when the cluster shards reach 100%? |
Uh oh!
There was an error while loading. Please reload this page.
Version
3.3.0
Installation Method
Security Onion ISO image
Description
configuration
Installation Type
Standalone
Location
on-prem with Internet access
Hardware Specs
Exceeds minimum requirements
CPU
32
RAM
1024
Storage for /
468 GB
Storage for /nsm
3838 GB
Network Traffic Collection
span port
Network Traffic Speeds
1Gbps to 10Gbps
Status
Yes, all services on all nodes are running OK
Salt Status
No, there are no failures
Logs
No, there are no additional clues
Detail
I run sudo so-elasticsearch-retention-estimate every other day and notice the information there to check storage limits. However, I need some direction on how I can best make use of the available space to keep logs as long as possible. I understand there are different types logs/sources SO can ingest from but I need a plan to know which logs would be more relevant than others.
At the moment I have installed custom Integrations for Cisco ISE, FTD, and Custom UDP for our Proxy server and SO Agent installed on a few Windows servers. I need these specific logs kept more than other. The Cisco FTD logs are biggest consumers of space, which I think can also be looked at and changed if someone can guide me.
Regarding ILM, all settings are set to default at the moment.
Any help would be appreciated.
Guidelines
All reactions