Integration tests: Add new eps limit configuration

[damarisg]

Target version	Related issue	Related PR
4.4	12512 13572	wazuh/wazuh#13608

Description

In order to validate the changes of the branch https://github.com/wazuh/wazuh/tree/dev-cloud-limits, some tests are required.

As part of wazuh/wazuh#12512, a new mechanism has been created to limit the amount of EPS than a manager can process.

This mechanism is implemented within the wazuh-analysisd daemon and works by using a circular buffer that tracks the total number of events over a defined period of time.

Whenever the circular buffer fills up, the events are held within the related queues until some space is freed. This works like a moving average, this is to support event spikes.

In this first iteration, main configuration will be located in global section from ossec.conf file. Link to documentation.

Configuration

ossec.conf

<global>
  <limits>
    <eps>
      <maximum>500</maximum>
      <timeframe>30</timeframe>
    </eps>
  </limits>
</global>

Events per second (EPS) limitation is disabled by default.

Logs

EPS functionality disabled:

2022/07/13 17:31:46 wazuh-analysisd: INFO: EPS limit disabled

EPS functionality enabled:

2022/07/13 17:31:46 wazuh-analysisd: INFO: EPS limit enabled, EPS: '500', timeframe: '30'

Feature validation

State files could be useful to check if EPS limits are working properly.

cat /var/ossec/var/run/wazuh-analysisd.state 
# State file for wazuh-analysisd

# Total events decoded
total_events_decoded='0'

# Syscheck events decoded
syscheck_events_decoded='0'
syscheck_edps='0'

# Syscollector events decoded
syscollector_events_decoded='0'
syscollector_edps='0'

# Rootcheck events decoded
rootcheck_events_decoded='0'
rootcheck_edps='0'

# Security configuration assessment events decoded
sca_events_decoded='0'
sca_edps='0'

# Hostinfo events decoded
hostinfo_events_decoded='0'
hostinfo_edps='0'

# Winevt events decoded
winevt_events_decoded='0'
winevt_edps='0'

# Database synchronization messages dispatched
dbsync_messages_dispatched='0'
dbsync_mdps='0'

# Other events decoded
other_events_decoded='0'
other_events_edps='0'

# Events processed (Rule matching)
events_processed='0'
events_edps='0'

# Events received
events_received='0'

# Events dropped
events_dropped='0'

# Alerts written to disk
alerts_written='0'

# Firewall alerts written to disk
firewall_written='0'

# FTS alerts written to disk
fts_written='0'

# Syscheck queue
syscheck_queue_usage='0.00'

# Syscheck queue size
syscheck_queue_size='16384'

# Syscollector queue
syscollector_queue_usage='0.00'

# Syscollector queue size
syscollector_queue_size='16384'

# Rootcheck queue
rootcheck_queue_usage='0.00'

# Rootcheck queue size
rootcheck_queue_size='16384'

# Security configuration assessment queue
sca_queue_usage='0.00'

# Security configuration assessment queue size
sca_queue_size='16384'

# Hostinfo queue
hostinfo_queue_usage='0.00'

# Hostinfo queue size
hostinfo_queue_size='16384'

# Winevt queue
winevt_queue_usage='0.00'

# Winevt queue size
winevt_queue_size='16384'

# Database synchronization message queue
dbsync_queue_usage='0.00'

# Database synchronization message queue size
dbsync_queue_size='16384'

# Upgrade module message queue
upgrade_queue_usage='0.00'

# Upgrade module message queue size
upgrade_queue_size='16384'

# Event queue
event_queue_usage='0.00'

# Event queue size
event_queue_size='16384'

# Rule matching queue
rule_matching_queue_usage='0.00'

# Rule matching queue size
rule_matching_queue_size='16384'

# Alerts log queue
alerts_queue_usage='0.00'

# Alerts log queue size
alerts_queue_size='16384'

# Firewall log queue
firewall_queue_usage='0.00'

# Firewall log queue size
firewall_queue_size='16384'

# Statistical log queue
statistical_queue_usage='0.00'

# Statistical log queue size
statistical_queue_size='16384'

# Archives log queue
archives_queue_usage='0.00'

# Archives log queue size
archives_queue_size='16384'

IMPORTANT: After merging these changes, the new API endpoints should be used to check manager stats.

Test cases

For the following tests, always use the same type of events (for example, dbsync) so that they are directed to the same queue. Also, keep in mind that this mechanism works like a moving average (freeing up some space every second), so the results may not always be exactly the same because they depend on the second the events reach the manager, so it should be better to compare between ranges of expected values instead of exact values.

• Check that wazuh-analysisd stops processing events when the limit is reached. For example, if the eps is 50 and the timeframe is 30, it should stop processing events when reaching 1500 in the last 30 seconds (timeframe) until the moving average frees up some space.

• Check that wazuh-analysisd starts queuing events when the limit is reached and the corresponding queue is not full. For example, if the eps is 50 and the timeframe is 30, it should start queuing events after reaching 1500 in the last 30 seconds (timeframe) and the corresponding queue is not full until the moving average frees up some space.

• Check that wazuh-analysisd starts dropping events when the limit is reached and the corresponding queue is full. For example, if the eps is 50 and the time frame is 30, it should start discarding events when reaching 1500 in the last 30 seconds (timeframe) and the corresponding queue is full until the moving average frees up some space.

• Check that wazuh-analysisd processes queued events first instead of new events when the moving average frees up some space. For example, if the eps is 50 and the time frame is 30, it should start processing queued events after freeing up a slot of the 1500 events from the last 30 seconds (timeframe) instead of new events (which are sent to the end of the queue).

• Test the new configuration block.

[fedepacher]

Update 2022/08/02

To test the EPS limits deployment, it has been used the simulate_agent.py tool of the Wazuh-QA repository. This tool simulate an agent that it is registered in the manager and it is configured to send a configurable amount of logcollector logs to the specified manager.
At this point the following tests have been implemented:

EPS enable test

This test checks that the following configuration works:

<global>
  <limits>
    <eps>
      <maximum>500</maximum>
      <timeframe>30</timeframe>
    </eps>
  </limits>
</global>

To do this, it checks the message .*INFO: EPS limit enabled, EPS: '{maximun}', timeframe: '{timeframe}' is got in the ossec.log file.

EPS disable test

This test checks that if it is sent maximun value 0 or empty, the message INFO: EPS limit disabled is got in the ossec.log file.

EPS invalid value test

This test checks:

• If maximun value is greater than 100000 the manager fails at the start process.
• If timeframe value is greater than 3600 the manager fails at the start process.
• If timeframe value is 0 the manager fails at the start process.

wazuh-analysisd stops processing events when the limit is reached

This test checks that the events_processed value in the /var/ossec/var/run/wazuh-analysisd.state file must be lower or equal than maximun * timeframe.

wazuh-analysisd starts queuing events when the limit is reached and the corresponding queue is not full

This test checks:

• The events_processed value in the /var/ossec/var/run/wazuh-analysisd.state file must be lower than maximun * timeframe.
• The events_received value must be greater than events_processed.
• The events_dropped value equal to 0.
• The event_queue_usage lower than 1.0.

wazuh-analysisd starts dropping events when the limit is reached and the corresponding queue is full

This test checks:

• The events_processed value in the /var/ossec/var/run/wazuh-analysisd.state file must be lower than maximun * timeframe.
• The events_received value must be graeter that events_processed.
• The events_dropped value greater than 0.
• The event_queue_usage equal to 1.0.

wazuh-analysisd processes queued events first instead of new events when the moving average frees up some space

This test gets a message in the middle of the first, second and third timeframe and get the timestamp of each message to be compared. The timestamp of the first message must be the lowest and the timestamp of the third message must be the greatest.

[fedepacher]

Update 2022/08/17

I order to send message to the manager, it has upgrade the tool simulate_agent.py. This upgrade implies send an amount of message to the manager to check the status file.

Message format

The format of the message send it to the manager is the following:

Mar 24 10:12:36 centos8 sshd[12249]: Invalid user random_user from 172.17.1.1 port 56550:Message number: 999

This message is formated to have a size of 1Kb.

And generate an alert as follow:

** Alert 1660734144.1424920: - syslog,sshd,authentication_failed,gdpr_IV_35.7.d,gdpr_IV_32.2,gpg13_7.1,hipaa_164.312.b,invalid_login,nist_800_53_AU.14,nist_800_53_AC.7,nist_800_53_AU.6,pci_dss_10.2.4,pci_dss_10.2.5,pci_dss_10.6.1,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,
2022 Aug 17 11:02:24 (1-BfU1Ob8tW0QRSkip-debian8) any->syslog
Rule: 5710 (level 5) -> 'sshd: Attempt to login using a non-existent user'
Src IP: 172.17.1.1
Src Port: 56550:Message
Mar 24 10:12:36 centos8 sshd[12249]: Invalid user random_user from 172.17.1.1 port 56550:Message number: 999AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA