Fix Alerts Summary table truncated results #5131

asteriscos · 2023-01-16T14:57:00Z

Description

Team,

this PR fixes the query result of the PDF report Alerts Summary Table to avoid truncating the number of rows. The parsing algorithm wasn't iterating all the bucket combinations, therefore it returned only the first combination of buckets. Now it iterates al possible combinations of the aggregation buckets and returns all possible results.

Note: We also found server-side functions truncate and sort indexer queries. The issue is described in the issue #5134

Issues Resolved

Closes #5130

Evidence

Test

Generate a report of each module with and without a pinned agent.
- The rendered Alerts Summary table should be consistent with the rest of the visualizations and the available alerts in the reported Module Dashboard
Setup the Allow agents restriction filter and verify it applies to the report table.

Check allowed Agents

Module	Allowed agents filtered
Security events	⚫

Security Information Management

Module	With pinned agent	Without agent
Security events	⚫	⚫
Integrity monitoring	⚫	⚫
Office 365	⚫	⚫
Amazon AWS	⚫	⚫
Google Cloud Platform	⚫	⚫
GitHub	⚫	⚫

Auditing and Policy Monitoring

Module	With pinned agent	Without agent
Policy monitoring	⚫	⚫
Security configuration assessment	⚫	⚫
System auditing	⚫	⚫
OpenSCAP	⚫	⚫
CIS-CAT	⚫	⚫

Threat Detection and Response

Module	With pinned agent	Without agent
Vulnerabilities	⚫	⚫
MITRE ATT&CK	⚫	⚫
VirusTotal	⚫	⚫
Osquery	⚫	⚫
Docker listener	⚫	⚫

Regulatory Compliance

Module	With pinned agent	Without agent
PCI DSS	⚫	⚫
NIST 800-53	⚫	⚫
GDPR	⚫	⚫
HIPAA	⚫	⚫
TSC	⚫	⚫

Check List

All tests pass
- yarn test:jest
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

yenienserrano

Security Information Management

Module	With pinned agent	Without agent
Security events	🟢	🟢
Integrity monitoring	🟢	🟢
Office 365	⚫	🟢
Amazon AWS	🟢	🟢
Google Cloud Platform	🟢	🟢
GitHub	🟢	🟢

Auditing and Policy Monitoring

Module	With pinned agent	Without agent
Policy monitoring	🟢	🟢
Security configuration assessment	⚫	⚫
System auditing	🟢	🟢
OpenSCAP	🟢	🟢
CIS-CAT	🟢	🟢

Threat Detection and Response

Module	With pinned agent	Without agent
Vulnerabilities	⚫	⚫
MITRE ATT&CK	⚫	⚫
VirusTotal	⚫	⚫
Osquery	⚫	⚫
Docker listener	⚫	⚫

Regulatory Compliance

Module	With pinned agent	Without agent
PCI DSS	⚫	⚫
NIST 800-53	⚫	⚫
GDPR	⚫	⚫
HIPAA	⚫	⚫
TSC	⚫	⚫

Tostti · 2023-01-17T22:38:39Z

Security Information Management

Module	With pinned agent	Without agent
Security events	🟢	🟢
Integrity monitoring	🟢	🟢
Office 365	⚪	🟢
Amazon AWS	🟢	🟢
Google Cloud Platform	🟢	🟢
GitHub	🟢	🟢

Auditing and Policy Monitoring

Module	With pinned agent	Without agent
Policy monitoring	🟢	🟢
Security configuration assessment	⚪	⚪
System auditing	🟢	🟢
OpenSCAP	🟢	🟢
CIS-CAT	🟢	🟢

Threat Detection and Response

Module	With pinned agent	Without agent
Vulnerabilities	⚪	⚪
MITRE ATT&CK	🟢	🟢
VirusTotal	🟢	🟢
Osquery	🟢	🟢
Docker listener	🟢	🟢

Regulatory Compliance

Module	With pinned agent	Without agent
PCI DSS	🟢	🟢
NIST 800-53	🟢	🟢
GDPR	🟢	🟢
HIPAA	🟢	🟢
TSC	🟢	🟢

Tostti

TEST:✔️
CR:✔️

LGTM

asteriscos · 2023-01-18T11:44:29Z

Important

I found out that the filters applied to the PDF report tables are not including the allowed_agents filtered when an agents RBAC policy is applied to the user.

Allowed Agents applied to the visualizations

Segregation of the Allowed Agents filter in the server-side controller

Final query filters

This mus be fixed in this PR

asteriscos · 2023-01-18T17:59:15Z

Added allowed_agents filter to the following server-side queries

Audit requests:

getTop3AgentsSudoNonSuccessful
getTop3AgentsFailedSyscalls
getTopFailedSyscalls

GDPR requests:

topGDPRRequirements
getRulesByRequirement

Overview requests:

topLevel15

PCI requests:

topPCIRequirements
getRulesByRequirement

Rootcheck requests:

top5RootkitsDetected
agentsWithHiddenPids
agentsWithHiddenPorts

Alerts Summary tables:

alerts summary tables

Syscheck requests:

top3agents
top3Rules
lastTenDeletedFiles
lastTenModifiedFiles

TSC requests:

topTSCRequirements
getRulesByRequirement

Vulnerability requests:

topAgentCount
topCVECount
uniqueSeverityCount
topPackages
topPackagesWithCVE

These requests need to be included in the tests, with the allowed_agents policy and without it.
Each module PDF report data consistency must be checked.

Desvelao · 2023-01-23T11:11:41Z

Tests

Module	With pinned agent	Without agent	Allowed agents with pinned agent	Allowed agent without pinned agent
Security events	🟢	🟢	🟢	🟢
Integrity monitoring	🟢	🟢	🟢	🟢
Office 365	⚪	🟢	⚪	🟢
Amazon AWS	🟢	🟢	🟢	🟢
Google Cloud Platform	🟢	🟢	🟢	🟢
GitHub	🟢	🟢	🟢	🟢
Policy monitoring	🟢	🟢	🟢	🟢
Security configuration assessment	⚪	⚪	⚪	⚪
System auditing	🟢	🟢	🟢	🟢
OpenSCAP	🟢	🟢	🟢	🟢
CIS-CAT	🟢	🟢	🟢	🟢
Vulnerabilities	⚪	⚪	⚪	⚪
MITRE ATT&CK	🟢	🟢	🟢	🟢
VirusTotal	🟢	🟢	🟢	🟢
Osquery	🟢	🟢	🟢	🟢
Docker listener	🟢	🟢	🟢	🟢
PCI DSS	🟢	🟢	🟢	🟢
NIST 800-53	🟢	🟢	🟢	🟢
GDPR	🟢	🟢	🟢	🟢
HIPAA	🟢	🟢	🟢	🟢
TSC	🟢	🟢	🟢	🟢

Desvelao · 2023-01-23T16:39:25Z

I was researching the problem with the tests. There are some tests failing that are related to the checking the md5 of PDF reports.

I think the problem is the note for the allowed agents is added when is not the use case:

@asteriscos, could you review it?

EDIT: I think there is a problem with this conditional: https://github.com/wazuh/wazuh-kibana-app/pull/5131/files#diff-e344f646f1844581c2ba5159d750a8f075c476e56350e3bcc6018f87136dd973R344-R346. It is always truthy and the allowed note is added to the PDF module report.

Tostti · 2023-01-23T19:48:33Z

Tests

Module	With pinned agent	Without agent	Allowed agents with pinned agent	Allowed agent without pinned agent
Security events	🟢	🟢	🟢	🟢
Integrity monitoring	🟢	🟢	🟢	🟢
Office 365	⚪	🟢	⚪	🟢
Amazon AWS	🟢	🟢	🟢	🟢
Google Cloud Platform	🟢	🟢	🟢	🟢
GitHub	🟢	🟢	🟢	🟢
Policy monitoring	🟢	🟢	🟢	🟢
Security configuration assessment	⚪	⚪	⚪	⚪
System auditing	🟢	🟢	🟢	🟢
OpenSCAP	🟢	🟢	🟢	🟢
CIS-CAT	🟢	🟢	🟢	🟢
Vulnerabilities	⚪	⚪	⚪	⚪
MITRE ATT&CK	🟢	🟢	🟢	🟢
VirusTotal	🟢	🟢	🟢	🟢
Osquery	🟢	🟢	🟢	🟢
Docker listener	🟢	🟢	🟢	🟢
PCI DSS	🟢	🟢	🟢	🟢
NIST 800-53	🟢	🟢	🟢	🟢
GDPR	🟢	🟢	🟢	🟢
HIPAA	🟢	🟢	🟢	🟢
TSC	🟢	🔴	🟢	🔴

🔴 TSC

When an agent is pinned, it works as expected and in Alerts summary only shows the alerts for that agent. However, if no agent is pinnned, it only shows alerts for one agent.
In my tests I was able to pin each agent individually and see the corresponding alerts, but without any agent pinned, I always see only the alerts for one of the agents.

Evidence

Agents

Without pinned agents, 9 pages and only showing agent 002 (4.4-2)

With agent 001 pinned, 9 pages and only showing agent 001 (4.4-3)

With agent 002 pinned, 9 pages and only showing agent 002 (4.4-2)

With agent 003 pinned , 9 pages and only showing agent 003 (4.4)

Desvelao · 2023-01-24T07:45:43Z

Tests

Module With pinned agent Without agent Allowed agents with pinned agent Allowed agent without pinned agent
Security events
Integrity monitoring
Office 365
Amazon AWS
Google Cloud Platform
GitHub
Policy monitoring
Security configuration assessment
System auditing
OpenSCAP
CIS-CAT
Vulnerabilities
MITRE ATT&CK
VirusTotal
Osquery
Docker listener
PCI DSS
NIST 800-53
GDPR
HIPAA
TSC

TSC

When an agent is pinned, it works as expected and in Alerts summary only shows the alerts for that agent. However, if no agent is pinnned, it only shows alerts for one agent. In my tests I was able to pin each agent individually and see the corresponding alerts, but without any agent pinned, I always see only the alerts for one of the agents.

Evidence

Agents

Without pinned agents, 9 pages and only showing agent 002 (4.4-2)

With agent 001 pinned, 9 pages and only showing agent 001 (4.4-3)

With agent 002 pinned, 9 pages and only showing agent 002 (4.4-2)

With agent 003 pinned , 9 pages and only showing agent 003 (4.4)

I think that problem is related to the following issue: #5134

github-actions · 2023-01-24T12:53:13Z

Code coverage (Jest)	%	values
Statements	8.76%	( 3237 / 36912 )
Branches	4.5%	( 1291 / 28673 )
Functions	7.63%	( 698 / 9143 )
Lines	8.83%	( 3122 / 35346 )

Desvelao

review:

code 🟢
test 🟢

I checked when the allowed agents should appear and it is working as expected. I verified the next cases:

case	result
With pinned agent	🟢
Without pinned agent	🟢
With pinned agent and allowed agents	🟢
Without pinned agent and allowed agents	🟢

Tostti

Checked when the allowed agents should appear and it is working as expected. Verified the next cases:

case	result
With pinned agent	🟢
Without pinned agent	🟢
With pinned agent and allowed agents	🟢
Without pinned agent and allowed agents	🟢

TEST:✔️
CR:✔️

LGTM

github-actions · 2023-01-25T09:02:22Z

The backport to 4.4-7.16 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-4.4-7.16 4.4-7.16
# Navigate to the new working tree
cd .worktrees/backport-4.4-7.16
# Create a new branch
git switch --create backport-5131-to-4.4-7.16
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c54058037361a724126d6bf8b7a6f6254e0a6670
# Push it to GitHub
git push --set-upstream origin backport-5131-to-4.4-7.16
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-4.4-7.16

Then, create a pull request where the base branch is 4.4-7.16 and the compare/head branch is backport-5131-to-4.4-7.16.

github-actions · 2023-01-25T09:02:23Z

The backport to 4.4-7.10 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-4.4-7.10 4.4-7.10
# Navigate to the new working tree
cd .worktrees/backport-4.4-7.10
# Create a new branch
git switch --create backport-5131-to-4.4-7.10
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c54058037361a724126d6bf8b7a6f6254e0a6670
# Push it to GitHub
git push --set-upstream origin backport-5131-to-4.4-7.10
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-4.4-7.10

Then, create a pull request where the base branch is 4.4-7.10 and the compare/head branch is backport-5131-to-4.4-7.10.

* Fix query result parsing algorithm * Added changelog * Add allowed_agents filter to the report queries * Fix agentsFilter conditional Co-authored-by: Antonio <34042064+Desvelao@users.noreply.github.com> (cherry picked from commit c540580)

Fix Alerts Summary table truncated results (#5131) * Fix query result parsing algorithm * Added changelog * Add allowed_agents filter to the report queries * Fix agentsFilter conditional Co-authored-by: Antonio <34042064+Desvelao@users.noreply.github.com> (cherry picked from commit c540580) Co-authored-by: Federico Rodriguez <federico.rodriguez@wazuh.com>

Fix query result parsing algorithm

13062b9

asteriscos changed the base branch from master to 4.4-2.4-wzd January 16, 2023 14:57

asteriscos self-assigned this Jan 16, 2023

asteriscos linked an issue Jan 16, 2023 that may be closed by this pull request

Alerts Summary table results truncated #5130

Closed

asteriscos marked this pull request as ready for review January 16, 2023 15:01

asteriscos requested a review from a team as a code owner January 16, 2023 15:01

asteriscos added backport 4.4-7.16 labels Jan 16, 2023

Added changelog

6d25dc6

yenienserrano previously approved these changes Jan 17, 2023

View reviewed changes

Tostti previously approved these changes Jan 17, 2023

View reviewed changes

Add allowed_agents filter to the report queries

8d967b7

asteriscos dismissed stale reviews from Tostti and yenienserrano via 8d967b7 January 18, 2023 17:35

asteriscos and others added 2 commits January 24, 2023 12:41

Fix agentsFilter conditional

afc9f91

Merge branch '4.4-2.4-wzd' into fix/alerts-summary-table-truncated-5130

2f036a7

Desvelao approved these changes Jan 24, 2023

View reviewed changes

Tostti approved these changes Jan 24, 2023

View reviewed changes

Desvelao merged commit c540580 into 4.4-2.4-wzd Jan 25, 2023

Desvelao deleted the fix/alerts-summary-table-truncated-5130 branch January 25, 2023 08:49

Desvelao mentioned this pull request Jan 25, 2023

[Backport 4.4-7.10] Fix Alerts Summary table truncated results #5161

Merged

Desvelao mentioned this pull request Jan 25, 2023

[Backport 4.4-7.16] Fix Alerts Summary table truncated results #5162

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Alerts Summary table truncated results #5131

Fix Alerts Summary table truncated results #5131

asteriscos commented Jan 16, 2023 •

edited by Desvelao

yenienserrano left a comment •

edited

Tostti commented Jan 17, 2023

Tostti left a comment

asteriscos commented Jan 18, 2023

asteriscos commented Jan 18, 2023

Desvelao commented Jan 23, 2023 •

edited

Desvelao commented Jan 23, 2023 •

edited

Tostti commented Jan 23, 2023

Evidence

Agents

Without pinned agents, 9 pages and only showing agent 002 (4.4-2)

With agent 001 pinned, 9 pages and only showing agent 001 (4.4-3)

With agent 002 pinned, 9 pages and only showing agent 002 (4.4-2)

With agent 003 pinned , 9 pages and only showing agent 003 (4.4)

Desvelao commented Jan 24, 2023

TSC

Evidence

Agents

Without pinned agents, 9 pages and only showing agent 002 (4.4-2)

With agent 001 pinned, 9 pages and only showing agent 001 (4.4-3)

With agent 002 pinned, 9 pages and only showing agent 002 (4.4-2)

With agent 003 pinned , 9 pages and only showing agent 003 (4.4)

github-actions bot commented Jan 24, 2023

Desvelao left a comment

Tostti left a comment

github-actions bot commented Jan 25, 2023

github-actions bot commented Jan 25, 2023

Fix Alerts Summary table truncated results #5131

Fix Alerts Summary table truncated results #5131

Conversation

asteriscos commented Jan 16, 2023 • edited by Desvelao

Description

Issues Resolved

Evidence

Test

Check allowed Agents

Security Information Management

Auditing and Policy Monitoring

Threat Detection and Response

Regulatory Compliance

Check List

yenienserrano left a comment • edited

Choose a reason for hiding this comment

Security Information Management

Auditing and Policy Monitoring

Threat Detection and Response

Regulatory Compliance

Tostti commented Jan 17, 2023

Security Information Management

Auditing and Policy Monitoring

Threat Detection and Response

Regulatory Compliance

Tostti left a comment

Choose a reason for hiding this comment

asteriscos commented Jan 18, 2023

Important

Allowed Agents applied to the visualizations

Segregation of the Allowed Agents filter in the server-side controller

Final query filters

asteriscos commented Jan 18, 2023

Added allowed_agents filter to the following server-side queries

Audit requests:

GDPR requests:

Overview requests:

PCI requests:

Rootcheck requests:

Alerts Summary tables:

Syscheck requests:

TSC requests:

Vulnerability requests:

Desvelao commented Jan 23, 2023 • edited

Desvelao commented Jan 23, 2023 • edited

Tostti commented Jan 23, 2023

🔴 TSC

Evidence

Agents

Without pinned agents, 9 pages and only showing agent 002 (4.4-2)

With agent 001 pinned, 9 pages and only showing agent 001 (4.4-3)

With agent 002 pinned, 9 pages and only showing agent 002 (4.4-2)

With agent 003 pinned , 9 pages and only showing agent 003 (4.4)

Desvelao commented Jan 24, 2023

TSC

Evidence

Agents

Without pinned agents, 9 pages and only showing agent 002 (4.4-2)

With agent 001 pinned, 9 pages and only showing agent 001 (4.4-3)

With agent 002 pinned, 9 pages and only showing agent 002 (4.4-2)

With agent 003 pinned , 9 pages and only showing agent 003 (4.4)

github-actions bot commented Jan 24, 2023

Desvelao left a comment

Choose a reason for hiding this comment

Tostti left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 25, 2023

github-actions bot commented Jan 25, 2023

asteriscos commented Jan 16, 2023 •

edited by Desvelao

yenienserrano left a comment •

edited

Desvelao commented Jan 23, 2023 •

edited

Desvelao commented Jan 23, 2023 •

edited