Fix 'tests_scan_results' tests in 4.2 #1567

damarisg · 2021-07-06T13:33:19Z

Issue information

Related issue
Closes #1531

When it comes to starting with the test fix, there is some information that can help you:

Module: Vulnerability Detector
Support Team:  "binary beasts"
Target: Manager   
OS: Linux

I add some information obtained while researched the logs that failed.

This issue contains differents tests that are related to the same error.

In this case, there are 4 cases because the method affected is the same and the error too. We believe that when solving one of the cases, the same fix could be applied to the other cases.

Case 1:

Type	Description
Test Path	test_scan_results/test_macos_inventory
Consistent	yes
Test Execution	3/3 executions failed
Cases	macos_scan_configuration-MAC0
	macos_scan_configuration-MAC1
Cases Fails	2
Summary	On `test_macos_vulnerabilities_report` when `check_vulnerability_scan_event` is called. it requires research to know if there is a problem with parser error or if we don't detect log.
Error message	"Could not find the report which says that the package {package} is vulnerable with {cve}"

Case 2:

Type	Description
Test Path	test_scan_results/test_scan_different_cves
Consistent	yes
Test Execution	3/3 executions failed
Cases Fails	9
Summary	On `test_vulnerabilities_report` when `check_vulnerability_scan_event` is called. it requires research to know if there is a problem with parser error or if we don't detect log.
Error message	"Could not find the report which says that the package {package} is vulnerable with {cve}"

Case 3:

Type	Description
Test Path	test_scan_results/test_scan_nvd_feed
Consistent	yes
Test Execution	3/3 executions failed
Cases Fails	13
Summary	On `test_vulnerabilities_report` when `check_vulnerability_scan_event` is called. it requires research to know if there is a problem with parser error or if we don't detect log.
Error message	"Could not find the report which says that the package {package} is vulnerable with {cve}"

Case 4:

Type	Description
Test Path	test_scan_results/test_scan_providers_and_nvd_feed
Consistent	yes
Test Execution	3/3 executions failed
Cases Fails	9
Summary	On `test_vulnerabilities_report` when `check_vulnerability_scan_event` is called. it requires research to know if there is a problem with parser error or if we don't detect log.
Error message	"Could not find the report which says that the package {package} is vulnerable with {cve}"

In order to finish this issue the following tasks should be fulfilled:

Research of fails.
Apply Fix
Full Green/ Full Yellow in test_scan_results/test_macos_inventory for 3 times.
Full Green/ Full Yellow in test_scan_results/test_scan_different_cves for 3 times.
Full Green/ Full Yellow in test_scan_results/test_scan_nvd_feed for 3 times.
Full Green/ Full Yellow in test_scan_results/test_scan_providers_and_nvd_feed for 3 times.
Documentation of any important change done for these tests or the used tools.

The text was updated successfully, but these errors were encountered:

pereyra-m · 2021-07-14T22:01:38Z

These are the results for the different cases, all of them with sca, syscollector and rootcheck disabled

Case 1

Results for test_scan_results/test_macos_inventory.py

Test Executions	Date	By	Status
test_macos_inventory_modules_off_1.log	2021-07-14	Matias	🟢
test_macos_inventory_modules_off_2.log	2021-07-14	Matias	🟢
test_macos_inventory_modules_off_3.log	2021-07-14	Matias	🟢

Case 2

Results for test_scan_results/test_scan_different_cves.py

Test Executions	Date	By	Status
test_scan_different_cves_modules_off_1.log	2021-07-14	Matias	🟢
test_scan_different_cves_modules_off_2.log	2021-07-14	Matias	🟢
test_scan_different_cves_modules_off_3.log	2021-07-14	Matias	🟢

Case 3

Results for test_scan_results/test_scan_nvd_feed.py

Test Executions	Date	By	Status
test_scan_nvd_feed_modules_off_1.log	2021-07-14	Matias	🟢
test_scan_nvd_feed_modules_off_2.log	2021-07-14	Matias	🟢
test_scan_nvd_feed_modules_off_3.log	2021-07-14	Matias	🟢

Case 4

Results for test_scan_results/test_scan_providers_and_nvd_feed.py

Test Executions	Date	By	Status
test_scan_providers_and_nvd_feed_modules_off_1.log	2021-07-14	Matias	🟢
test_scan_providers_and_nvd_feed_modules_off_2.log	2021-07-14	Matias	🟢
test_scan_providers_and_nvd_feed_modules_off_3.log	2021-07-14	Matias	🟢

damarisg · 2021-07-15T11:43:16Z

Results when disabling modules: sca, syscollector and rootcheck.

Case 1

Results for test_scan_results/test_macos_inventory.py

Test Executions	Date	By	Status
Resul1.log	2021-07-15	Seyla	🟡
Resul2.log	2021-07-15	Seyla	🟢
Resul3.log	2021-07-15	Seyla	🟢

Case 2

Results for test_scan_results/test_scan_different_cves.py

Test Executions	Date	By	Status
Resul1.log	2021-07-15	Seyla	🟢
Resul2.log	2021-07-15	Seyla	🟢
Resul3.log	2021-07-15	Seyla	🟢

Case 3

Results for test_scan_results/test_scan_nvd_feed.py

Test Executions	Date	By	Status
ScanNVD.log	2021-07-15	Seyla	🟢
ScanNVD2.log	2021-07-15	Seyla	🟢
ScanNVD3.log	2021-07-15	Seyla	🟢

Case 4

Results for test_scan_results/test_scan_providers_and_nvd_feed.py

Test Executions	Date	By	Status
ScanProvNVD1.log	2021-07-15	Seyla	🟢
ScanProvNVD2.log	2021-07-15	Seyla	🟢
ScanProvNVD3.log	2021-07-15	Seyla	🟢

mdengra · 2021-07-15T12:40:05Z

2021-07-15

Test results with the modules sca, syscollector and rootcheck modules disabled in the ossec.conf:

Case 1

Test Executions	Date	By	Status
test_macos_inventory_local_dm_r1.log	2021-07-15	Miguel	🟡
test_macos_inventory_local_dm_r2.log	2021-07-15	Miguel	🟡
test_macos_inventory_local_dm_r3.log	2021-07-15	Miguel	🟡

Case 2

Test Executions	Date	By	Status
test_scan_different_cves_local_dm_r1.log	2021-07-15	Miguel	🟡
test_scan_different_cves_local_dm_r2.log	2021-07-15	Miguel	🟡
test_scan_different_cves_local_dm_r3.log	2021-07-15	Miguel	🟡

Case 3

Test Executions	Date	By	Status
test_scan_nvd_feed_local_dm_r1.log	2021-07-15	Miguel	🟡
test_scan_nvd_feed_local_dm_r2.log	2021-07-15	Miguel	🟡
test_scan_nvd_feed_local_dm_r3.log	2021-07-15	Miguel	🟡

Case 4

Test Executions	Date	By	Status
test_scan_providers_and_nvd_feed_local_dm_r1.log	2021-07-15	Miguel	🟡
test_scan_providers_and_nvd_feed_local_dm_r2.log	2021-07-15	Miguel	🟡
test_scan_providers_and_nvd_feed_local_dm_r3.log	2021-07-15	Miguel	🟡

damarisg · 2021-07-21T18:51:25Z

This issue is closed by #1605

Test Results from `1531-full-yellow-vuln-det`

Test Executions	Tests	Date	By	Status
YellowResultsScanMacosInventory.log	test_scan_results/test_macos_inventory	2021-07-21	Seyla	🟢
YellowResultsScanMacosInventory2.log	test_scan_results/test_macos_inventory	2021-07-21	Seyla	🟡
YellowResultsScanMacosInventory3.log	test_scan_results/test_macos_inventory	2021-07-21	Seyla	🟡
YellowResultsScanDifferentCve.log	test_scan_results/test_scan_different_cves	2021-07-21	Seyla	🟡
YellowResultsScanDifferentCve2.log	test_scan_results/test_scan_different_cves	2021-07-21	Seyla	🟡
YellowResultsScanDifferentCve3.log	test_scan_results/test_scan_different_cves	2021-07-21	Seyla	🟡
YellowResultsScanNVDFeed2.log	test_scan_results/test_scan_nvd_feed	2021-07-21	Seyla	🟡
YellowResultsScanNVDFeed.log	test_scan_results/test_scan_nvd_feed	2021-07-21	Seyla	🟡
YellowResultsScanNVDFeed4.log	test_scan_results/test_scan_nvd_feed	2021-07-21	Seyla	🟢
YellowResultsScanProvidersAndNVDFeed.log	test_scan_results/test_scan_providers_and_nvd_feed	2021-07-21	Seyla	🟡
YellowResultsScanProvidersAndNVDFeed2.log	test_scan_results/test_scan_providers_and_nvd_feed	2021-07-21	Seyla	🟡
YellowResultsScanProvidersAndNVDFeed3.log	test_scan_results/test_scan_providers_and_nvd_feed	2021-07-21	Seyla	🟡

Reference	Status
🟡	Pass with Warnings
🟢	Pass without Warning

Note

This issue fails sometimes by the bug #1602

mdengra · 2021-07-22T07:57:19Z

2021-07-22

Used Wazuh-QA branch: 1531-full-yellow-vuln-det
Test results with the default settings in the ossec.conf:

Case 1

Test Executions	Date	By	Status
test_macos_inventory_local_r1.log	2021-07-22	Miguel	🟡
test_macos_inventory_local_r2.log	2021-07-22	Miguel	🟡
test_macos_inventory_local_r3.log	2021-07-22	Miguel	🟡

Case 2

Test Executions	Date	By	Status
test_scan_different_cves_local_r1.log	2021-07-22	Miguel	🟡
test_scan_different_cves_local_r2.log	2021-07-22	Miguel	🟡
test_scan_different_cves_local_r3.log	2021-07-22	Miguel	🟡

Case 3

Test Executions	Date	By	Status
test_scan_nvd_feed_local_r1.log	2021-07-22	Miguel	🟡
test_scan_nvd_feed_local_r2.log	2021-07-22	Miguel	🟡
test_scan_nvd_feed_local_r3.log	2021-07-22	Miguel	🟡

Case 4

Test Executions	Date	By	Status
test_scan_providers_and_nvd_feed_local_r1.log	2021-07-22	Miguel	🟡
test_scan_providers_and_nvd_feed_local_r2.log	2021-07-22	Miguel	🟡
test_scan_providers_and_nvd_feed_local_r3.log	2021-07-22	Miguel	🟡

damarisg · 2021-09-10T18:16:12Z

This issue is closed because the test was disabled by #1602.

damarisg added team/qa feature/vuln-detector labels Jul 6, 2021

damarisg mentioned this issue Jul 6, 2021

Full Yellow Local - Fix Vulnerability Detector tests in 4.2 #1531

Closed

34 tasks

damarisg changed the title ~~Fix 'tests_scan_results` tests in 4.2~~ Fix tests_scan_results tests in 4.2 Jul 6, 2021

damarisg changed the title ~~Fix tests_scan_results tests in 4.2~~ Fix 'tests_scan_results' tests in 4.2 Jul 6, 2021

pereyra-m self-assigned this Jul 14, 2021

damarisg assigned damarisg and mdengra Jul 15, 2021

This was referenced Jul 19, 2021

Disabling unused modules to improve stability of the vulnerability detector module integration tests #1607

Merged

Disable unused modules for Vulnerability Detector Tests #1605

Closed

damarisg closed this as completed Sep 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix 'tests_scan_results' tests in 4.2 #1567

Fix 'tests_scan_results' tests in 4.2 #1567

damarisg commented Jul 6, 2021 •

edited

pereyra-m commented Jul 14, 2021

damarisg commented Jul 15, 2021

mdengra commented Jul 15, 2021

damarisg commented Jul 21, 2021 •

edited

mdengra commented Jul 22, 2021 •

edited

damarisg commented Sep 10, 2021

Fix 'tests_scan_results' tests in 4.2 #1567

Fix 'tests_scan_results' tests in 4.2 #1567

Comments

damarisg commented Jul 6, 2021 • edited

Issue information

Case 1:

Case 2:

Case 3:

Case 4:

pereyra-m commented Jul 14, 2021

Case 1

Case 2

Case 3

Case 4

damarisg commented Jul 15, 2021

Case 1

Case 2

Case 3

Case 4

mdengra commented Jul 15, 2021

2021-07-15

Case 1

Case 2

Case 3

Case 4

damarisg commented Jul 21, 2021 • edited

Test Results from 1531-full-yellow-vuln-det

Note

mdengra commented Jul 22, 2021 • edited

2021-07-22

Case 1

Case 2

Case 3

Case 4

damarisg commented Sep 10, 2021

damarisg commented Jul 6, 2021 •

edited

damarisg commented Jul 21, 2021 •

edited

Test Results from `1531-full-yellow-vuln-det`

mdengra commented Jul 22, 2021 •

edited