E2E tests: Local debugging and testing #3166

juliamagan · 2022-08-11T08:24:22Z

Description

After the development of the tests in #2872 and the improvements made in later tasks, the tests are ready to be merged into the numbered branch, but first, we must make sure that they are fully functional and there are no errors. To do this, all the people involved in this development must launch several runs of these tests and that the result of all of them is successful.

Previous requirements

Environment

CentOS manager (all-in-one)
CentOS agent
Ubuntu agent
Windows agent (with Office installed)

Inventory

inventory.yaml

managers:
  hosts:
    centos-manager:
      ansible_host: <IP_ADDRESS>
      ansible_connection: ssh
      ansible_user: <USER>
      ansible_ssh_private_key_file: <PRIVATE_KEY>
      ansible_python_interpreter: /usr/bin/python3
      dashboard_user: <DASHBOARD_USER>
      dashboard_password: <DASHBOARD_PASSWORD>

agents:
  children:
    linux:
      hosts:
        ubuntu-agent:
          ansible_host: <IP_ADDRESS>
          ansible_connection: ssh
          ansible_user: <USER>
          ansible_ssh_private_key_file: <PRIVATE_KEY>
          ansible_python_interpreter: /usr/bin/python3
        centos-agent:
          ansible_host: <IP_ADDRESS>
          ansible_connection: ssh
          ansible_user: <USER>
          ansible_ssh_private_key_file: <PRIVATE_KEY>
          ansible_python_interpreter: /usr/bin/python3
    windows:
      hosts:
        windows-agent:
          ansible_host: <IP_ADDRESS>
          ansible_user: <USER>
          ansible_password: <PASSWORD>
          ansible_connection: winrm
          ansible_winrm_server_cert_validation: ignore
          ansible_winrm_transport: basic
          ansible_winrm_port: 5985 (Only in local executions)
          ansible_python_interpreter: C:\Users\vagrant\AppData\Local\Programs\Pyhton\Python39\python.exe

all:
  vars:
    virustotal_key: <VIRUSTOTAL_KEY>
    bucket_name: <BUCKET_NAME>
    aws_region: <AWS_REGION>
    aws_access_key_id: <AWS_ACCESS_KEY>
    aws_secret_access_key: <AWS_SECRET_KEY>
    web_hook_url: <SLACK_WEB_HOOK_URL>
    slack_token: <SLACK_TOKEN>
    slack_channel: <SLACK_CHANNEL>
    s3_url: <S3_URL> (Should be always present)

Packages

Type	OS	Link
Manager	CentOS	https://packages-dev.wazuh.com/warehouse/test/4.4/rpm/var/wazuh-manager-4.4.0-3190.e2e.x86_64.rpm
Indexer	CentOS	https://packages-dev.wazuh.com/staging/yum/wazuh-indexer-4.4.0-0.40400.20220808.x86_64.rpm
Dashboard	CentOS	https://packages-dev.wazuh.com/staging/yum/wazuh-dashboard-4.4.0-0.40400.20220808.x86_64.rpm
Agent	CentOS	https://packages-dev.wazuh.com/warehouse/test/4.4/rpm/var/wazuh-agent-4.4.0-3190.e2e.x86_64.rpm
Agent	Ubuntu	https://packages-dev.wazuh.com/warehouse/test/4.4/deb/var/wazuh-agent_4.4.0-3190.e2e_amd64.deb
Agent	Windows	https://packages-dev.wazuh.com/warehouse/test/4.4/windows/wazuh-agent-4.4.0-3190.e2e.msi

Execution

python -m pytest tests/end_to_end/ --inventory_path=<INVENTORY_PATH> --html=R${i}-3166-e2e-<TESTER>.html --self-contained-html

Color	Status
🟢	All tests passed successfully
🟡	All tests passed but there are some warnings
🔴	Some tests have failures or errors
🔵	Test execution in progress
⚫	To Do

Proposed use cases

Open issues

The text was updated successfully, but these errors were encountered:

mauromalara · 2022-08-25T15:34:13Z

T1: Run a test using an invalid hostname in the inventory 🔴

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_suricata_integration`	CentOS and Ubuntu	🔴 🔴 🔴
@BelenValdivia	`test_sql_injection`	CentOS	🔴 🔴 🔴
@juliamagan	`test_audit`	CentOS	🔴 🔴 🔴

Tester: @BelenValdivia 🔴

Force the error by modifying the inventory:

        .
        .
        .
        agent-centos: <---- FORCING THE ERROR (expected by E2E tests: centos-agent)
          ansible_host: XX.XX.XX.XX
          ansible_connection: ssh
          ansible_user: <user>
          ansible_ssh_private_key_file: <private_key_path>
        .
        .
        .

One of the tests that use this agent is test_sql_injection, so I will run this test as follows:

python -m pytest --tb=short tests/end_to_end/test_basic_cases/test_sql_injection/  --html=T1-3166-e2e-belenvaldivia.html --inventory_path /home/belen/inventory.yaml

Result 🔴

The test fails but the validation stage does not check that the name of the hosts follows the E2E standard: <os>-<wazuh-installation-type>, e.g.: centos-agent

Tester: @juliamagan 🔴

Force the error by modifying the inventory:

testing: <---- FORCING THE ERROR (expected by E2E tests: managers)
  hosts:
    centos-manager:
      .
      .
      .

One test that uses managers is test_audit:

python -m pytest tests/end_to_end/test_basic_cases/test_audit/ --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T1-R${i}-e2e-juliamagan.html --self-contained-html

Result 🔴

The test fails because the group name is invalid, but it wasn't checked.

Tester: @mauromalara 🔴

Force the error by modifying the inventory:

    .
    .
    .
    agent-ubuntu: <---- FORCING THE ERROR (expected by E2E tests: ubuntu-agent)
      ansible_host: 172.31.5.95
      ansible_connection: ssh
      ansible_user: qa
      ansible_ssh_private_key_file: /home/mauro/ephemeral.pem
    .
    .
manager: <---- FORCING THE ERROR (expected: "managers")
  hosts:
    centos-manager:

One of the tests that use this agent is test_suricata_integration, so I will run this test as follows:

python -m pytest --tb=short tests/end_to_end/test_basic_cases/test_suricata_integration/ --inventory_path /home/mauro/inventory.yaml --html /home/mauro/T1-3166-e2e-mauromalara.html --self-contained-html

Result 🔴

The test fails but the validation stage does not check that the name of the hosts follows the E2E standard:

inventory hostname: <os>-<wazuh-installation-type>, e.g.: ubuntu-agent
groups: [managers,agents]

BelenValdivia · 2022-08-25T18:13:40Z

T2: Run a test using an invalid IP in the inventory 🟡

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_brute_force_rdp`	CentOS and Windows	🟡 🟡 🟡
@BelenValdivia	`test_sql_injection`	CentOS	🟡 🟡 🟡
@juliamagan	`test_audit`	CentOS	🟡 🟡 🟡

Tester: @BelenValdivia 🟡

Force the error by modifying the inventory:

        .
        .
        .
        centos-agent: 
          ansible_host: XX.XX.XX.XX <---- FORCING THE ERROR (invalid IP)
          ansible_connection: ssh
          ansible_user: <user>
          ansible_ssh_private_key_file: <private_key_path>
        .
        .
        .

One of the tests that use this agent is test_sql_injection, so I will run this test as follows:

python -m pytest --tb=short tests/end_to_end/test_basic_cases/test_sql_injection/  --html=T2-3166-e2e-belenvaldivia.html --inventory_path /home/belen/inventory.yaml

Result 🟢

The validation stage checks that the IP of the host is invalid

Tester: @juliamagan 🟡

Force the error by modifying the inventory:

managers:
  hosts:
    centos-manager:
      ansible_host: 1.1.1.1 <---- FORCING THE ERROR (invalid IP)
      .
      .
      .

One of the tests that use this agent is test_audit, so I will run this test as follows:

python -m pytest tests/end_to_end/test_basic_cases/test_audit/ --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T2-R${i}-e2e-juliamagan.html --self-contained-html

Result 🟡

The validation stage checks that the IP of the host is invalid, but the error is not correctly displayed.

Tester: @mauromalara 🟡

Force the error by modifying the inventory:

agents:
  hosts:
    windows-agent:
      ansible_host: 172.31.10.1 <--- FORCING THE ERROR (non-existent IP)
.
.
.
managers:
  hosts:
    centos-manager:
      ansible_host: 172.31.3.228 <--- FORCING THE ERROR (non-existent IP)

One of the tests that uses test_brute_force_rdp uses that agent and manager, so I will run that test as follows:

for i in {1..3}; do python -m pytest tests/end_to_end/test_basic_cases/test_brute_force/test_brute_force_rdp/ --inventory_path=/home/mauro/inventory.yaml --html=3166-T2-R${i}-e2e-mauromalara.html --self-contained-html; done

Result 🟡

The validation stage checks that the IP of the host is invalid, but the error is not correctly displayed.

BelenValdivia · 2022-08-25T18:31:23Z

T3: Run a test using an invalid indexer password in the inventory 🟡

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_docker_monitoring`	CentOS	🟡 🟡 🟡
@BelenValdivia	`test_audit`	CentOS	🟡 🟡 🟡
@juliamagan	`test_detecting_suspicious_binaries`	CentOS	🟡 🟡 🟡

Tester: @BelenValdivia 🟡

Force the error by modifying the indexer password in the inventory:

        .
        .
        .
        centos-manager: 
          ansible_host: XX.XX.XX.XX 
          ansible_connection: ssh
          ansible_user: <user>
          ansible_ssh_private_key_file: /home/belen/ZZZZ.pem
          dashboard_user:  <user>
          dashboard_password: <invalid_password>  <---- FORCING THE ERROR (invalid indexer password)
        .
        .
        .

Test executed test_audit, I will run this test as follows:

python -m pytest --tb=short tests/end_to_end/test_basic_cases/test_audit/  --html=T3-3166-e2e-belenvaldivia.html --inventory_path /home/belen/inventory.yaml

Result 🟡

401 Unauthorized response from indexer. We could check this in the validation phase.

Tester: @juliamagan 🟡

Force the error by modifying the indexer password in the inventory:

        .
        .
        .
        centos-manager: 
          dashboard_password: <invalid_password>  <---- FORCING THE ERROR (invalid indexer password)
        .
        .
        .

Test executed test_detecting_supsicious_binaries, I will run this test as follows:

python -m pytest tests/end_to_end/test_basic_cases/test_detecting_suspicious_binaries/ --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T3-R${i}-e2e-juliamagan.html --self-contained-html

Result 🟡

The API request fails as expected, but maybe, we could check this in the validation phase.

Tester: @mauromalara 🟡

Forcing the error by modifying the inventory:

managers:
  hosts:
    centos-manager:
      dashboard_user: admin
      dashboard_password: bad_pass <--- FORCING THE ERROR

Run test_docker_monitoring as follows:

for i in {1..3}; do python -m pytest tests/end_to_end/test_basic_cases/test_docker_monitoring/ --inventory_path=/home/mauro/inventory.yaml --html=3166-T3-R${i}-e2e-mauromalara.html --self-contained-html; done

Result 🟡

The API request fails as expected, but maybe, we could check this in the validation phase.

BelenValdivia · 2022-08-25T18:55:03Z

T4: Run 2 tests with different environments 🟢

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_unauthorized_processes_detection` and `test_fim_windows`	CentOS and Windows	🟢 🟢 🟢
@BelenValdivia	`test_audit` and `test_windows_defender`	CentOS and Windows	🟢 🟢 🟢
@juliamagan	`test_detecting_suspicious_binaries` and `test_ip_reputation`	CentOS and Windows	🟢 🟢 🟢

Tester: @BelenValdivia 🟢

Execute 1 test that needs centos-agent and 1 tests that needs windows-agent

Tests to run:
- Test Audit (centos-agent)
- Test Windows Defender (windows agent)
I will run these tests as follows:

python -m pytest --tb=short tests/end_to_end/test_basic_cases/test_audit/ tests/end_to_end/test_basic_cases/test_windows_defender/  --html=T4-3166-e2e-belenvaldivia.html --inventory_path /home/belen/inventory.yaml

Result 🟢

Both tests passed

Tester: @juliamagan 🟢

Execute 1 test that needs centos-manager and 1 tests that needs windows-agent

Tests to run:
- test_detectiong_suspicious_binaries (centos-manager)
- test_ip_reputation (centos-manager and windows-agent)
I will run these tests as follows:

python -m pytest tests/end_to_end/test_basic_cases/test_detecting_suspicious_binaries/ tests/end_to_end/test_basic_cases/test_ip_reputation/ --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T4-R${i}-e2e-juliamagan.html --self-contained-html

Result 🟢

Both tests passed

Tester: @mauromalara 🟢

Run the tests as follows:

for i in {1..3}; do python -m pytest tests/end_to_end/test_basic_cases/test_fim/test_fim_windows/ tests/end_to_end/test_basic_cases/test_unauthorized_processes_detection/ --inventory_path=/home/mauro/inventory.yaml --html=3166-T3-R${i}-e2e-mauromalara.html --self-contained-html; done

Result 🟢

All tests have passed successfully.

BelenValdivia · 2022-08-25T19:16:20Z

T5: Run 2 tests with the same environment 🟢

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_brute_force_rdp` and `test_fim_windows`	CentOS and Windows	🟢 🟢 🟢
@BelenValdivia	`test_emotet` and `test_windows_defender`	CentOS and Windows	🟢 🟢 🟢
@juliamagan	`test_ip_reputation` and `test_windows_defender`	CentOS and Windows	🟢 🟢 🟢

Tester: @BelenValdivia 🟢

Execute 2 tests that need Windows agent and CentOS manager

Tests to run:
- Test Emotet
- Test Windows defender
I will run these tests as follows:

python -m pytest --tb=short tests/end_to_end/test_basic_cases/test_audit/ tests/end_to_end/test_basic_cases/test_sql_injection/  --html=T5-3166-e2e-belenvaldivia.html --inventory_path /home/belen/inventory.yaml

Result 🟢

Both tests passed.

Report: T5-3166-e2e-belenvaldivia.zip

Tester: @juliamagan 🟢

Run two tests with the same environment:

test_ip_reputation (centos-manager and windows-agent)
test_windows_defender (centos-manager and windows-agent)

I will run the tests as follows:

python -m pytest tests/end_to_end/test_basic_cases/test_windows_defender/ tests/end_to_end/test_basic_cases/test_ip_reputation/ --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T5-R${i}-e2e-juliamagan.html --self-contained-html

Result 🟢

Both tests passed.

Tester: @mauromalara 🟢

Run the tests as follow:

for i in {1..3}; do python -m pytest tests/end_to_end/test_basic_cases/test_fim/test_fim_windows/ tests/end_to_end/test_basic_cases/test_brute_force/test_brute_force_rdp/ --inventory_path=/home/mauro/inventory.yaml --html=3166-T3-R${i}-e2e-mauromalara.html --self-contained-html; done

Result 🟢

Both tests have passed.

mauromalara · 2022-08-26T01:13:37Z

T6: Run 1 test (with more than 1 test case) selecting 1 test case (-k) 🔴

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_fim -k delete_file_linux`	CentOS	🔴 🔴 🔴
@BelenValdivia	`test_virustotal_integration`	CentOS	🟢 🟢 🟢
@juliamagan	`test_docker_monitoring`	CentOS	🟢 🟢 🟢

Tester: @BelenValdivia 🟢

Test with multiple cases to run test_virustotal_integration
Run one test case:

python -m pytest tests/end_to_end/test_basic_cases/test_virustotal_integration/ -k remove_malicious_file  --inventory_path=/home/belen/inventory.yml --html=3166-T6-R${i}-e2e-belenvaldivia.html --self-contained-html

Result 🟢

The specific test case passed

Tester: @juliamagan 🟢

Choose a test with multiple test cases: test_docker_monitoring
Run one test case:

python -m pytest tests/end_to_end/test_basic_cases/test_docker_monitoring/ -k docker_pull  --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T6-R${i}-e2e-juliamagan.html --self-contained-html

Result 🟢

The specific test case passed

Tester: @mauromalara 🔴

Choose a test with multiple test cases: test_fim
Run one test case:

for i in {1..3}; do python -m pytest tests/end_to_end/test_basic_cases/test_fim/ -k delete_file_linux --inventory_path=/home/mauro/inventory.yaml --html=3166-T3-R${i}-e2e-mauromalara.html --self-contained-html; done

Result 🔴

The test has failed, the failure must be researched.

mauromalara · 2022-08-26T01:15:29Z

T7: Run all tests but select only 1 test (-k) 🟢

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_sql_injection`	CentOS	🟢 🟢 🟢
@BelenValdivia	`test_suricata_integration`	CentOS	🟢 🟢 🟢
@juliamagan	`test_audit`	CentOS	🟢 🟢 🟢

Tester: @BelenValdivia 🟢

1. Launch all tests but select just one: `test_suricata_integration` 2. Command:

python -m pytest tests/end_to_end/test_basic_cases/ -k suricata_integration  --inventory_path=/home/belen/inventory.yml --html=3166-T7-R${i}-e2e-belenvaldivia.html --self-contained-html

Result 🟢

The specific test passed

Tester: @juliamagan 🟢

Launch all tests but select just one: test_audit
Command:

python -m pytest tests/end_to_end/test_basic_cases/ -k audit  --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T7-R${i}-e2e-juliamagan.html --self-contained-html

Result 🟢

The specific test passed

Tester: @mauromalara 🟢

Launch all tests but select just one: test_sql_injection:

for i in {1..3}; do python -m pytest tests/end_to_end/test_basic_cases/ -k sql_injection --inventory_path=/home/mauro/inventory.yaml --html=3166-T7-R${i}-e2e-mauromalara.html --self-contained-html; done

Result 🟢

The specific test passed

mauromalara · 2022-08-26T01:16:11Z

T8: Run a test with an unsupported Linux distribution installed in a target host 🔴

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_brute_force_ssh`	CentOS	🟢 🟢 🟢
@BelenValdivia	`test_sql_injection`	CentOS	🟢 🟢 🟢
@juliamagan	`test_suricata_integration`	CentOS and Ubuntu	🔴 🔴 🔴

Tester: @BelenValdivia 🟢

1. Force the error. Define a Centos agent in the Inventory but the host is a Ubuntu agent.

agents:
  children:
    linux:
      hosts:
        centos-agent: ---> FORCE ERROR
          ansible_host: <ip_ubuntu_agent>
          ansible_user: <user>
          ansible_connection: ssh
          ansible_ssh_private_key_file: /home/belen/zzzzz.pem

Test executed test_sql_injection, I will run this test as follows:

python -m pytest tests/end_to_end/test_basic_cases/test_sql_injection/ --inventory_path=/home/belen/inventory.yml --html=3166-T9-R${i}-e2e-belenvaldivia.html --self-contained-html

Result 🟢

The validation stage checks the incorrect Linux distribution installed in a target host.

Tester: @juliamagan 🔴

The test_suricata_integration test needs an Ubuntu agent, we have configured a CentOS agent instead:

agents:
  children:
    linux:
      hosts:
        ubuntu-agent: ---> FORCE ERROR
          ansible_host: <ip_centos_agent>

Run the test as follows:

python -m pytest tests/end_to_end/test_basic_cases/test_suricata_integration/  --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T8-R${i}-e2e-juliamagan.html --self-contained-html

Result 🔴

The distribution wasn't checked in the validation phase, so the test was launched and failed.

Tester: @mauromalara 🟢

The test_brute_force_ssh test needs a CentOS agent, we have configured an Ubuntu agent instead:

        centos-agent:
          ansible_host: 172.31.13.31 <--- FORCING THE ERROR (ubuntu agent IP)

Run the test

Result 🟢

The validation works as expected.

mauromalara · 2022-08-26T01:17:26Z

T9: Run a test in a target host with an incorrect OS 🟡

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_emotet`	Centos and Windows	🟡 🟡 🟡
@BelenValdivia	`test_sql_injection`	CentOS	🟢 🟢 🟢
@juliamagan	`test_windows_defender`	CentOS and Windows	🟢 🟢 🟢

Tester: @BelenValdivia 🟢

1. Force the error in the Inventory. Define a CentOS agent but the host is a Windows agent

agents:
    hosts:
        centos-agent: ---> FORCE ERROR
          ansible_host: XX.XX.XX.XX ---> windows agent
          ansible_user: <user>
          ansible_connection: ssh
          ansible_ssh_private_key_file: /home/belen/zzzzz.pem

Test executed test_sql_injection, I will run this test as follows:

python -m pytest tests/end_to_end/test_basic_cases/test_sql_injection/ --inventory_path=/home/belen/inventory.yml --html=3166-T9-R${i}-e2e-belenvaldivia.html --self-contained-html

Result 🟢

The test fails but the validation stage does not check the incorrect OS defined in the inventory

Tester: @juliamagan 🟢

The test_windows_defender test needs a Windows agent, we have configured an Ubuntu agent instead:

agents:
  children:
    linux:
      hosts:
        windows-agent: ---> FORCE ERROR
          ansible_host: <ip_ubuntu_agent>

Run the test as follows:

python -m pytest tests/end_to_end/test_basic_cases/test_windows_defender/  --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T9-R${i}-e2e-juliamagan.html --self-contained-html

Result 🟢

The OS was checked in the validation phase and failed, so the test wasn't launched.

Tester: @mauromalara 🟡

The test_emotet test needs a CentOS manager, so we force the error by modifying the inventory as follows:

managers:
  hosts:
    centos-manager:
      ansible_host: 172.31.11.210 <--- Force the error (Window agent IP)

Result 🟡

The validation phase fails, but it does not check the OS, instead, it fails while gathering Ansible facts.

mauromalara · 2022-08-26T01:22:35Z

T10: Run a test in a target host with the manager service stopped 🟡

Conclusion

🟡: All executions were run successfully, but @mauromalara detect that the validation stage does not include the validation for a Windows agent service.

Tester	Test	OS	Results
@mauromalara	`test_yara_integration`	CentOS	🟢 🟢 🟢
@BelenValdivia	`test_detecting_suspicious_binaries`	CentOS	🟢 🟢 🟢
@juliamagan	`test_audit`	CentOS	🟢 🟢 🟢

Tester: @BelenValdivia 🟢

1. Force the error by stopping the Wazuh manager.

Run test_detecting_suspicious_binaries, so I will run this test as follows:

python -m pytest  tests/end_to_end/test_basic_cases/test_detecting_suspicious_binaries/  --html=T10-3166-e2e-belenvaldivia.html --inventory_path /home/belen/inventory.yaml

Result 🟢

The test fails, the validation stage checks that the Wazuh manager is stopped

Tester: @juliamagan 🟢

Force the error by stopping the manager.
Run the test as follows:

python -m pytest tests/end_to_end/test_basic_cases/test_audit/  --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T10-R${i}-e2e-juliamagan.html --self-contained-html

Result 🟢

It is checked that the manager should be running correctly, so the test isn't launched.

Tester: @mauromalara 🟢

Stop the wazuh-manager service: systemctl stop wazuh-manager
Run the test_yara_integration, which requires 1 manager to run.

Result 🟢

juliamagan · 2022-08-26T09:06:29Z

T11: Run all test cases from the E2E folder 🔴

Conclusion

Tester	Test	OS	Results
@mauromalara	`end_to_end`	CentOS, Ubuntu and Windows	🔴 🔴 🔴
@BelenValdivia	`end_to_end`	CentOS, Ubuntu and Windows	🔴 🔴 🔴
@juliamagan	`test_basic_cases`	CentOS, Ubuntu and Windows	🔴🔴🔴

Tester: @BelenValdivia 🔴

1. Launched every tests:

python -m pytest tests/end_to_end/test_basic_cases/ --inventory_path=/home/belen/inventory.yml --html=3166-T11-R${i}-e2e-juliamagan.html --self-contained-html

Result 🔴

The following tests should not have failed:

test_brute_force_rdp -> R1, R2, R3. Error: The alert has not ocurred
test_virustotal_integration[remove_malicious_file] -> R1. Error: Alert triggered but not indexed
test_vulnerability_detector_windows -> R1, R2, R3. Error: The alert has not ocurred. Bug opened
test_vulnerability_detector_linux -> R1, R2, R3. Error: The alert has not ocurred
- test_docker_monitoring[docker_pull] -> R3. Error: The alert has not ocurred

Tester: @juliamagan 🔴

Launch every test

python -m pytest tests/end_to_end/test_basic_cases/  --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T11-R${i}-e2e-juliamagan.html --self-contained-html

Result 🔴

The following tests should not have failed:

test_aws_infrastructure_monitoring -> R1
test_virustotal_integration[remove_malicious_file] -> R1, R2, R3
test_vulnerability_detector-> R1, R2, R3

Tester @mauromalara 🔴

In the root directory of the QA repo, go to tests/end_to_end
Run all tests as follows:

python -m pytest --tb=short ./ --inventory_path /home/mauro/inventory.yaml --html /home/mauro/T11-3166-e2e-mauromalara.html --self-contained-html

Result 🔴

The following tests have failed:

test_vulnerability_detector_linux
test_vulnerability_detector_windows
test_brute_force_ssh[ssh_brute_force]
test_emotet[emotet_attack]
test_virustotal_integration[remove_malicious_file]
test_aws_infrastructure_monitoring[CloudTrail service]

juliamagan · 2022-08-26T11:36:05Z

T12: Run all tests that require Linux 🔴

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_basic_cases -m linux`	CentOS and Ubuntu	🔴 🔴 🔴
@BelenValdivia	`test_basic_cases -m linux`	CentOS and Ubuntu	🔴 🔴 🔴
@juliamagan	`test_basic_cases -m linux`	CentOS and Ubuntu	🔴 🔴 🔴

Tester: @BelenValdivia 🔴

1. Run all tests that require Linux as follows:

python -m pytest tests/end_to_end/test_basic_cases/ -m linux  --inventory_path=/home/belen/inventory.yml --html=3166-T12-R${i}-e2e-belenvaldivia.html --self-contained-html

Result 🔴

We can't run our tests using markers because this feature isn't implemented yet.

Tester: @juliamagan 🔴

Run all tests that require Linux as follows:

python -m pytest tests/end_to_end/test_basic_cases/ -m linux  --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T12-R${i}-e2e-juliamagan.html --self-contained-html

Result 🔴

We can't run our tests using markers because this feature isn't implemented yet.

Tester: @mauromalara 🔴

Run all tests that require Linux as follows:

python -m pytest tests/end_to_end/test_basic_cases/ -m linux --inventory_path=/home/mauro/inventory.yaml

Result 🔴

We can't run our tests using markers because this feature isn't implemented yet.

juliamagan · 2022-08-26T11:36:32Z

T13: Run all tests that require Windows 🔴

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_basic_cases -m windows`	CentOS and Windows	🔴 🔴 🔴
@BelenValdivia	`test_basic_cases -m windows`	CentOS and Windows	🔴 🔴 🔴
@juliamagan	`test_basic_cases -m windows`	CentOS and Windows	🔴🔴🔴

Tester: @BelenValdivia 🔴

1. Run all tests that require Windows as follows:

python -m pytest tests/end_to_end/test_basic_cases/ -m windows  --inventory_path=/home/belen/inventory.yml --html=3166-T13-R${i}-e2e-belenvaldivia.html --self-contained-html

Result 🔴

We can't run our tests using markers because this feature isn't implemented yet.

Tester: @juliamagan 🔴

Run all tests that require Windows as follows:

python -m pytest tests/end_to_end/test_basic_cases/ -m windows  --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T13-R${i}-e2e-juliamagan.html --self-contained-html

Result 🔴

We can't run our tests using markers because this feature isn't implemented yet.

Tester: @mauromalara 🔴

Run all tests that require Windows as follows:

python -m pytest tests/end_to_end/test_basic_cases/ -m windows --inventory_path=/home/mauro/inventory.yaml

Result 🔴

We can't run our tests using markers because this feature isn't implemented yet.

juliamagan · 2022-08-26T11:37:02Z

T14: Run a test without specifying the inventory 🟢

Conclusion

Tester	Test	OS	Results
@mauromalara	`all`	CentOS, Ubuntu and Windows	🟢 🟢 🟢
@BelenValdivia	`test_windows_defender`	Windows	🟢 🟢 🟢
@juliamagan	`test_audit`	CentOS	🟢 🟢 🟢

Tester: @BelenValdivia 🟢

1. Run the test without specifying the inventory:

python -m pytest tests/end_to_end/test_basic_cases/test_windows_defender  --html=3166-T14-R${i}-e2e-belenvaldivia.html --self-contained-html

Result 🟢

Nothing is launched because there is no inventory

Tester: @juliamagan 🟢

Run the test without specifying the inventory:

python -m pytest tests/end_to_end/test_basic_cases/test_audit/  --html=3166-T14-R${i}-e2e-juliamagan.html --self-contained-html

Result 🟢

Nothing is launched because there is no inventory

Tester: @mauromalara 🟢

Run the test without specifying the inventory:

python -m pytest tests/end_to_end/test_basic_cases/

Result 🟢

No test was executed because there is no inventory.

juliamagan · 2022-08-26T11:37:37Z

T15: Run all tests but unselect tests (-k not) 🔴

Conclusion

Tester	Test	OS	Results
@mauromalara	`test_basic_cases -k "not aws and not docker"`	CentOS, Ubuntu and Windows	🔴 🔴 🔴
@BelenValdivia	`test_basic_cases -k "not yara"`	CentOS, Ubuntu and Windows	🔴 🔴 🔴
@juliamagan	`test_basic_cases -k "not vulnerability"`	CentOS, Ubuntu and Windows	🔴🔴🔴

Tester: @BelenValdivia 🔴

1. Launched every test except `test_yara_integration`:

python -m pytest tests/end_to_end/test_basic_cases/ -k "not yara" --inventory_path=/home/belen/inventory.yml --html=3166-T15-R${i}-e2e-belenvaldivia.html --self-contained-html

Result 🔴

The following tests should not have failed:

test_brute_force_rdp -> R1, R2, R3. Error: The alert has not ocurred
test_virustotal_integration[remove_malicious_file] -> R1. Error: The alert has not ocurred
test_vulnerability_detector_windows -> R1, R2, R3. Error: The alert has not ocurred. Bug opened
test_vulnerability_detector_linux -> R1, R2, R3. Error: The alert has not ocurred
test_suricata_integration -> R2. Error: The alert has not occurred

Tester: @juliamagan 🔴

Launched every test except test_vulnerability_detector:

python -m pytest tests/end_to_end/test_basic_cases/ -k "not vulnerability" --inventory_path=/home/juliamagan/Desktop/QA/2893/inventory.yml --html=3166-T15-R${i}-e2e-juliamagan.html --self-contained-html

Result 🔴

The following tests should not have failed:

test_brute_force_ssh -> R1
test_virustotal_integration[remove_malicious_file] -> R1, R2 and R3
test_osquery_integration -> R2
test_aws_infrastructure_monitoring -> R3

Tester: @mauromalara 🔴

Run the tests as follows: python -m pytest tests/end_to_end/test_basic_cases/ -k "not aws and not docker" --html=3166-T15-R${i}-e2e-mauromalara.html --inventory_path=/home/mauro/inventory.yaml --self-contained-html

Result 🔴

test_brute_force_ssh
test_emotet
test_virustotal_integration (in R3 the test passed, maybe is the timeout)
test_vulnerability_detector_linux
test_vulnerability_detector_windows

juliamagan added team/qa test/e2e labels Aug 11, 2022

juliamagan assigned BelenValdivia, mauromalara and juliamagan Aug 11, 2022

juliamagan mentioned this issue Aug 11, 2022

E2E tests for demo environment #2830

Closed

13 tasks

wazuh deleted a comment from mauromalara Aug 25, 2022

This was referenced Aug 29, 2022

E2E tests: Improve validation stage #3203

Closed

E2E tests: Add markers to specify supported OS #3204

Closed

E2E tests: Research Virustotal test failure #3205

Closed

This was referenced Aug 29, 2022

E2E tests: Fix FIM test failures #3207

Closed

E2E tests: Research Brute Force tests failures #3208

Closed

E2E tests: Research Emotet test failures #3209

Closed

E2E tests: Fix Docker Monitoring test failure #3210

Closed

mauromalara mentioned this issue Aug 30, 2022

E2E tests: Fix AWS infrastructure monitoring test failures #3211

Closed

This was referenced Aug 30, 2022

E2E tests: Research Osquery test failure #3212

Closed

E2E tests: Research Suricata test failure #3213

Closed

juliamagan closed this as completed Aug 30, 2022

mauromalara mentioned this issue Aug 30, 2022

E2E tests: Improve validation stage #3215

Merged

mauromalara mentioned this issue Sep 13, 2022

E2E Tests - Fix unstable tests #3287

Closed

4 tasks

E2E tests: Local debugging and testing #3166

E2E tests: Local debugging and testing #3166

Comments

juliamagan commented Aug 11, 2022 • edited by mauromalara

Description

Previous requirements

Environment

Inventory

Packages

Execution

Proposed use cases

Open issues

mauromalara commented Aug 25, 2022 • edited by juliamagan

T1: Run a test using an invalid hostname in the inventory 🔴

Conclusion

Result 🔴

Result 🔴

Result 🔴

BelenValdivia commented Aug 25, 2022 • edited

T2: Run a test using an invalid IP in the inventory 🟡

Conclusion

Result 🟢

Result 🟡

Result 🟡

BelenValdivia commented Aug 25, 2022 • edited

T3: Run a test using an invalid indexer password in the inventory 🟡

Conclusion

Result 🟡

Result 🟡

Result 🟡

BelenValdivia commented Aug 25, 2022 • edited by mauromalara

T4: Run 2 tests with different environments 🟢

Conclusion

Result 🟢

Result 🟢

Result 🟢

BelenValdivia commented Aug 25, 2022 • edited by mauromalara

T5: Run 2 tests with the same environment 🟢

Conclusion

Result 🟢

Result 🟢

Result 🟢

mauromalara commented Aug 26, 2022 • edited

T6: Run 1 test (with more than 1 test case) selecting 1 test case (-k) 🔴

Conclusion

Result 🟢

Result 🟢

Result 🔴

mauromalara commented Aug 26, 2022 • edited

T7: Run all tests but select only 1 test (-k) 🟢

Conclusion

Result 🟢

Result 🟢

Result 🟢

mauromalara commented Aug 26, 2022 • edited

T8: Run a test with an unsupported Linux distribution installed in a target host 🔴

Conclusion

Result 🟢

Result 🔴

Result 🟢

mauromalara commented Aug 26, 2022 • edited

T9: Run a test in a target host with an incorrect OS 🟡

Conclusion

Result 🟢

Result 🟢

Result 🟡

mauromalara commented Aug 26, 2022 • edited

T10: Run a test in a target host with the manager service stopped 🟡

Conclusion

Result 🟢

Result 🟢

Result 🟢

juliamagan commented Aug 26, 2022 • edited by BelenValdivia

T11: Run all test cases from the E2E folder 🔴

Conclusion

Result 🔴

Result 🔴

Result 🔴

juliamagan commented Aug 26, 2022 • edited by mauromalara

T12: Run all tests that require Linux 🔴

juliamagan commented Aug 11, 2022 •

edited by mauromalara

mauromalara commented Aug 25, 2022 •

edited by juliamagan

BelenValdivia commented Aug 25, 2022 •

edited

BelenValdivia commented Aug 25, 2022 •

edited

BelenValdivia commented Aug 25, 2022 •

edited by mauromalara

BelenValdivia commented Aug 25, 2022 •

edited by mauromalara

mauromalara commented Aug 26, 2022 •

edited

mauromalara commented Aug 26, 2022 •

edited

mauromalara commented Aug 26, 2022 •

edited

mauromalara commented Aug 26, 2022 •

edited

mauromalara commented Aug 26, 2022 •

edited

juliamagan commented Aug 26, 2022 •

edited by BelenValdivia

juliamagan commented Aug 26, 2022 •

edited by mauromalara

juliamagan commented Aug 26, 2022 •

edited by mauromalara

juliamagan commented Aug 26, 2022 •

edited by mauromalara

juliamagan commented Aug 26, 2022 •

edited by mauromalara