-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generate isolated resources for each AWS integration test #4714
Generate isolated resources for each AWS integration test #4714
Comments
Issue UpdateI have set up a testing environment that consists of 2 vagrant machines with all the necessary tools to run the AWS ITs, all tests will be run simultaneously to determine exactly which test and more specifically which resource will cause an error on parallel executions First VM detailsroot@vagrant:/wazuh/tests/integration# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@vagrant:/wazuh/tests/integration# uname -a
Linux vagrant 5.15.0-83-generic #92-Ubuntu SMP Mon Aug 14 09:30:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
VM ID root@vagrant:/wazuh/tests/integration/test_aws# dmidecode -s system-uuid
fbc1928d-7955-9b42-89b7-9717dfe5a251
Second VM detailsroot@vagrant:/wazuh/tests/integration/test_aws# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@vagrant:/wazuh/tests/integration/test_aws# uname -a
Linux vagrant 5.15.0-83-generic #92-Ubuntu SMP Mon Aug 14 09:30:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
VM ID root@vagrant:/wazuh/tests/integration/test_aws# dmidecode -s system-uuid
4b9b295c-828a-644f-b0bd-bfbddc4c82c4
Testing Results
Test logstest_basic 🟢root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_basic.py
=================================== test session starts ===================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 16 items
test_basic.py ................ [100%]
======================= 16 passed, 7 warnings in 336.66s (0:05:36) ========================
root@vagrant:/wazuh/tests/integration/test_aws#
──────────────────────────────────────────────────────────────────────────────────────────
root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_basic.py
=================================== test session starts ===================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 16 items
test_basic.py ................ [100%]
======================= 16 passed, 5 warnings in 329.51s (0:05:29) ========================
root@vagrant:/wazuh/tests/integration/test_aws# test_custom_bucket 🟢root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_custom_bucket.py
=================================== test session starts ===================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 2 items
test_custom_bucket.py .. [100%]
──────────────────────────────────────────────────────────────────────────────────────────
root@vagrant:/wazuh/tests/integration/test_aws# pytest test_custom_bucket.py
========================= test session starts ==========================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 2 items
test_custom_bucket.py .. [100%]
==================== 2 passed, 5 warnings in 43.81s ====================
test_discard_regex 🟢root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_discard_regex.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 17 items
test_discard_regex.py ................. [100%]
========================================================================== 17 passed, 7 warnings in 280.54s (0:04:40) ==========================================================================
──────────────────────────────────────────────────────────────────────────────────────────
root@vagrant:/wazuh/tests/integration/test_aws# pytest test_discard_regex.py
=============================== test session starts ===============================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 17 items
test_discard_regex.py ................. [100%]
============== 17 passed, 5 warnings in 278.25s (0:04:38) ==============
=========================================================================================== FAILURES =========================================================================================== test_log_groups 🔴root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_log_groups.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 2 items
test_log_groups.py .. [100%]
================================================================================ 2 passed, 7 warnings in 47.69s ================================================================================
──────────────────────────────────────────────────────────────────────────────────────────
root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_log_groups.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 2 items
test_log_groups.py EF [100%]
============================================================================================ ERRORS ============================================================================================
test_only_logs_after 🔴root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_only_logs_after.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 47 items
test_only_logs_after.py F..FF.....F.F.F..............................x. [100%]
──────────────────────────────────────────────────────────────────────────────────────────
root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_only_logs_after.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 47 items
test_only_logs_after.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFxFE test_parser 🟢root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_parser.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 26 items
test_parser.py ..............F........... [100%]
──────────────────────────────────────────────────────────────────────────────────────────
root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_parser.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 26 items
test_parser.py ..............F........... [100%] test_path 🔴root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_path.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 42 items
test_path.py .......................................... [100%]
========================================================================== 42 passed, 7 warnings in 923.10s (0:15:23) ==========================================================================
──────────────────────────────────────────────────────────────────────────────────────────
root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_path.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 42 items
test_path.py FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF [100%]
=========================================================================================== FAILURES =========================================================================================== test_path_suffix 🔴root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_path_suffix.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 9 items
test_path_suffix.py ......... [100%]
========================================================================== 9 passed, 7 warnings in 195.22s (0:03:15) ===========================================================================
──────────────────────────────────────────────────────────────────────────────────────────
root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_path_suffix.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 9 items
test_path_suffix.py FFFFFFFFF [100%]
=========================================================================================== FAILURES =========================================================================================== test_regions 🔴root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_regions.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 24 items
test_regions.py ........................ [100%]
========================================================================== 24 passed, 8 warnings in 555.34s (0:09:15) ==========================================================================
──────────────────────────────────────────────────────────────────────────────────────────
root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_regions.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 24 items
test_regions.py FFFFFFFFFFFFFFFFFFFFFFFF [100%]
=========================================================================================== FAILURES =========================================================================================== test_remove_from_bucket 🔴root@vagrant:/wazuh/tests/integration/test_aws# pytest --disable-warnings test_remove_from_bucket.py
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 15 items
test_remove_from_bucket.py ...sss......... [100%]
==================================================================== 12 passed, 3 skipped, 7 warnings in 287.86s (0:04:47) =====================================================================
──────────────────────────────────────────────────────────────────────────────────────────
root@ubuntu2204:/wazuh/tests/integration/test_aws# pytest test_remove_from_bucket.py
============================== test session starts ===============================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: html-3.1.1, metadata-3.0.0
collected 15 items
test_remove_from_bucket.py ...sss......... [100%]
|
Issue updateDuring the testing and debugging of the test_log_groups and the test_remove_from_bucket, it was found that to test a successful parallel execution, a log stream was tried to be created resulting in an error due to lack of permission to create it. An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:iam::966237403726:user/eduardo.leon is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:us-east-1:966237403726:log-group:temporary-log-group-9935:log-stream: because no identity-based policy allows the logs:CreateLogGroup action The permission has already been requested and the test will continue once it has been granted. |
Issue UpdateAfter discussing with the team manager and analyzing different approaches to the issue it was decided to add new fixtures to isolate the creation of every needed AWS resource for the launch of the tests, these will include S3 buckets and services with an added hash to ensure entropy within the parallel executions of tests. The official AWS documentation regarding the cost of resources states, that there is no cost associated with the creation of the buckets and log groups or streams, and since the size of logs used for every test averages 50B, it can be concluded that there is not going to be an increase of cost on the execution of tests. This will also allow us to not depend on a specific AWS account to run the tests but be able to run them in any sandbox account. Currently working on the S3 fixture extracting the parameters from the data set of the tests and applying them to the creation of the bucket endpoint, this is being added to the framework alongside other methods of AWS resources manipulation. |
Issue UpdateAfter discussing with the team about possible implementations for the fixture they will be made following these guidelines:
All fixtures have been developed. The next step is to test the fixtures and ensure tests are passing with the inclusion of the uuid on the name. |
Issue UpdateAfter testing the changes it was found that there were some flaws in the design developed, Some due to errors in the scopes of the fixtures, and also it wasn't as readable. For better design, the On the other hand, The fixtures of the core repository were also reworked, and each resource method now includes only its creation and later addition to a new list of all resources for them to be deleted upon completion of the test. Finally, all metadata has been edited to include a new key called |
Issue UpdateWorking on core repository fixtures import errors due to the changes. Some fixtures are under a big refactor since the deletion part of the resource it's only necessary for the log group and the buckets, because all resources created inside these will be deleted as well. Because of the mentioned behaviour the logic of yielding filenames, log streams or events is no longer needed. Therefore, it is being removed and optimized. |
Issue UpdateAfter a back and forth on the modification of the |
Issue UpdateWorking on fixing various import and execution errors, adapting imports, and standardizing boto interface to be client interface only. |
Issue UpdateAfter several debugs of boto errors, It was found that the errors were due to versioning, Framework outdated version of boto has different parameters on the API calls for resource creation. Some tests will be launched to determine if upgrading the package in the testing framework is not a breaking change. |
Issue updateAll testing on the creation and deletion of resources has been tested successfully. However, it was found that a new fixture to edit the test cases was needed. The current test works by loading into the
|
Issue UpdateAfter testing the proposal, It was changed. The configurator is now in charge of managing the resources and configuring the files accordingly.
The creation and deletion still require some though and work process since the resource_type is needed to verify the type of resource that is going to be created and later on deleted. |
Issue UpdateIt was found that the fixtures needed modification because the encapsulation of the configurator had to be changed to a test wrapper to make it dynamic and usable as a fixture for each test in every module. This change allows them to use the same session_id for resource identification during parallel runs. |
Issue updateIt was decided to also add a decorator to set the module name for the configurator for each test. |
Issue updateAfter realizing an error during the execution of the design due to the incompatibility of the timing imports between decorators and fixtures it was decided to go with a more modular design. Abstracting the responsibility of the configurator according to what was said here. It was consulted with the manager to conclude the design regarding the responsibility of the creation of the resources. |
Issue Update
Currently debugging an error in the configuration setup of the module, which is not setting the profile used making the regex not match causing the test to fail. |
Issue update
root@vagrant:/wazuh/tests/integration/test_aws# pytest -sx --disable-warnings test_basic.py
================================ test session starts ================================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collected 16 items
test_basic.py ................
==================== 16 passed, 2 warnings in 294.42s (0:04:54) ===================== Currently working on the custom bucket test suite on the |
Issue updateThe upload of the file has been developed and tested.
root@vagrant:/wazuh/tests/integration/test_aws# pytest -sx --disable-warnings test_custom_bucket.py
======================== test session starts =========================
platform linux -- Python 3.10.12, pytest-7.1.2, pluggy-1.3.0
rootdir: /wazuh/tests/integration, configfile: pytest.ini
plugins: metadata-3.0.0, html-3.1.1
collecting ...
collected 2 items
test_custom_bucket.py ..
============== 2 passed, 2 warnings in 75.93s (0:01:15) ============== |
Issue UpdateDue to internal changes, the tasks needed to complete this issue based on each test module will be listed. Note This tasks are not conclusive, they are just a proposal. For all modules.
Discard regex
Log groups
Only logs after
Parser
If it is decided to manage the exception the use of a logger Path Suffix
Path
Remove from bucket
Ex of change: # From
assert not file_exists(filename=metadata['uploaded_file'], bucket_name=bucket_name)
# To
expected_log = "INSERT LOG WHEN THE BUCKET OR LOG STREAM IS EMPTY"
log_monitor.start(
timeout=session_parameters.default_timeout,
callback=event_monitor.make_aws_callback(expected_log)
) Also, if multiple files are present in the bucket when asserting and the empty bucket is not a reassurance that the removal was correct the regex can be defined to look for the file using the specific key assigned to the metadata of the test and assert for that the callback_result should be None since no match should occur. Regions
|
ConclusionSome minor changes were left, but since the issue points to an epic with a specific branch, the team considers that these can be carried out in wazuh/wazuh#22514 and wazuh/qa-integration-framework#133 which will also point to the epic to avoid overextending this issue. |
Description
During the investigation in wazuh/wazuh#18478, we've found that the AWS integration tests create resources using hardcoded parameters, meaning that every test tries to create the same resource and they fail if it already exists. Because of this, it's not possible to run two suites in paralell.
In order to solve this, we should generate isolated and temporary resources with randomly created parameters (
account_id
,name
, etc.) to avoid conflicts. We could include a cleanup function that deletes the resources created after the execution.Current behavior
The tests fail because of already existing resources.
Expected behavior
The tests generate isolated and temporary resources for each execution.
The text was updated successfully, but these errors were encountered: