Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor default_config method to take into account user config found in .aws/config file #16531

Merged
merged 9 commits into from Jun 29, 2023

Conversation

EduLeon12
Copy link
Contributor

@EduLeon12 EduLeon12 commented Mar 29, 2023

Related issue
Closes #14978

Description

The default_config was not taken into consideration that the .aws/config file could not have the necessary retries parameters to avoid a throttling error a solution was made to not only look for the retry parameter but also all primary botocore parameters and set them into a botocore Config object.

More information on the solution is described here:

Logs/Alerts example

Now after running the module it will always check for the retries parameters on the config file if the file exists:

  • No Retry Parameters
root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 -b wazuh-aws-wodle-kms -t custom -p dev -d2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: No retries configuration found in profile config generating default configuration for retries: mode standard - max_attempts 10
DEBUG: Created Config object using profile: dev parameters
DEBUG: +++ Marker: 2023/03/29
  • With retry parameters
root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 -b wazuh-aws-wodle-kms -t custom -p dev -d2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: Created Config object using profile: dev parameters
DEBUG: +++ Marker: 2023/03/29
DEBUG: +++ DB Maintenance

Tests

  • Latest Test results:
Test name Pass XPass Skip XFail Fail Status
Wodles UTs 277 0 0 0 1 🟢
AWS ITs 191 0 3 1 0 🟢
ASL Manual execution - - - - - 🟢

Some scenarios were tested to ensure the module was functioning properly
Ex:

  • No regions matching
root@wazuh-master:/# /var/ossec/wodles/aws/aws-s3 -b wazuh-aws-wodle-kms -t custom -r us-west-1 -p dev -d2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: Region passed as parameter does not match region found in dev config file
  • Api unit test:
(ut-3.9-venv) eduardoleon@pop-os:~/git/wazuh$ PYTHONPATH=/home/eduardoleon/git/wazuh/api:/home/eduardoleon/git/wazuh/framework python3 -m pytest api/api --disable-warnings
========================================== test session starts ===========================================
platform linux -- Python 3.9.9, pytest-7.0.1, pluggy-0.13.1
rootdir: /home/eduardoleon/git/wazuh/api
plugins: asyncio-0.18.1, aiohttp-1.0.4, trio-0.7.0, cov-3.0.0, anyio-3.6.2, metadata-2.0.2, html-3.0.0
asyncio: mode=auto
collected 544 items                                                                                      

api/api/controllers/test/test_active_response_controller.py .                                      [  0%]
api/api/controllers/test/test_agent_controller.py ...........................................      [  8%]
api/api/controllers/test/test_cdb_list_controller.py ......                                        [  9%]
api/api/controllers/test/test_ciscat_controller.py .                                               [  9%]
api/api/controllers/test/test_cluster_controller.py ........................                       [ 13%]
api/api/controllers/test/test_decoder_controller.py .......                                        [ 15%]
api/api/controllers/test/test_default_controller.py .                                              [ 15%]
api/api/controllers/test/test_experimental_controller.py ...............                           [ 18%]
api/api/controllers/test/test_manager_controller.py ..................                             [ 21%]
api/api/controllers/test/test_mitre_controller.py .......                                          [ 22%]
api/api/controllers/test/test_overview_controller.py .                                             [ 22%]
api/api/controllers/test/test_rootcheck_controller.py ....                                         [ 23%]
api/api/controllers/test/test_rule_controller.py ........                                          [ 25%]
api/api/controllers/test/test_sca_controller.py ..                                                 [ 25%]
api/api/controllers/test/test_security_controller.py ............................................. [ 33%]
......                                                                                             [ 34%]
api/api/controllers/test/test_syscheck_controller.py ....                                          [ 35%]
api/api/controllers/test/test_syscollector_controller.py .........                                 [ 37%]
api/api/controllers/test/test_task_controller.py .                                                 [ 37%]
api/api/controllers/test/test_vulnerability_controller.py ....                                     [ 38%]
api/api/models/test/test_model.py ...........................                                      [ 43%]
api/api/test/test_alogging.py ..................                                                   [ 46%]
api/api/test/test_authentication.py ...........                                                    [ 48%]
api/api/test/test_configuration.py ............................................                    [ 56%]
api/api/test/test_encoder.py ...                                                                   [ 56%]
api/api/test/test_middlewares.py ............                                                      [ 59%]
api/api/test/test_uri_parser.py ...                                                                [ 59%]
api/api/test/test_util.py ..............................................                           [ 68%]
api/api/test/test_validator.py ................................................................... [ 80%]
.................................................................................................. [ 98%]
........                                                                                           [100%]

==================================== 544 passed, 15 warnings in 2.23s ====================================

@EduLeon12 EduLeon12 self-assigned this Mar 29, 2023
@EduLeon12 EduLeon12 added type/enhancement New feature or request module/aws module/cloud monitoring Monitoring external services (AWS, Azure, GCP, O365...) labels Mar 29, 2023
@EduLeon12 EduLeon12 linked an issue Mar 30, 2023 that may be closed by this pull request
5 tasks
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
nico-stefani
nico-stefani previously approved these changes Apr 3, 2023
Copy link
Member

@nico-stefani nico-stefani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

nico-stefani
nico-stefani previously approved these changes Apr 20, 2023
nico-stefani
nico-stefani previously approved these changes Apr 26, 2023
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
wodles/aws/aws_s3.py Outdated Show resolved Hide resolved
Copy link
Member

@davidjiglesias davidjiglesias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need Integration tests to validate this change.

@Selutario
Copy link
Member

Selutario commented Jun 27, 2023

@EduLeon12, open a new issue to add Unit tests for these changes. The new issue should be blocked by:

To do

  • Address changes requested in this review.
  • Run at least some of the AWS integration tests of the Wazuh-qa repository (using this PR branch to build Wazuh) and share the output here.

Eduardo Leon and others added 5 commits June 28, 2023 10:00
… config file:

- Refactored method to take user config file parameters.
- Added method to modularize default_config method.
- Added new error code.
- Added Docstring to method.
Co-authored-by: Nico Stefani <nicolas.stefi@wazuh.com>
EduLeon12 and others added 4 commits June 28, 2023 10:00
Co-authored-by: Nico Stefani <nicolas.stefi@wazuh.com>
- Added new Error Code
- Fixed Conditional for dictionaries
- Handled error when profile does not exist
- Added new debug messages
@EduLeon12 EduLeon12 force-pushed the 14978-modify-default-config-in-aws-module branch from 5d5ba75 to 781a0a1 Compare June 28, 2023 13:02
Copy link
Member

@nico-stefani nico-stefani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Selutario Selutario dismissed davidjiglesias’s stale review June 29, 2023 12:39

Changes were applied, Integration tests were discarded, unit tests will be added in a new issue once they are merged (still in development).

@Selutario Selutario merged commit 2ad5be1 into 4.6.0 Jun 29, 2023
49 checks passed
@Selutario Selutario deleted the 14978-modify-default-config-in-aws-module branch June 29, 2023 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module/aws module/cloud monitoring Monitoring external services (AWS, Azure, GCP, O365...) type/enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Review and modify default_config method in AWS module
4 participants