Improve VPCFlow and Config items iteration #16325
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nico-stefani
force-pushed
the
feature/16138-improve-vpc-config-items-iteration
branch
from
b456785
to
6f71d17
Compare
6 tasks
nico-stefani
force-pushed
the
feature/16138-improve-vpc-config-items-iteration
branch
from
6f71d17
to
3a59747
Compare
|
Because
root@9ab624e54776:/var/ossec# wodles/aws/aws-s3 --bucket wazuh-vpcflow-integration-tests --aws_profile qa --only_logs_after 2022-NOV-20 --type vpcflow --debug 2 --reparse
DEBUG: +++ Debug mode on - Level: 2
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: +++ Working on 819751203818 - us-east-1
DEBUG: +++ Marker: AWSLogs/819751203818/vpcflowlogs/us-east-1/2022/11/20
DEBUG: ++ Reparse mode enabled
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/819751203818/vpcflowlogs/us-east-1/2022/11/21/819751203818_vpcflowlogs_us-east-1_fl-0754d951c16f517fa_20221121T1329Z_4594161841401432054.log
DEBUG: ++ Found new log: AWSLogs/819751203818/vpcflowlogs/us-east-1/2022/11/21/819751203818_vpcflowlogs_us-east-1_fl-0754d951c16f517fa_20221121T1329Z_4594161841401432054.log
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/819751203818/vpcflowlogs/us-east-1/2022/11/21/819751203818_vpcflowlogs_us-east-1_fl-0754d951c16f517fa_20221121T1329Z_4594161841401432054.log
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/819751203818/vpcflowlogs/us-east-1/2022/11/23/819751203818_vpcflowlogs_us-east-1_fl-0754d951c16f517fa_20221123T1329Z_4594161841401432054.log
DEBUG: ++ Found new log: AWSLogs/819751203818/vpcflowlogs/us-east-1/2022/11/23/819751203818_vpcflowlogs_us-east-1_fl-0754d951c16f517fa_20221123T1329Z_4594161841401432054.log
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/819751203818/vpcflowlogs/us-east-1/2022/11/23/819751203818_vpcflowlogs_us-east-1_fl-0754d951c16f517fa_20221123T1329Z_4594161841401432054.log
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/819751203818/vpcflowlogs/us-east-1/2022/11/26/819751203818_vpcflowlogs_us-east-1_fl-0754d951c16f517fa_20221126T1329Z_4594161841401432054.log
DEBUG: ++ Found new log: AWSLogs/819751203818/vpcflowlogs/us-east-1/2022/11/26/819751203818_vpcflowlogs_us-east-1_fl-0754d951c16f517fa_20221126T1329Z_4594161841401432054.log
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/819751203818/vpcflowlogs/us-east-1/2022/11/26/819751203818_vpcflowlogs_us-east-1_fl-0754d951c16f517fa_20221126T1329Z_4594161841401432054.log
DEBUG: +++ DB Maintenance
DEBUG: +++ Working on 819751203818 - us-east-2
DEBUG: +++ Marker: AWSLogs/819751203818/vpcflowlogs/us-east-2/2022/11/20
DEBUG: ++ Reparse mode enabled
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/819751203818/vpcflowlogs/us-east-2/2022/11/21/819751203818_vpcflowlogs_us-east-2_fl-01462ab0a6f5abdcb_20221121T1329Z_4594161841401432054.log
DEBUG: ++ Found new log: AWSLogs/819751203818/vpcflowlogs/us-east-2/2022/11/21/819751203818_vpcflowlogs_us-east-2_fl-01462ab0a6f5abdcb_20221121T1329Z_4594161841401432054.log
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/819751203818/vpcflowlogs/us-east-2/2022/11/21/819751203818_vpcflowlogs_us-east-2_fl-01462ab0a6f5abdcb_20221121T1329Z_4594161841401432054.log
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/819751203818/vpcflowlogs/us-east-2/2022/11/26/819751203818_vpcflowlogs_us-east-2_fl-01462ab0a6f5abdcb_20221126T1329Z_4594161841401432054.log
DEBUG: ++ Found new log: AWSLogs/819751203818/vpcflowlogs/us-east-2/2022/11/26/819751203818_vpcflowlogs_us-east-2_fl-01462ab0a6f5abdcb_20221126T1329Z_4594161841401432054.log
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/819751203818/vpcflowlogs/us-east-2/2022/11/26/819751203818_vpcflowlogs_us-east-2_fl-01462ab0a6f5abdcb_20221126T1329Z_4594161841401432054.log
DEBUG: +++ DB Maintenance
root@9ab624e54776:/var/ossec# wodles/aws/aws-s3 --bucket wazuh-config-integration-tests --aws_profile qa --only_logs_after 2022-NOV-20 --type config --debug 2 --reparse
DEBUG: +++ Debug mode on - Level: 2
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: +++ Working on 819751203818 - us-east-1
DEBUG: +++ Marker: AWSLogs/819751203818/Config/us-east-1/2022/11/20
DEBUG: ++ Reparse mode enabled
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/819751203818/Config/us-east-1/2022/11/21/819751203818_Config_us-east-1_ConfigHistory_AWS_20221121T1419Z_9135633071561314633.json
DEBUG: ++ Found new log: AWSLogs/819751203818/Config/us-east-1/2022/11/21/819751203818_Config_us-east-1_ConfigHistory_AWS_20221121T1419Z_9135633071561314633.json
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/819751203818/Config/us-east-1/2022/11/23/819751203818_Config_us-east-1_ConfigHistory_AWS_20221123T1419Z_9135633071561314633.json
DEBUG: ++ Found new log: AWSLogs/819751203818/Config/us-east-1/2022/11/23/819751203818_Config_us-east-1_ConfigHistory_AWS_20221123T1419Z_9135633071561314633.json
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/819751203818/Config/us-east-1/2022/11/26/819751203818_Config_us-east-1_ConfigHistory_AWS_20221126T1419Z_9135633071561314633.json
DEBUG: ++ Found new log: AWSLogs/819751203818/Config/us-east-1/2022/11/26/819751203818_Config_us-east-1_ConfigHistory_AWS_20221126T1419Z_9135633071561314633.json
DEBUG: +++ DB Maintenance
DEBUG: +++ Working on 819751203818 - us-east-2
DEBUG: +++ Marker: AWSLogs/819751203818/Config/us-east-2/2022/11/20
DEBUG: ++ Reparse mode enabled
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/819751203818/Config/us-east-2/2022/11/21/819751203818_Config_us-east-2_ConfigHistory_AWS_20221121T1419Z_9135633071561314633.json
DEBUG: ++ Found new log: AWSLogs/819751203818/Config/us-east-2/2022/11/21/819751203818_Config_us-east-2_ConfigHistory_AWS_20221121T1419Z_9135633071561314633.json
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/819751203818/Config/us-east-2/2022/11/26/819751203818_Config_us-east-2_ConfigHistory_AWS_20221126T1419Z_9135633071561314633.json
DEBUG: ++ Found new log: AWSLogs/819751203818/Config/us-east-2/2022/11/26/819751203818_Config_us-east-2_ConfigHistory_AWS_20221126T1419Z_9135633071561314633.json
DEBUG: +++ DB Maintenance |
fdalmaup
requested changes
Mar 6, 2023
nico-stefani
force-pushed
the
feature/16138-improve-vpc-config-items-iteration
branch
from
867b3b8
to
2ca5e85
Compare
fdalmaup
requested changes
Mar 13, 2023
nico-stefani
force-pushed
the
feature/16138-improve-vpc-config-items-iteration
branch
from
2aa83cb
to
2a0bbe1
Compare
nico-stefani
force-pushed
the
feature/16138-improve-vpc-config-items-iteration
branch
from
e51466b
to
aa02469
Compare
nico-stefani
force-pushed
the
feature/16138-improve-vpc-config-items-iteration
branch
from
aa02469
to
ace0d20
Compare
* Handle exception when an inexistent region was provided * Get regions from constant
nico-stefani
force-pushed
the
feature/16138-improve-vpc-config-items-iteration
branch
from
ace0d20
to
4e8d28b
Compare
davidjiglesias
deleted the
feature/16138-improve-vpc-config-items-iteration
branch
nico-stefani
added a commit
that referenced
this pull request
Apr 18, 2023
* Implement basic changes * Use methods inherited from AWSBucket * Improve items iteration AWSVPCFlowBucket * Use methods inherited from AWSBucket in AWSVPCFlowBucket * Clean up unused methods and variables * Fix too many blank lines * Handle exception when an inexistent region was provided (#16332) * Handle exception when an inexistent region was provided * Get regions from constant
nico-stefani
added a commit
that referenced
this pull request
Apr 26, 2023
* Implement basic changes * Use methods inherited from AWSBucket * Improve items iteration AWSVPCFlowBucket * Use methods inherited from AWSBucket in AWSVPCFlowBucket * Clean up unused methods and variables * Fix too many blank lines * Handle exception when an inexistent region was provided (#16332) * Handle exception when an inexistent region was provided * Get regions from constant
nico-stefani
added a commit
that referenced
this pull request
May 9, 2023
* Implement basic changes * Use methods inherited from AWSBucket * Improve items iteration AWSVPCFlowBucket * Use methods inherited from AWSBucket in AWSVPCFlowBucket * Clean up unused methods and variables * Fix too many blank lines * Handle exception when an inexistent region was provided (#16332) * Handle exception when an inexistent region was provided * Get regions from constant
davidjiglesias
pushed a commit
that referenced
this pull request
May 10, 2023
* Validated the region passed before instantiating the service class (#16463) * Validated the region passed before instantiating the service class * Apply suggestions from code review Co-authored-by: Facundo Dalmau <facundo.dalmau@wazuh.com> * Exit with error when receive and invalid region --------- Co-authored-by: Facundo Dalmau <facundo.dalmau@wazuh.com> * Show explicit messages when there aren't logs to process (#16365) * Improve filter aguments for custom buckets * Improve check for empty bucket in server access * Fix custom bucket markers (#16410) * Fix query paramters for CustomBucket.sql_find_last_key_processed * Show message when there aren't logs to process in custom buckets * Apply improvements to ServerAccessBucket.iter_files_in_bucket * Add missing counter for processed_logs * Apply suggestions from code review Co-authored-by: Facundo Dalmau <facundo.dalmau@wazuh.com> * Fix method callbacks --------- Co-authored-by: Facundo Dalmau <facundo.dalmau@wazuh.com> * Improve VPCFlow and Config items iteration (#16325) * Implement basic changes * Use methods inherited from AWSBucket * Improve items iteration AWSVPCFlowBucket * Use methods inherited from AWSBucket in AWSVPCFlowBucket * Clean up unused methods and variables * Fix too many blank lines * Handle exception when an inexistent region was provided (#16332) * Handle exception when an inexistent region was provided * Get regions from constant * Use AWSBucket.empty_bucket_message_template in AWSLBBucket class * Add AWS parser validations (#16493) * Fix bucket and service empty messages * Fix bucket and service invalid value messages * Improve regex validation for bucket name * Improve regex validation for prefix and rename function * Improved regex validation for region and avoided repeated ones * Show error and exit for empty log group * Added function to validate aws_log_groups argument * Sorted regions after validation * Use AWSBucket.empty_bucket_message_template native guarduty case --------- Co-authored-by: Facundo Dalmau <facundo.dalmau@wazuh.com>
mhamra
pushed a commit
that referenced
this pull request
May 24, 2023
* Validated the region passed before instantiating the service class (#16463) * Validated the region passed before instantiating the service class * Apply suggestions from code review Co-authored-by: Facundo Dalmau <facundo.dalmau@wazuh.com> * Exit with error when receive and invalid region --------- Co-authored-by: Facundo Dalmau <facundo.dalmau@wazuh.com> * Show explicit messages when there aren't logs to process (#16365) * Improve filter aguments for custom buckets * Improve check for empty bucket in server access * Fix custom bucket markers (#16410) * Fix query paramters for CustomBucket.sql_find_last_key_processed * Show message when there aren't logs to process in custom buckets * Apply improvements to ServerAccessBucket.iter_files_in_bucket * Add missing counter for processed_logs * Apply suggestions from code review Co-authored-by: Facundo Dalmau <facundo.dalmau@wazuh.com> * Fix method callbacks --------- Co-authored-by: Facundo Dalmau <facundo.dalmau@wazuh.com> * Improve VPCFlow and Config items iteration (#16325) * Implement basic changes * Use methods inherited from AWSBucket * Improve items iteration AWSVPCFlowBucket * Use methods inherited from AWSBucket in AWSVPCFlowBucket * Clean up unused methods and variables * Fix too many blank lines * Handle exception when an inexistent region was provided (#16332) * Handle exception when an inexistent region was provided * Get regions from constant * Use AWSBucket.empty_bucket_message_template in AWSLBBucket class * Add AWS parser validations (#16493) * Fix bucket and service empty messages * Fix bucket and service invalid value messages * Improve regex validation for bucket name * Improve regex validation for prefix and rename function * Improved regex validation for region and avoided repeated ones * Show error and exit for empty log group * Added function to validate aws_log_groups argument * Sorted regions after validation * Use AWSBucket.empty_bucket_message_template native guarduty case --------- Co-authored-by: Facundo Dalmau <facundo.dalmau@wazuh.com>
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR closes #16138. It improves item iteration for
VPCFlowandConfigintegrations. Now the module only scrolls over existent items in the bucket.Before the changes we got:
VPC output
Config output
And now:
VPCFlow output
Config output
Tests
Unit Tests
Integration Tests
Tier 1
Tier 0
The failed test is related to #15763