Reduce number of calls to db_maintenance in AWS integration
#11828
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
noise-kngdm
added
module/aws
module/cloud monitoring
noise-kngdm
force-pushed
the
fix/10850-aws-db-maintenance-calls
branch
from
6df1188
to
4e49458
Compare
CarlosRS9
approved these changes
Feb 2, 2022
davidjiglesias
approved these changes
Feb 9, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
In this PR we optimize the number of calls to the
db_maintenancemethod by using it only once at the end of every execution for every region, instead of once per batch of logs as it was the case.Tests performed
To test that the changes are working fine for every service we decided to make the following test:
After changing the
self.retain_db_recordsattribute from 500 to 5, ensuring that, for every service and type of bucket, once the module has been executed and has collected more than 5 logs only 5 logs are stored in the database. Also, the lineDEBUG: +++ DB Maintenanceshould appear only once per region and execution. The results of the tests are the following:Conclusions
The changes are working as expected for all the services, however, we discovered a bug in the load balancer buckets, so we opened #11826 to fix that.
Cloudtrail
Command output
After processing 14 files, the number of entries stored in the database is:
VPCFlow
Command output
After processing 6 files, the number of entries stored in the database is:
Config
Command output
After processing 9 files, the number of entries stored in the database is:
ALB, NLB and CLB
ALB command output
After processing 45 files, the number of entries stored in the database is:
After examining why the db maintenance is not working, we realized that there is a problem with how the
db_maintenancemethod works that affects theALB,CLB, andNLBbuckets. The issue #11826 was opened to address that.KMS
Command output
We piped the output of the command to `grep -v Skipping` to avoid an unnecessary amount of `DEBUG: ++ Skipping file with another prefix:` lines in the output.After processing 6 files, the number of entries stored in the database is:
Macie
Command output
After processing 6 files, the number of entries stored in the database is:
Trusted Advisor
Command output
After processing 6 files, the number of entries stored in the database is:
GuardDuty
Command output
After processing 6 files, the number of entries stored in the database is:
WAF
Command output
After processing 7 files, the number of entries stored in the database is:
Umbrella
Command output
After processing 6 files, the number of entries stored in the database is:
Inspector
Since we only had two example executions we could use to test the behavior of the
db_maintenancemethod, we decided to change the value ofretain_db_recordsto 1 instead of 5.Command output
After processing 2 executions, the number of entries stored in the database is: