Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix --reparse option in AWS' VPCFlow and Config modules #11608

Merged
merged 1 commit into from Dec 30, 2021
Merged

Fix --reparse option in AWS' VPCFlow and Config modules #11608

merged 1 commit into from Dec 30, 2021

Conversation

noise-kngdm
Copy link
Contributor

Related issue
Closes #11473

Description

In this PR we fix the use of the --reparse parameter of the AWS integration Config and VPCFlow modules.

Tests performed

Test reparse in VPCFlow

First we execute the module using 2021-Dec-09 as only_logs_after to populate the database.

Command output 
root@155babb9fa80:/var/ossec/wodles/aws# /var/ossec/wodles/aws/aws-s3 -b wazuh-aws-wodle-vpcflow -t vpcflow -p dev -d2 -s 2021-dec-09
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Table does not exist; create
DEBUG: +++ Working on XXXXXXXXXXXX - us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/09
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/10
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/11
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/12
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/13
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/14
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/15
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/16
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/17
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/18
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/19
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/20
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/21
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/22
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/23
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/23/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211223T0000Z_ce6176d8.log.gz
DEBUG: +++ DB Maintenance
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/24
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/24/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211240T0000Z_ce6176d8.log.gz
DEBUG: +++ DB Maintenance
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/25
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/26
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/27
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/28
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/29
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/30
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/30/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211230T0000Z_ce6176d8.log.gz
DEBUG: +++ DB Maintenance

Then we execute the same command using the --reparse parameter to check that the fix is working as it should.

Command output 
root@155babb9fa80:/var/ossec/wodles/aws# /var/ossec/wodles/aws/aws-s3 -b wazuh-aws-wodle-vpcflow -t vpcflow -p dev -d2 -s 2021-dec-09 --reparse
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Working on XXXXXXXXXXXX - us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/09
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/10
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/11
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/12
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/13
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/14
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/15
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/16
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/17
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/18
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/19
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/20
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/21
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/22
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/23
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/23/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211223T0000Z_ce6176d8.log.gz
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/23/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211223T0000Z_ce6176d8.log.gz
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/23/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211223T0000Z_ce6176d8.log.gz
DEBUG: +++ DB Maintenance
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/24
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/24/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211240T0000Z_ce6176d8.log.gz
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/24/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211240T0000Z_ce6176d8.log.gz
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/24/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211240T0000Z_ce6176d8.log.gz
DEBUG: +++ DB Maintenance
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/25
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/26
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/27
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/28
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/29
DEBUG: +++ No logs to process for fl-03fbf155728a27ded flow log ID in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/30
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/30/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211230T0000Z_ce6176d8.log.gz
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/30/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211230T0000Z_ce6176d8.log.gz
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/30/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211230T0000Z_ce6176d8.log.gz
DEBUG: +++ DB Maintenance

Lastly, we will execute the module using the --reparse option and without specifying an only_logs_after parameter. It should only fetch the logs of the date of execution.

Command output 
root@155babb9fa80:/var/ossec/wodles/aws# /var/ossec/wodles/aws/aws-s3 -b wazuh-aws-wodle-vpcflow -t vpcflow -p dev -d2 --reparse
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Working on XXXXXXXXXXXX - us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/30
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/30/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211230T0000Z_ce6176d8.log.gz
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/30/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211230T0000Z_ce6176d8.log.gz
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/XXXXXXXXXXXX/vpcflowlogs/us-east-1/2021/12/30/XXXXXXXXXXXX_vpcflowlogs_us-east-1_fl-03fbf155728a27ded_20211230T0000Z_ce6176d8.log.gz
DEBUG: +++ DB Maintenance

Conclusions

The --reparse option is working correctly.

Test reparse in Config

First we execute the module using 2021-Dec-09 as only_logs_after to populate the database.

Command output 
root@155babb9fa80:/var/ossec/wodles/aws# /var/ossec/wodles/aws/aws-s3 -b wazuh-aws-wodle-config -t config -p dev -d2  -s 2021-dec-09
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Table does not exist; create
DEBUG: +++ Working on XXXXXXXXXXXX - us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/9
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/10
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/11
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/12
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/13
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/14
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/15
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/16
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/17
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/18
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/19
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/20
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/21
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/22
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/23
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/23/XXXXXXXXXXXX_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20211223T004303Z_20211223T025123Z_1.json.gz
DEBUG: +++ DB Maintenance
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/24
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/25
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/26
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/27
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/28
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/29
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/30
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/30/XXXXXXXXXXXX_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20211230T004303Z_20211230T025123Z_1.json.gz
DEBUG: +++ DB Maintenance
DEBUG: +++ DB Maintenance

Then we execute the same command using the --reparse parameter to check that the fix is working as it should.

Command output 
root@155babb9fa80:/var/ossec/wodles/aws# /var/ossec/wodles/aws/aws-s3 -b wazuh-aws-wodle-config -t config -p dev -d2  -s 2021-dec-09 --reparse
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Working on XXXXXXXXXXXX - us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/9
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/10
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/11
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/12
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/13
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/14
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/15
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/16
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/17
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/18
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/19
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/20
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/21
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/22
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/23
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/23/XXXXXXXXXXXX_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20211223T004303Z_20211223T025123Z_1.json.gz
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/23/XXXXXXXXXXXX_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20211223T004303Z_20211223T025123Z_1.json.gz
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/23/XXXXXXXXXXXX_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20211223T004303Z_20211223T025123Z_1.json.gz
DEBUG: +++ DB Maintenance
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/24
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/25
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/26
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/27
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/28
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/29
DEBUG: +++ No logs to process in bucket: XXXXXXXXXXXX/us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/30
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/30/XXXXXXXXXXXX_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20211230T004303Z_20211230T025123Z_1.json.gz
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/30/XXXXXXXXXXXX_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20211230T004303Z_20211230T025123Z_1.json.gz
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/30/XXXXXXXXXXXX_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20211230T004303Z_20211230T025123Z_1.json.gz
DEBUG: +++ DB Maintenance
DEBUG: +++ DB Maintenance

Lastly, we will execute the module using the --reparse option and without specifying an only_logs_after parameter. It should only fetch the logs of the date of execution.

Command output 
root@155babb9fa80:/var/ossec/wodles/aws# /var/ossec/wodles/aws/aws-s3 -b wazuh-aws-wodle-config -t config -p dev -d2 --reparse
DEBUG: +++ Debug mode on - Level: 2
DEBUG: +++ Working on XXXXXXXXXXXX - us-east-1
DEBUG: +++ Marker: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/30
DEBUG: ++ File previously processed, but reparse flag set: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/30/XXXXXXXXXXXX_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20211230T004303Z_20211230T025123Z_1.json.gz
DEBUG: ++ Found new log: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/30/XXXXXXXXXXXX_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20211230T004303Z_20211230T025123Z_1.json.gz
DEBUG: +++ File already marked complete, but reparse flag set: AWSLogs/XXXXXXXXXXXX/Config/us-east-1/2021/12/30/XXXXXXXXXXXX_Config_us-east-1_ConfigHistory_AWS::Config::ResourceCompliance_20211230T004303Z_20211230T025123Z_1.json.gz
DEBUG: +++ DB Maintenance
DEBUG: +++ DB Maintenance

Conclusions

The --reparse option is working correctly.

@noise-kngdm noise-kngdm added module/aws module/cloud monitoring Monitoring external services (AWS, Azure, GCP, O365...) labels Dec 30, 2021
@noise-kngdm noise-kngdm added this to the Release 4.3.0 RC 2 milestone Dec 30, 2021
@noise-kngdm noise-kngdm self-assigned this Dec 30, 2021
@noise-kngdm noise-kngdm linked an issue Dec 30, 2021 that may be closed by this pull request
Fix the --reparse option in the AWS VPCFlow and Config integrations #11473
Closed
4 tasks
@davidjiglesias davidjiglesias merged commit 5412ef3 into 4.3 Dec 30, 2021
@davidjiglesias davidjiglesias deleted the fix/11473-aws-vpcflow-reparse branch December 30, 2021 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@davidjiglesias davidjiglesias davidjiglesias approved these changes

Assignees

@noise-kngdm noise-kngdm

Labels
module/aws module/cloud monitoring Monitoring external services (AWS, Azure, GCP, O365...)
Projects
None yet
Milestone
Release 4.3.0 RC 2
Development

Successfully merging this pull request may close these issues.

Fix the --reparse option in the AWS VPCFlow and Config integrations
2 participants
@noise-kngdm @davidjiglesias