Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix WAF log parsing when there are multiple dicts in one line #9775

Merged
merged 1 commit into from
Sep 3, 2021

Conversation

Selutario
Copy link
Contributor

Related issue
Closes #9526

Description

Due to the way the WAF integration is currently designed, in case there was more than one dictionary in a single line (stored in the S3 bucket from which they have to be extracted), an error would occur when using json.loads() since that format is invalid:

wazuh/wodles/aws/aws_s3.py

Lines 2079 to 2092 in af255be

def load_information_from_file(self, log_key):
"""Load data from a WAF log file."""
content = []
with self.decompress_file(log_key=log_key) as f:
for line in f.readlines():
try:
event = json.loads(line.rstrip())
except json.JSONDecodeError:
print("ERROR: Events from {} file could not be loaded.".format(log_key.split('/')[-1]))
sys.exit(9)
event['source'] = 'waf'
content.append(event)
return json.loads(json.dumps(content))

This PR updates said part of the code so that, using raw_decoder, each line is divided into several dictionaries (if there are more than one per line). Therefore, now the following content would be valid, unlike what happened previously:

{"dict_1_key": "value", "dict_1_key2": "value"}
{"dict_2_key": "value", "dict_2_key2": "value"}{"dict_3_key": "value", "dict_3_key2": "value"}{"dict_4_key": "value", "dict_4_key2": "value"}
{"dict_5_key": "value", "dict_5_key2": "value"}
{"dict_6_key": "value", "dict_6_key2": "value"}

This change does not break previous functionality, it simply increases the flexibility of the WAF integration.

WAF log files

Part of the issue requested reviewing the WAF logs that we used for our developments. After doing so, I have been able to verify that they are not correct since they do not correspond to the information that said tool throws up. Here is an example of the logs we had:

{"ticker_symbol":"AZL","sector":"HEALTHCARE","change":-0.2,"price":17.47}{"ticker_symbol":"WAS","sector":"RETAIL","change":-0.43,"price":12.08}{"ticker_symbol":"KFU","sector":"ENERGY","change":3.29,"price":45.26}

To speed up the process, I have decided to add the logs that appear in the WAF documentation instead of generating our own:

{
    "timestamp":1592357192516
    ,"formatVersion":1
    ,"webaclId":"arn:aws:wafv2:us-east-1:123456789012:global/webacl/hello-world/5933d6d9-9dde-js82-v8aw-9ck28nv9"
    ,"terminatingRuleId":"Default_Action"
    ,"terminatingRuleType":"REGULAR"
    ,"action":"ALLOW"
    ,"terminatingRuleMatchDetails":[]
    ,"httpSourceName":"-"
    ,"httpSourceId":"-"
    ,"ruleGroupList":[]
    ,"rateBasedRuleList":[]
    ,"nonTerminatingMatchingRules":
    [{
        "ruleId":"TestRule"
        ,"action":"COUNT"
        ,"ruleMatchDetails":
        [{
            "conditionType":"SQL_INJECTION"
            ,"location":"HEADER"
            ,"matchedData":[
                "10"
                ,"and"
                ,"1"]
            }]
    }]
    ,"httpRequest":{
        "clientIp":"3.3.3.3"
        ,"country":"US"
        ,"headers":[
            {"name":"Host","value":"localhost:1989"}
            ,{"name":"User-Agent","value":"curl/7.61.1"}
            ,{"name":"Accept","value":"*/*"}
            ,{"name":"foo","value":"10 AND 1=1"}
            ]
            ,"uri":"/foo","args":""
            ,"httpVersion":"HTTP/1.1"
            ,"httpMethod":"GET"
            ,"requestId":"rid"
    },
    "labels": [
        {
            "name": "value"
        }
    ]
}

Regards.

@Selutario Selutario self-assigned this Aug 20, 2021
@Selutario Selutario linked an issue Aug 30, 2021 that may be closed by this pull request
3 tasks
@davidjiglesias davidjiglesias merged commit 9dbbe4f into 4.2 Sep 3, 2021
@davidjiglesias davidjiglesias deleted the fix/9526-waf-multiple-dicts-per-line branch September 3, 2021 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate and fix possible AWS WAF integration bug
3 participants