Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add geoip for AWS ALB client IP addresses #16198

Closed
4 tasks done
davidjiglesias opened this issue Feb 14, 2023 · 4 comments · Fixed by #15582
Closed
4 tasks done

Add geoip for AWS ALB client IP addresses #16198

davidjiglesias opened this issue Feb 14, 2023 · 4 comments · Fixed by #15582

Comments

@davidjiglesias
Copy link
Member

davidjiglesias commented Feb 14, 2023

We have a Pull request from @rh0dy incorporating this feature in #15582.

Description

As part of release 4.4, the AWS S3 Wodle will now extract IP addresses from AWS ALB logs, as per #13095 and #14525. The IP addresses are extracted to the following fields: data.aws.target_ip and data.aws.source_ip. This PR is to preprocess those fields so that they are geolocated by Filebeat (to add country, latitude, longitude, etc.) before being stored in Elasticsearch.

Checks

The following elements have been updated or reviewed (should also be checked if no modification is required):

  • Tests (unit tests, API integration tests).
  • Changelog.
  • Documentation.
  • Integration test mapping (using api/test/integration/mapping/_test_mapping.py).
@nico-stefani
Copy link
Member

nico-stefani commented Feb 23, 2023

Issue Update

I was trying the proposed changes without success.

I took the modified pipeline.json files and replace the current files in the stack of 4.3.10 (/usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json, /usr/share/filebeat/module/wazuh/archives/ingest/pipeline.json).

Then, I restarted the container and ran the module.

root@wazuh:/# /var/ossec/wodles/aws/aws-s3 -b wazuh-aws-wodle-alb -t alb -s 2021-Dec-21 -p dev -d2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: +++ Table does not exist; create
DEBUG: +++ Working on xxxx - us-west-1
DEBUG: +++ Marker: AWSLogs/xxxx/elasticloadbalancing/us-west-1/2021/12/21
DEBUG: ++ Found new log: AWSLogs/xxxx/elasticloadbalancing/us-west-1/2021/12/21/xxxx_elasticloadbalancing_us-west-1_app.ALB-framework-dev.959dfdbaed241613_20211221T0000Z_52.52.208.49_14pczeay.log.gz
DEBUG: ++ Found new log: AWSLogs/xxxx/elasticloadbalancing/us-west-1/2021/12/22/xxxx_elasticloadbalancing_us-west-1_app.ALB-framework-dev.959dfdbaed241613_20211222T0000Z_52.52.208.49_14pczeay.log.gz
DEBUG: ++ Found new log: AWSLogs/xxxx/elasticloadbalancing/us-west-1/2021/12/23/xxxx_elasticloadbalancing_us-west-1_app.ALB-framework-dev.959dfdbaed241613_20211223T0000Z_52.52.208.49_14pczeay.log.gz
DEBUG: ++ Found new log: AWSLogs/xxxx/elasticloadbalancing/us-west-1/2022/08/16/xxxx_elasticloadbalancing_us-west-1_app.ALB-framework-dev.959dfdbaed241613_20211221T0000Z_52.52.208.49_14pczeay.log
DEBUG: ++ Found new log: AWSLogs/xxxx/elasticloadbalancing/us-west-1/2022/08/16/xxxx_elasticloadbalancing_us-west-1_app.ALB-framework-dev.959dfdbaed241613_20211221T0000Z_52.52.208.49_14pczeay.log.gz
DEBUG: +++ DB Maintenance

After that, I checked the ingested alerts in the indexer and any of them hadn't the desired fields.

curl -XGET -k -u admin:xxx 'https://localhost:9200/wazuh-alerts-4.x-2023.02.22/_search?pretty'

Ingested alerts
{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 3,
    "successful": 3,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 28,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "wazuh-alerts-4.x-2023.02.22",
        "_type": "_doc",
        "_id": "k6_veYYBo5fkBbkD6OPH",
        "_score": 1.0,
        "_source": {
          "agent": {
            "name": "wazuh.manager",
            "id": "000"
          },
          "manager": {
            "name": "wazuh.manager"
          },
          {
            "_index": "wazuh-alerts-4.x-2023.02.22",
            "_type": "_doc",
            "_id": "DK_MeoYBo5fkBbkDLOQd",
            "_score": 1.0,
            "_source": {
              "agent": {
                "name": "wazuh.manager",
                "id": "000"
              },
              "manager": {
                "name": "wazuh.manager"
              },
              "data": {
                "integration": "aws",
                "aws": {
                  "received_bytes": "136",
                  "request": "GET http://52.52.208.49:80/ HTTP/1.1",
                  "target_ip_list": "10.0.0.125 10.0.0.126",
                  "target_status_code_list": "403",
                  "target_port_list": "80 81",
                  "target_processing_time": "0.001",
                  "log_info": {
                    "s3bucket": "wazuh-aws-wodle-alb",
                    "log_file": "AWSLogs/xx/elasticloadbalancing/us-west-1/2022/08/16/xx_elasticloadbalancing_us-west-1_app.ALB-framework-dev.959dfdbaed241613_20211221T0000Z_52.52.208.49_14pczeay.log"
                  },
                  "ssl_cipher": "-",
                  "source": "alb",
                  "type": "http",
                  "sent_bytes": "5173",
                  "client_port": "51444",
                  "target_port": "80",
                  "domain_name": "-",
                  "error_reason": "-",
                  "classification_reason": "-",
                  "elb": "app/ALB-framework-dev/959dfdbaed241613",
                  "client_ip": "209.17.97.74",
                  "user_agent": "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)",
                  "ssl_protocol": "-",
                  "target_group_arn": "arn:aws:elasticloadbalancing:us-west-1:xx:targetgroup/EC2/a7985a8385b86dc0",
                  "trace_id": "Root=1-5fbc4c52-5a3a21203a0b9d20551c0535",
                  "matched_rule_priority": "0",
                  "elb_status_code": "403",
                  "classification": "-",
                  "request_processing_time": "0.001",
                  "chosen_cert_arn": "-",
                  "response_processing_time": "0.000",
                  "target_status_code": "403",
                  "target_ip": "10.0.0.125",
                  "request_creation_time": "2020-11-23T23:57:06.778000Z",
                  "time": "2020-11-23T23:57:06.780380Z",
                  "redirect_url": "-",
                  "action_executed": "forward"
                }
              },
              "rule": {
                "firedtimes": 4,
                "mail": false,
                "level": 5,
                "description": "AWS ALB: Status error:  - forward - Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com) [ELB: app/ALB-framework-dev/959dfdbaed241613].",
                "groups": [
                  "amazon",
                  "aws",
                  "aws_alb"
                ],
                "id": "80328"
              },
              "decoder": {
                "name": "json"
              },
              "input": {
                "type": "log"
              },
              "@timestamp": "2023-02-22T20:24:04.672Z",
              "location": "Wazuh-AWS",
              "id": "1677097444.24519",
              "timestamp": "2023-02-22T20:24:04.672+0000"
            }
          },
          {
            "_index": "wazuh-alerts-4.x-2023.02.22",
            "_type": "_doc",
            "_id": "FK_WeoYBo5fkBbkDA-TU",
            "_score": 1.0,
            "_source": {
              "agent": {
                "name": "wazuh.manager",
                "id": "000"
              },
              "manager": {
                "name": "wazuh.manager"
              },
              "data": {
                "integration": "aws",
                "aws": {
                  "received_bytes": "136",
                  "request": "GET http://52.52.208.49:80/ HTTP/1.1",
                  "target_ip_list": "10.0.0.125",
                  "target_status_code_list": "403",
                  "target_port_list": "80",
                  "target_processing_time": "0.001",
                  "log_info": {
                    "s3bucket": "wazuh-aws-wodle-alb",
                    "log_file": "AWSLogs/xx/elasticloadbalancing/us-west-1/2021/12/21/xx_elasticloadbalancing_us-west-1_app.ALB-framework-dev.959dfdbaed241613_20211221T0000Z_52.52.208.49_14pczeay.log.gz"
                  },
                  "ssl_cipher": "-",
                  "source": "alb",
                  "type": "http",
                  "sent_bytes": "5173",
                  "client_port": "51444",
                  "target_port": "80",
                  "domain_name": "-",
                  "error_reason": "-",
                  "classification_reason": "-",
                  "elb": "app/ALB-framework-dev/959dfdbaed241613",
                  "client_ip": "209.17.97.74",
                  "user_agent": "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)",
                  "ssl_protocol": "-",
                  "target_group_arn": "arn:aws:elasticloadbalancing:us-west-1:xx:targetgroup/EC2/a7985a8385b86dc0",
                  "trace_id": "Root=1-5fbc4c52-5a3a21203a0b9d20551c0535",
                  "matched_rule_priority": "0",
                  "elb_status_code": "403",
                  "classification": "-",
                  "request_processing_time": "0.001",
                  "chosen_cert_arn": "-",
                  "response_processing_time": "0.000",
                  "target_status_code": "403",
                  "target_ip": "10.0.0.125",
                  "request_creation_time": "2020-11-23T23:57:06.778000Z",
                  "time": "2020-11-23T23:57:06.780380Z",
                  "redirect_url": "-",
                  "action_executed": "forward"
                }
              },
              "rule": {
                "firedtimes": 1,
                "mail": false,
                "level": 5,
                "description": "AWS ALB: Status error:  - forward - Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com) [ELB: app/ALB-framework-dev/959dfdbaed241613].",
                "groups": [
                  "amazon",
                  "aws",
                  "aws_alb"
                ],
                "id": "80328"
              },
              "decoder": {
                "name": "json"
              },
              "input": {
                "type": "log"
              },
              "@timestamp": "2023-02-22T20:34:48.870Z",
              "location": "Wazuh-AWS",
              "id": "1677098088.30978",
              "timestamp": "2023-02-22T20:34:48.870+0000"
            }
          },
          {
            "_index": "wazuh-alerts-4.x-2023.02.22",
            "_type": "_doc",
            "_id": "1sLdeoYBNzN6mwUfvUvT",
            "_score": 1.0,
            "_source": {
              "agent": {
                "name": "wazuh.manager",
                "id": "000"
              },
              "manager": {
                "name": "wazuh.manager"
              },
              "data": {
                "integration": "aws",
                "aws": {
                  "received_bytes": "136",
                  "request": "GET http://52.52.208.49:80/ HTTP/1.1",
                  "target_ip_list": "10.0.0.125",
                  "target_status_code_list": "403",
                  "target_port_list": "80",
                  "target_processing_time": "0.001",
                  "log_info": {
                    "s3bucket": "wazuh-aws-wodle-alb",
                    "log_file": "AWSLogs/xx/elasticloadbalancing/us-west-1/2021/12/21/xx_elasticloadbalancing_us-west-1_app.ALB-framework-dev.959dfdbaed241613_20211221T0000Z_52.52.208.49_14pczeay.log.gz"
                  },
                  "ssl_cipher": "-",
                  "source": "alb",
                  "type": "http",
                  "sent_bytes": "5173",
                  "client_port": "51444",
                  "target_port": "80",
                  "domain_name": "-",
                  "error_reason": "-",
                  "classification_reason": "-",
                  "elb": "app/ALB-framework-dev/959dfdbaed241613",
                  "client_ip": "209.17.97.74",
                  "user_agent": "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)",
                  "ssl_protocol": "-",
                  "target_group_arn": "arn:aws:elasticloadbalancing:us-west-1:xx:targetgroup/EC2/a7985a8385b86dc0",
                  "trace_id": "Root=1-5fbc4c52-5a3a21203a0b9d20551c0535",
                  "matched_rule_priority": "0",
                  "elb_status_code": "403",
                  "classification": "-",
                  "request_processing_time": "0.001",
                  "chosen_cert_arn": "-",
                  "response_processing_time": "0.000",
                  "target_status_code": "403",
                  "target_ip": "10.0.0.125",
                  "request_creation_time": "2020-11-23T23:57:06.778000Z",
                  "time": "2020-11-23T23:57:06.780380Z",
                  "redirect_url": "-",
                  "action_executed": "forward"
                }
              },
              "rule": {
                "firedtimes": 1,
                "mail": false,
                "level": 5,
                "description": "AWS ALB: Status error:  - forward - Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com) [ELB: app/ALB-framework-dev/959dfdbaed241613].",
                "groups": [
                  "amazon",
                  "aws",
                  "aws_alb"
                ],
                "id": "80328"
              },
              "decoder": {
                "name": "json"
              },
              "input": {
                "type": "log"
              },
              "@timestamp": "2023-02-22T20:43:08.532Z",
              "location": "Wazuh-AWS",
              "id": "1677098588.46728",
              "timestamp": "2023-02-22T20:43:08.532+0000"
            }
          },
          {
            "_index": "wazuh-alerts-4.x-2023.02.22",
            "_type": "_doc",
            "_id": "18LdeoYBNzN6mwUfvUvT",
            "_score": 1.0,
            "_source": {
              "agent": {
                "name": "wazuh.manager",
                "id": "000"
              },
              "manager": {
                "name": "wazuh.manager"
              },
              "data": {
                "integration": "aws",
                "aws": {
                  "received_bytes": "136",
                  "request": "GET http://52.52.208.49:80/ HTTP/1.1",
                  "target_ip_list": "10.0.0.125",
                  "target_status_code_list": "403",
                  "target_port_list": "80",
                  "target_processing_time": "0.001",
                  "log_info": {
                    "s3bucket": "wazuh-aws-wodle-alb",
                    "log_file": "AWSLogs/xx/elasticloadbalancing/us-west-1/2021/12/22/xx_elasticloadbalancing_us-west-1_app.ALB-framework-dev.959dfdbaed241613_20211222T0000Z_52.52.208.49_14pczeay.log.gz"
                  },
                  "ssl_cipher": "-",
                  "source": "alb",
                  "type": "http",
                  "sent_bytes": "5173",
                  "client_port": "51444",
                  "target_port": "80",
                  "domain_name": "-",
                  "error_reason": "-",
                  "classification_reason": "-",
                  "elb": "app/ALB-framework-dev/959dfdbaed241613",
                  "client_ip": "209.17.97.74",
                  "user_agent": "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)",
                  "ssl_protocol": "-",
                  "target_group_arn": "arn:aws:elasticloadbalancing:us-west-1:xx:targetgroup/EC2/a7985a8385b86dc0",
                  "trace_id": "Root=1-5fbc4c52-5a3a21203a0b9d20551c0535",
                  "matched_rule_priority": "0",
                  "elb_status_code": "403",
                  "classification": "-",
                  "request_processing_time": "0.001",
                  "chosen_cert_arn": "-",
                  "response_processing_time": "0.000",
                  "target_status_code": "403",
                  "target_ip": "10.0.0.125",
                  "request_creation_time": "2020-11-23T23:57:06.778000Z",
                  "time": "2020-11-23T23:57:06.780380Z",
                  "redirect_url": "-",
                  "action_executed": "forward"
                }
              },
              "rule": {
                "firedtimes": 2,
                "mail": false,
                "level": 5,
                "description": "AWS ALB: Status error:  - forward - Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com) [ELB: app/ALB-framework-dev/959dfdbaed241613].",
                "groups": [
                  "amazon",
                  "aws",
                  "aws_alb"
                ],
                "id": "80328"
              },
              "decoder": {
                "name": "json"
              },
              "input": {
                "type": "log"
              },
              "@timestamp": "2023-02-22T20:43:08.759Z",
              "location": "Wazuh-AWS",
              "id": "1677098588.49825",
              "timestamp": "2023-02-22T20:43:08.759+0000"
            }
          },
          {
            "_index": "wazuh-alerts-4.x-2023.02.22",
            "_type": "_doc",
            "_id": "4cLneoYBNzN6mwUf0EsG",
            "_score": 1.0,
            "_source": {
              "agent": {
                "name": "wazuh.manager",
                "id": "000"
              },
              "manager": {
                "name": "wazuh.manager"
              },
              "data": {
                "integration": "aws",
                "aws": {
                  "received_bytes": "136",
                  "request": "GET http://52.52.208.49:80/ HTTP/1.1",
                  "target_ip_list": "10.0.0.125",
                  "target_status_code_list": "403",
                  "target_port_list": "80",
                  "target_processing_time": "0.001",
                  "log_info": {
                    "s3bucket": "wazuh-aws-wodle-alb",
                    "log_file": "AWSLogs/xx/elasticloadbalancing/us-west-1/2021/12/22/xx_elasticloadbalancing_us-west-1_app.ALB-framework-dev.959dfdbaed241613_20211222T0000Z_52.52.208.49_14pczeay.log.gz"
                  },
                  "ssl_cipher": "-",
                  "source": "alb",
                  "type": "http",
                  "sent_bytes": "5173",
                  "client_port": "51444",
                  "target_port": "80",
                  "domain_name": "-",
                  "error_reason": "-",
                  "classification_reason": "-",
                  "elb": "app/ALB-framework-dev/959dfdbaed241613",
                  "client_ip": "209.17.97.74",
                  "user_agent": "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)",
                  "ssl_protocol": "-",
                  "target_group_arn": "arn:aws:elasticloadbalancing:us-west-1:xx:targetgroup/EC2/a7985a8385b86dc0",
                  "trace_id": "Root=1-5fbc4c52-5a3a21203a0b9d20551c0535",
                  "matched_rule_priority": "0",
                  "elb_status_code": "403",
                  "classification": "-",
                  "request_processing_time": "0.001",
                  "chosen_cert_arn": "-",
                  "response_processing_time": "0.000",
                  "target_status_code": "403",
                  "target_ip": "10.0.0.125",
                  "request_creation_time": "2020-11-23T23:57:06.778000Z",
                  "time": "2020-11-23T23:57:06.780380Z",
                  "redirect_url": "-",
                  "action_executed": "forward"
                }
              },
              "rule": {
                "firedtimes": 7,
                "mail": false,
                "level": 5,
                "description": "AWS ALB: Status error:  - forward - Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com) [ELB: app/ALB-framework-dev/959dfdbaed241613].",
                "groups": [
                  "amazon",
                  "aws",
                  "aws_alb"
                ],
                "id": "80328"
              },
              "decoder": {
                "name": "json"
              },
              "input": {
                "type": "log"
              },
              "@timestamp": "2023-02-22T20:54:10.492Z",
              "location": "Wazuh-AWS",
              "id": "1677099250.65332",
              "timestamp": "2023-02-22T20:54:10.492+0000"
            }
          },
          {
            "_index": "wazuh-alerts-4.x-2023.02.22",
            "_type": "_doc",
            "_id": "5MLneoYBNzN6mwUf0EsG",
            "_score": 1.0,
            "_source": {
              "agent": {
                "name": "wazuh.manager",
                "id": "000"
              },
              "manager": {
                "name": "wazuh.manager"
              },
              "data": {
                "integration": "aws",
                "aws": {
                  "received_bytes": "136",
                  "request": "GET http://52.52.208.49:80/ HTTP/1.1",
                  "target_ip_list": "10.0.0.125",
                  "target_status_code_list": "403",
                  "target_port_list": "80",
                  "target_processing_time": "0.001",
                  "log_info": {
                    "s3bucket": "wazuh-aws-wodle-alb",
                    "log_file": "AWSLogs/xx/elasticloadbalancing/us-west-1/2022/08/16/xx_elasticloadbalancing_us-west-1_app.ALB-framework-dev.959dfdbaed241613_20211221T0000Z_52.52.208.49_14pczeay.log.gz"
                  },
                  "ssl_cipher": "-",
                  "source": "alb",
                  "type": "http",
                  "sent_bytes": "5173",
                  "client_port": "51444",
                  "target_port": "80",
                  "domain_name": "-",
                  "error_reason": "-",
                  "classification_reason": "-",
                  "elb": "app/ALB-framework-dev/959dfdbaed241613",
                  "client_ip": "209.17.97.74",
                  "user_agent": "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)",
                  "ssl_protocol": "-",
                  "target_group_arn": "arn:aws:elasticloadbalancing:us-west-1:xx:targetgroup/EC2/a7985a8385b86dc0",
                  "trace_id": "Root=1-5fbc4c52-5a3a21203a0b9d20551c0535",
                  "matched_rule_priority": "0",
                  "elb_status_code": "403",
                  "classification": "-",
                  "request_processing_time": "0.001",
                  "chosen_cert_arn": "-",
                  "response_processing_time": "0.000",
                  "target_status_code": "403",
                  "target_ip": "10.0.0.125",
                  "request_creation_time": "2020-11-23T23:57:06.778000Z",
                  "time": "2020-11-23T23:57:06.780380Z",
                  "redirect_url": "-",
                  "action_executed": "forward"
                }
              },
              "rule": {
                "firedtimes": 10,
                "mail": false,
                "level": 5,
                "description": "AWS ALB: Status error:  - forward - Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com) [ELB: app/ALB-framework-dev/959dfdbaed241613].",
                "groups": [
                  "amazon",
                  "aws",
                  "aws_alb"
                ],
                "id": "80328"
              },
              "decoder": {
                "name": "json"
              },
              "input": {
                "type": "log"
              },
              "@timestamp": "2023-02-22T20:54:11.347Z",
              "location": "Wazuh-AWS",
              "id": "1677099251.74645",
              "timestamp": "2023-02-22T20:54:11.347+0000"
            }
          }
        ]
      }
    }

@rh0dy
Copy link
Contributor

rh0dy commented Feb 23, 2023

@nico-stefani does your stack include #14525? It's required to generate the desired fields to geolocate.

@nico-stefani
Copy link
Member

Yes, @rh0dy I included these changes to make the tests. You can see in the ingested alerts the field client_ip, but any wasn't transformed to GeoLocation.

@nico-stefani
Copy link
Member

Issue Update

After reindexing the pipelines with filebeat setup --pipelines, the geolocation is working

image

image

@davidjiglesias davidjiglesias changed the title Add geoip for AWS ALB client and target IP addresses #15582 Add geoip for AWS ALB client and target IP addresses Mar 1, 2023
@davidjiglesias davidjiglesias changed the title Add geoip for AWS ALB client and target IP addresses Add geoip for AWS ALB client IP addresses Mar 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants