[FR] ES|QL remote validation support newline split indices

[eric-forte-elastic]

Pull Request

Issue link(s):
Resolves #5355

Summary - What I changed

Adds support for newline separated ESQL indices in the from line. Currently, this is done through regex parsing, which requires maintenance such as this PR, given an ESQL parser, this would have been caught there. See issue for more detail.

How To Test

Create a rule with some indices split by newlines. E.g.

from
  logs-network_traffic.http-*, logs-network_traffic.tls-*,
  logs-nginx.access-*,
  logs-apache.access-*,
  logs-apache_tomcat.access-*,
  logs-iis.access-*

reconnaissance_web_server_unusual_user_agents.toml.txt

Run view-rule with remote validation to check this new feature and then run python -m detection_rules dev test esql-remote-validation to test validation on the rest of the rules.

Testing Results

detection-rules on  5355-fr-esql-remote-validation-support-newline-split-indices [$!?] is  v1.5.15 via  v3.12.12 (detection-rules-build) on  eric.forte took 2s 
❯ python -m detection_rules dev integrations build-manifests -i ngidetection-rules on  5355-fr-esql-remote-validation-support-newline-split-indices [$?] is  v1.5.15 via  v3.12.12 (detection-rules-build) on  eric.forte 
  ❯ python -m detection_rules dev test esql-remote-validation
  Loaded config file: /home/forteea1/Code/dac_demo/dac_clean/detection-rules/.detection-rules-cfg.json

  █▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
  █  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
  █▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

  Total rules: 69
  Failed rules: 0
  Failed rules written to failed_rules.log

detection-rules on  5355-fr-esql-remote-validation-support-newline-split-indices [$!?] is  v1.5.15 via  v3.12.12 (detection-rules-build) on  eric.forte took 22s 
❯ python -m detection_rules view-rule rules/cross-platform/reconnaissance_web_server_unusual_user_agents.toml --esql-remote-validation
Loaded config file: /home/forteea1/Code/dac_demo/dac_clean/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

  _transport = transport_class(
/home/forteea1/Code/dac_demo/dac_clean/detection-rules/detection_rules/index_mappings.py:366: ElasticsearchWarning: No limit defined, adding default limit of [1000]
  response = elastic_client.esql.query(query=query)
{
  "author": [
    "Elastic"
  ],
  "description": "This rule detects unusual spikes in web server requests with uncommon or suspicious user-agent strings. Such activity may indicate reconnaissance attempts by attackers trying to identify vulnerabilities in web applications or servers. These user-agents are often associated with automated tools used for scanning, vulnerability assessment, or brute-force attacks.",
  "from": "now-9m",
  "interval": "10m",
  "language": "esql",
  "license": "Elastic License v2",
  "name": "Web Server Suspicious User Agent Requests",
  "query": "from\n  logs-network_traffic.http-*, logs-network_traffic.tls-*,\n  logs-nginx.access-*,\n  logs-apache.access-*,\n  logs-apache_tomcat.access-*,\n  logs-iis.access-*\n\n| eval Esql.user_agent_original_lower = to_lower(user_agent.original)\n\n| where\n    (url.original is not null or url.full is not null) and\n  (\n\t\tEsql.user_agent_original_lower like \"mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/74.0.3729.169 safari/537.36\" or // Nikto\n\t\tEsql.user_agent_original_lower like \"nikto*\" or // Nikto \n\t\tEsql.user_agent_original_lower like \"mozilla/4.0 (compatible; msie 8.0; windows nt 5.1; trident/4.0)\" or // Nessus Vulnerability Scanner\n        Esql.user_agent_original_lower like \"*nessus*\" or // Nessus Vulnerability Scanner\n\t\tEsql.user_agent_original_lower like \"sqlmap/*\" or // SQLMap\n\t\tEsql.user_agent_original_lower like \"wpscan*\" or // WPScan\n\t\tEsql.user_agent_original_lower like \"feroxbuster/*\" or // Feroxbuster \n\t\tEsql.user_agent_original_lower like \"masscan*\" or // Masscan & masscan-ng\n\t\tEsql.user_agent_original_lower like \"fuzz*\" or // Ffuf\n\t\tEsql.user_agent_original_lower like \"mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/user_agent.original like~ 87.0.4280.88 safari/537.36\" or // Dirsearch\n\t\tEsql.user_agent_original_lower like \"mozilla/4.0 (compatible; msie 6.0; windows nt 5.1)\" or // Dirb\n\t\tEsql.user_agent_original_lower like \"dirbuster*\" or // Dirbuster\n\t\tEsql.user_agent_original_lower like \"gobuster/*\" or // Gobuster\n        Esql.user_agent_original_lower like \"*dirsearch*\" or // dirsearch \n\t\tEsql.user_agent_original_lower like \"*nmap*\" or // Nmap Scripting Engine\n        Esql.user_agent_original_lower like \"*hydra*\" or // Hydra Brute Forcer\n        Esql.user_agent_original_lower like \"*w3af*\" or // w3af Web Application Attack and Audit Framework\n        Esql.user_agent_original_lower like \"*arachni*\" or // Arachni Web Application Security Scanner\n        Esql.user_agent_original_lower like \"*skipfish*\" or // Skipfish Web Application Security Scanner\n        Esql.user_agent_original_lower like \"*openvas*\" or // OpenVAS Vulnerability Scanner\n        Esql.user_agent_original_lower like \"*acunetix*\" or // Acunetix Vulnerability Scanner\n        Esql.user_agent_original_lower like \"*zap*\" or // OWASP ZAP \n        Esql.user_agent_original_lower like \"*burp*\" // Burp Suite\n  )\n\n| eval Esql.url_text  = case(url.original is not null, url.original, url.full)\n| eval Esql.url_lower = to_lower(Esql.url_text)\n\n| keep\n    @timestamp,\n    event.dataset,\n    user_agent.original,\n    source.ip,\n    agent.id,\n    host.name,\n\tEsql.url_lower,\n    Esql.user_agent_original_lower\n| stats\n    Esql.event_count = count(),\n    Esql.url_path_count_distinct = count_distinct(Esql.url_lower),\n    Esql.host_name_values = values(host.name),\n    Esql.agent_id_values = values(agent.id),\n    Esql.url_path_values = values(Esql.url_lower),\n    Esql.user_agent_original_values = values(Esql.user_agent_original_lower),\n    Esql.event_dataset_values = values(event.dataset)\n    by source.ip, agent.id\n| where\n    Esql.event_count > 50 and Esql.url_path_count_distinct > 10\n",
  "related_integrations": [
    {
      "package": "nginx",
      "version": "^2.0.0"
    },
    {
      "package": "apache",
      "version": "^2.0.0"
    },
    {
      "package": "apache_tomcat",
      "version": "^1.9.0"
    },
    {
      "package": "iis",
      "version": "^1.21.0"
    },
    {
      "package": "network_traffic",
      "version": "^1.33.0"
    }
  ],
  "required_fields": [
    {
      "ecs": false,
      "name": "Esql.agent_id_values",
      "type": "keyword"
    },
    {
      "ecs": false,
      "name": "Esql.event_count",
      "type": "long"
    },
    {
      "ecs": false,
      "name": "Esql.event_dataset_values",
      "type": "keyword"
    },
    {
      "ecs": false,
      "name": "Esql.host_name_values",
      "type": "keyword"
    },
    {
      "ecs": false,
      "name": "Esql.url_path_count_distinct",
      "type": "long"
    },
    {
      "ecs": false,
      "name": "Esql.url_path_values",
      "type": "keyword"
    },
    {
      "ecs": false,
      "name": "Esql.user_agent_original_values",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "agent.id",
      "type": "keyword"
    },
    {
      "ecs": true,
      "name": "source.ip",
      "type": "ip"
    }
  ],
  "risk_score": 21,
  "rule_id": "a1b7ffa4-bf80-4bf1-86ad-c3f4dc718b35",
  "severity": "low",
  "tags": [
    "Domain: Web",
    "Domain: Network",
    "Use Case: Threat Detection",
    "Tactic: Reconnaissance",
    "Tactic: Credential Access",
    "Data Source: Network Packet Capture",
    "Data Source: Nginx",
    "Data Source: Apache",
    "Data Source: Apache Tomcat",
    "Data Source: IIS"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0043",
        "name": "Reconnaissance",
        "reference": "https://attack.mitre.org/tactics/TA0043/"
      },
      "technique": [
        {
          "id": "T1595",
          "name": "Active Scanning",
          "reference": "https://attack.mitre.org/techniques/T1595/",
          "subtechnique": [
            {
              "id": "T1595.001",
              "name": "Scanning IP Blocks",
              "reference": "https://attack.mitre.org/techniques/T1595/001/"
            },
            {
              "id": "T1595.002",
              "name": "Vulnerability Scanning",
              "reference": "https://attack.mitre.org/techniques/T1595/002/"
            },
            {
              "id": "T1595.003",
              "name": "Wordlist Scanning",
              "reference": "https://attack.mitre.org/techniques/T1595/003/"
            }
          ]
        }
      ]
    },
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0006",
        "name": "Credential Access",
        "reference": "https://attack.mitre.org/tactics/TA0006/"
      },
      "technique": [
        {
          "id": "T1110",
          "name": "Brute Force",
          "reference": "https://attack.mitre.org/techniques/T1110/"
        }
      ]
    }
  ],
  "timestamp_override": "event.ingested",
  "type": "esql",
  "version": 1
}

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a new schema feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Schema Related Checks

• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Link to the relevant Kibana PR or issue provided
• Test export/import flow:
  • Exported detection rule(s) from Kibana to showcase the feature(s)
  • Converted the exported ndjson file(s) to toml in the detection-rules repo
  • Re-exported the toml rule(s) to ndjson and re-imported into Kibana
• Updated necessary unit tests to accommodate the feature
• Incorporated a comprehensive test rule in unit tests for full schema coverage
• Applied min_compat restrictions to limit the feature to a specified minimum stack version
• Executed all unit tests locally with a test toml rule to confirm passing
• Included Kibana PR implementer as an optional reviewer for insights on the feature
• Implemented requisite downgrade functionality
• Cross-referenced the feature with product documentation for consistency
• Confirm that the proper version label is applied to the PR patch, minor, major.