Skip to content

fix: type hinting fixes and additional code checks #4790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 57 commits into from
Jul 1, 2025
Merged

fix: type hinting fixes and additional code checks #4790

merged 57 commits into from
Jul 1, 2025

Conversation

traut
Copy link
Contributor

@traut traut commented Jun 11, 2025 •
edited by Mikaayenson
Loading

Pull Request

Issue link(s):

Summary - What I changed

  • adding ruff and pyright checks in CI workflow
  • making sure pyright has no complains

How To Test

Checklist

  • Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
  • Added the meta:rapid-merge label if planning to merge within 24 hours
  • Secret and sensitive material has been managed correctly
  • Automated testing was updated or added to match the most common scenarios
  • Documentation and comments were added for features that require explanation

Contributor checklist

@traut traut changed the title [WIP] Type hint fixes and adding code checks [WIP] fix: type hint fixes and adding code checks Jun 11, 2025
@traut traut added python Internal python for the repository ci/cd maintenance Internal changes minor labels Jun 17, 2025
@traut traut marked this pull request as ready for review June 17, 2025 16:32
@github-actions GitHub Actions
Copy link
Contributor

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a new schema feature to the code.

Documentation and Context

  • Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
  • Include additional context or screenshots.
  • Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

  • Code follows established design patterns within the repo and avoids duplication.
  • Code changes do not introduce new warnings or errors.
  • Variables and functions are well-named and descriptive.
  • Any unnecessary / commented-out code is removed.
  • Ensure that the code is modular and reusable where applicable.
  • Check for proper exception handling and messaging.

Testing

  • New unit tests have been added to cover the enhancement.
  • Existing unit tests have been updated to reflect the changes.
  • Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
  • Validate that any rules affected by the enhancement are correctly updated.
  • Ensure that performance is not negatively impacted by the changes.
  • Verify that any release artifacts are properly generated and tested.

Additional Schema Related Checks

  • Ensure that the enhancement does not break existing functionality. (e.g., run make test-cli)
  • Review the enhancement with a peer or team member for additional insights.
  • Verify that the enhancement works across all relevant environments (e.g., different OS versions).
  • Confirm that all dependencies are up-to-date and compatible with the changes.
  • Link to the relevant Kibana PR or issue provided
  • Exported detection rule(s) from Kibana to showcase the feature(s)
  • Converted the exported ndjson file(s) to toml in the detection-rules repo
  • Re-exported the toml rule(s) to ndjson and re-imported into Kibana
  • Updated necessary unit tests to accommodate the feature
  • Applied min_compat restrictions to limit the feature to a specified minimum stack version
  • Executed all unit tests locally with a test toml rule to confirm passing
  • Included Kibana PR implementer as an optional reviewer for insights on the feature
  • Implemented requisite downgrade functionality
  • Cross-referenced the feature with product documentation for consistency
  • Incorporated a comprehensive test rule in unit tests for full schema coverage
  • Conducted system testing, including fleet, import, and create APIs (e.g., run make test-remote-cli)
  • Confirm that the proper version label is applied to the PR patch, minor, major.

@traut traut changed the title [WIP] fix: type hint fixes and adding code checks fix: type hinting fixes and additional code checks Jun 17, 2025
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
Mikaayenson
Mikaayenson reviewed Jun 17, 2025
View reviewed changes
detection_rules/ecs.py
"""Get schema for KQL."""
indexes = indexes or ()
converted = flatten_multi_fields(get_schema(version, name='ecs_flat'))
indexes = indexes or []
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious as to why this was a tuple

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no idea. I don't think there is a risk of mutation, so we might as well simplify and have list here

@Mikaayenson
Copy link
Contributor

Mikaayenson commented Jun 17, 2025 •
edited
Loading

  • Any reason why the build didn't run? Waiting for status to be reported
  • Note, I think we need to run the lint tests locally and add to this PR (since the workflow won't run until the action is on main)
  • We'll also want to open a maintenance window and test the backporting logic.

@traut @Mikaayenson
Update .github/CODEOWNERS
e395c5f 
Co-authored-by: Mika Ayenson, PhD <Mikaayenson@users.noreply.github.com>
@Mikaayenson
Copy link
Contributor

Mikaayenson commented Jun 30, 2025 •
edited
Loading

🟡 Make test-remote-cli

  • import rules needs to be checked to address deprecated_date missing positional argument error. Its not in the existing makefile yet, but you can run this command. (python -m detection_rules kibana --ignore-ssl-errors true import-rules -o -e -ac)

🟢 Make test-hunting-cli

Bash Output 

(detection-rules-build) ➜  detection-rules git:(style-fixes) ✗ make test-hunting-cli
Installing all dependencies...
./env/detection-rules-build/bin/pip install .[hunting]
Processing /Users/stryker/workspace/Elastic/detection-rules
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting detection-rules-kql@ git+https://github.com/elastic/detection-rules.git#subdirectory=lib/kql (from detection_rules==1.3.0)
  Cloning https://github.com/elastic/detection-rules.git to /private/var/folders/tf/sv2j9kkn1h3cbxwhrg0d895m0000gn/T/pip-install-nxq3q94d/detection-rules-kql_321f74d7dd80432583600292bb7a7bfe
  Running command git clone --filter=blob:none --quiet https://github.com/elastic/detection-rules.git /private/var/folders/tf/sv2j9kkn1h3cbxwhrg0d895m0000gn/T/pip-install-nxq3q94d/detection-rules-kql_321f74d7dd80432583600292bb7a7bfe
  Resolved https://github.com/elastic/detection-rules.git to commit 0847c323334f83857f0b250b939926bab33a00c1
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting detection-rules-kibana@ git+https://github.com/elastic/detection-rules.git#subdirectory=lib/kibana (from detection_rules==1.3.0)
  Cloning https://github.com/elastic/detection-rules.git to /private/var/folders/tf/sv2j9kkn1h3cbxwhrg0d895m0000gn/T/pip-install-nxq3q94d/detection-rules-kibana_326528515c604c64af12cc127247835a
  Running command git clone --filter=blob:none --quiet https://github.com/elastic/detection-rules.git /private/var/folders/tf/sv2j9kkn1h3cbxwhrg0d895m0000gn/T/pip-install-nxq3q94d/detection-rules-kibana_326528515c604c64af12cc127247835a
  Resolved https://github.com/elastic/detection-rules.git to commit 0847c323334f83857f0b250b939926bab33a00c1
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: Click~=8.1.7 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (8.1.7)
Requirement already satisfied: elasticsearch~=8.12.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (8.12.1)
Requirement already satisfied: eql==0.9.19 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (0.9.19)
Requirement already satisfied: jsl==0.2.4 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (0.2.4)
Requirement already satisfied: jsonschema>=4.21.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (4.21.1)
Requirement already satisfied: marko==2.0.3 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (2.0.3)
Collecting marshmallow-dataclass[union]>=8.7 (from detection_rules==1.3.0)
  Obtaining dependency information for marshmallow-dataclass[union]>=8.7 from https://files.pythonhosted.org/packages/3e/f5/6764f3f3203d14a0e6df0fce4838f8195ccc61ec7d48d7ed89acfb8adeed/marshmallow_dataclass-8.7.1-py3-none-any.whl.metadata
  Downloading marshmallow_dataclass-8.7.1-py3-none-any.whl.metadata (12 kB)
Requirement already satisfied: marshmallow-jsonschema~=0.13.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (0.13.0)
Requirement already satisfied: marshmallow-union~=0.1.15 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (0.1.15.post1)
Collecting marshmallow~=3.26.1 (from detection_rules==1.3.0)
  Obtaining dependency information for marshmallow~=3.26.1 from https://files.pythonhosted.org/packages/34/75/51952c7b2d3873b44a0028b1bd26a25078c18f92f256608e8d1dc61b39fd/marshmallow-3.26.1-py3-none-any.whl.metadata
  Downloading marshmallow-3.26.1-py3-none-any.whl.metadata (7.3 kB)
Requirement already satisfied: pytoml==0.1.21 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (0.1.21)
Requirement already satisfied: PyYAML~=6.0.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (6.0.1)
Requirement already satisfied: requests~=2.31.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (2.31.0)
Requirement already satisfied: toml==0.10.2 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (0.10.2)
Requirement already satisfied: typing-inspect==0.9.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (0.9.0)
Collecting typing-extensions>=4.12 (from detection_rules==1.3.0)
  Obtaining dependency information for typing-extensions>=4.12 from https://files.pythonhosted.org/packages/69/e0/552843e0d356fbb5256d21449fa957fa4eff3bbc135a74a691ee70c7c5da/typing_extensions-4.14.0-py3-none-any.whl.metadata
  Downloading typing_extensions-4.14.0-py3-none-any.whl.metadata (3.0 kB)
Requirement already satisfied: XlsxWriter~=3.2.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (3.2.0)
Requirement already satisfied: semver==3.0.2 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (3.0.2)
Requirement already satisfied: PyGithub==2.2.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (2.2.0)
Collecting setuptools==75.2.0 (from detection_rules==1.3.0)
  Obtaining dependency information for setuptools==75.2.0 from https://files.pythonhosted.org/packages/31/2d/90165d51ecd38f9a02c6832198c13a4e48652485e2ccf863ebb942c531b6/setuptools-75.2.0-py3-none-any.whl.metadata
  Downloading setuptools-75.2.0-py3-none-any.whl.metadata (6.9 kB)
Requirement already satisfied: tabulate==0.9.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from detection_rules==1.3.0) (0.9.0)
Requirement already satisfied: lark-parser~=0.12.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from eql==0.9.19->detection_rules==1.3.0) (0.12.0)
Requirement already satisfied: pynacl>=1.4.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from PyGithub==2.2.0->detection_rules==1.3.0) (1.5.0)
Requirement already satisfied: pyjwt[crypto]>=2.4.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from PyGithub==2.2.0->detection_rules==1.3.0) (2.8.0)
Requirement already satisfied: urllib3>=1.26.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from PyGithub==2.2.0->detection_rules==1.3.0) (2.2.1)
Requirement already satisfied: Deprecated in ./env/detection-rules-build/lib/python3.12/site-packages (from PyGithub==2.2.0->detection_rules==1.3.0) (1.2.14)
Requirement already satisfied: mypy-extensions>=0.3.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from typing-inspect==0.9.0->detection_rules==1.3.0) (1.0.0)
Requirement already satisfied: elastic-transport<9,>=8 in ./env/detection-rules-build/lib/python3.12/site-packages (from elasticsearch~=8.12.1->detection_rules==1.3.0) (8.12.0)
Requirement already satisfied: attrs>=22.2.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from jsonschema>=4.21.1->detection_rules==1.3.0) (23.2.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in ./env/detection-rules-build/lib/python3.12/site-packages (from jsonschema>=4.21.1->detection_rules==1.3.0) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in ./env/detection-rules-build/lib/python3.12/site-packages (from jsonschema>=4.21.1->detection_rules==1.3.0) (0.34.0)
Requirement already satisfied: rpds-py>=0.7.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from jsonschema>=4.21.1->detection_rules==1.3.0) (0.18.0)
Requirement already satisfied: packaging>=17.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from marshmallow~=3.26.1->detection_rules==1.3.0) (24.0)
WARNING: marshmallow-dataclass 8.7.1 does not provide the extra 'union'
Collecting typeguard<5,>=4.0 (from marshmallow-dataclass[union]>=8.7->detection_rules==1.3.0)
  Obtaining dependency information for typeguard<5,>=4.0 from https://files.pythonhosted.org/packages/1b/a9/e3aee762739c1d7528da1c3e06d518503f8b6c439c35549b53735ba52ead/typeguard-4.4.4-py3-none-any.whl.metadata
  Downloading typeguard-4.4.4-py3-none-any.whl.metadata (3.3 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in ./env/detection-rules-build/lib/python3.12/site-packages (from requests~=2.31.0->detection_rules==1.3.0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./env/detection-rules-build/lib/python3.12/site-packages (from requests~=2.31.0->detection_rules==1.3.0) (3.6)
Requirement already satisfied: certifi>=2017.4.17 in ./env/detection-rules-build/lib/python3.12/site-packages (from requests~=2.31.0->detection_rules==1.3.0) (2024.2.2)
Requirement already satisfied: cryptography>=3.4.0 in ./env/detection-rules-build/lib/python3.12/site-packages (from pyjwt[crypto]>=2.4.0->PyGithub==2.2.0->detection_rules==1.3.0) (42.0.5)
Requirement already satisfied: cffi>=1.4.1 in ./env/detection-rules-build/lib/python3.12/site-packages (from pynacl>=1.4.0->PyGithub==2.2.0->detection_rules==1.3.0) (1.16.0)
Requirement already satisfied: wrapt<2,>=1.10 in ./env/detection-rules-build/lib/python3.12/site-packages (from Deprecated->PyGithub==2.2.0->detection_rules==1.3.0) (1.16.0)
Requirement already satisfied: pycparser in ./env/detection-rules-build/lib/python3.12/site-packages (from cffi>=1.4.1->pynacl>=1.4.0->PyGithub==2.2.0->detection_rules==1.3.0) (2.21)
Downloading setuptools-75.2.0-py3-none-any.whl (1.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 9.0 MB/s eta 0:00:00
Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.9/50.9 kB 4.2 MB/s eta 0:00:00
Downloading typing_extensions-4.14.0-py3-none-any.whl (43 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.8/43.8 kB 5.4 MB/s eta 0:00:00
Downloading typeguard-4.4.4-py3-none-any.whl (34 kB)
Downloading marshmallow_dataclass-8.7.1-py3-none-any.whl (19 kB)
Building wheels for collected packages: detection_rules, detection-rules-kibana
  Building wheel for detection_rules (pyproject.toml) ... done
  Created wheel for detection_rules: filename=detection_rules-1.3.0-py3-none-any.whl size=53243253 sha256=82f4c0f7a9900b5c8a2ada69250bc552bd79a27f77f90680514570523a32fc80
  Stored in directory: /Users/stryker/Library/Caches/pip/wheels/9a/02/14/21d8b892171aaf0a403a7a67eb40807a9cf970e0d1582d7753
  Building wheel for detection-rules-kibana (pyproject.toml) ... done
  Created wheel for detection-rules-kibana: filename=detection_rules_kibana-0.4.4-py3-none-any.whl size=8784 sha256=c92b3162b4322410af300e11087dd35e2024b043674f968537172daa6ce85de0
  Stored in directory: /private/var/folders/tf/sv2j9kkn1h3cbxwhrg0d895m0000gn/T/pip-ephem-wheel-cache-cs3hg21c/wheels/08/96/aa/385e2ed061d591561ceb888c9cc4321cda2494a338957e581e
Successfully built detection_rules detection-rules-kibana
Installing collected packages: typing-extensions, setuptools, marshmallow, typeguard, marshmallow-dataclass, detection-rules-kibana, detection_rules
  Attempting uninstall: typing-extensions
    Found existing installation: typing_extensions 4.10.0
    Uninstalling typing_extensions-4.10.0:
      Successfully uninstalled typing_extensions-4.10.0
  Attempting uninstall: setuptools
    Found existing installation: setuptools 69.2.0
    Uninstalling setuptools-69.2.0:
      Successfully uninstalled setuptools-69.2.0
  Attempting uninstall: marshmallow
    Found existing installation: marshmallow 3.21.1
    Uninstalling marshmallow-3.21.1:
      Successfully uninstalled marshmallow-3.21.1
  Attempting uninstall: typeguard
    Found existing installation: typeguard 3.0.2
    Uninstalling typeguard-3.0.2:
      Successfully uninstalled typeguard-3.0.2
  Attempting uninstall: marshmallow-dataclass
    Found existing installation: marshmallow_dataclass 8.6.1
    Uninstalling marshmallow_dataclass-8.6.1:
      Successfully uninstalled marshmallow_dataclass-8.6.1
  Attempting uninstall: detection-rules-kibana
    Found existing installation: detection-rules-kibana 0.2.1
    Uninstalling detection-rules-kibana-0.2.1:
      Successfully uninstalled detection-rules-kibana-0.2.1
  Attempting uninstall: detection_rules
    Found existing installation: detection_rules 0.1.0
    Uninstalling detection_rules-0.1.0:
      Successfully uninstalled detection_rules-0.1.0
Successfully installed detection-rules-kibana-0.4.4 detection_rules-1.3.0 marshmallow-3.26.1 marshmallow-dataclass-8.7.1 setuptools-75.2.0 typeguard-4.4.4 typing-extensions-4.14.0

[notice] A new release of pip is available: 23.2.1 -> 25.1.1
[notice] To update, run: pip install --upgrade pip
Executing test_hunting_cli script...
Running hunting CLI tests...
Searching: Search for T1078.004 subtechnique in AWS data source
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json
Searching for queries based on provided filters...
Filtering by data source: aws
Searching for MITRE techniques: ('T1078.004',)

Found 4 matching queries:

╒══════════════════════════════════════════════════════════╤══════════════════════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╤════════════════════╤═══════════╕
│ Name                                                     │ UUID                                 │ Location                                                                                                                           │ Data Source        │ MITRE     │
╞══════════════════════════════════════════════════════════╪══════════════════════════════════════╪════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╪════════════════════╪═══════════╡
│ IAM User Activity with No MFA Session                    │ 913a47be-649c-11ef-a693-f661ea17fbcc │ /Users/stryker/workspace/Elastic/detection-rules/hunting/aws/queries/iam_user_activity_with_no_mfa_session.toml                    │ ['aws.cloudtrail'] │ T1078.004 │
├──────────────────────────────────────────────────────────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────────────────┼───────────┤
│ Signin Single Factor Console Login via Federated Session │ 953b1252-5efd-11ef-a997-f661ea17fbce │ /Users/stryker/workspace/Elastic/detection-rules/hunting/aws/queries/signin_single_factor_console_login_via_federated_session.toml │ ['aws.cloudtrail'] │ T1078.004 │
├──────────────────────────────────────────────────────────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────────────────┼───────────┤
│ AWS IAM Unusual AWS Access Key Usage for User            │ 18ce3dbc-b1b3-11ef-9e63-f661ea17fbce │ /Users/stryker/workspace/Elastic/detection-rules/hunting/aws/queries/iam_unusual_access_key_usage_for_user.toml                    │ ['aws.cloudtrail'] │ T1078.004 │
├──────────────────────────────────────────────────────────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────────────────┼───────────┤
│ IAM Unusual Default Aviatrix Role Activity               │ 9fe48b6e-d83a-11ef-84a6-f661ea17fbcd │ /Users/stryker/workspace/Elastic/detection-rules/hunting/aws/queries/iam_unusual_default_aviatrix_role_activity.toml               │ ['aws.cloudtrail'] │ T1078.004 │
╘══════════════════════════════════════════════════════════╧══════════════════════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╧════════════════════╧═══════════╛
Refreshing index
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json
Refreshing the index.yml and index.md files.
Index YAML updated at: /Users/stryker/workspace/Elastic/detection-rules/hunting/index.yml
Index Markdown updated at: /Users/stryker/workspace/Elastic/detection-rules/hunting/index.md
Index refresh complete.
Generating Markdown: initial_access_higher_than_average_failed_authentication.toml
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json
Generating Markdown for single file: hunting/okta/queries/initial_access_higher_than_average_failed_authentication.toml
Processing specific TOML file: hunting/okta/queries/initial_access_higher_than_average_failed_authentication.toml
Markdown generated: hunting/okta/docs/initial_access_higher_than_average_failed_authentication.md
Index YAML updated at: /Users/stryker/workspace/Elastic/detection-rules/hunting/index.yml
Index Markdown updated at: /Users/stryker/workspace/Elastic/detection-rules/hunting/index.md
Running Query: low_volume_external_network_connections_from_process.toml
Requires .detection-rules-cfg.json credentials file set.
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json

Hunting Description:

This hunt identifies low volume external network connections initiated by processes on Linux systems. It focuses on
connections attempted by processes that have been seen infrequently (five or fewer connections) and by unique agents.
This can help identify potentially suspicious activity that might be missed due to low volume.

Running all eligible queries...

Running Query 1:
from logs-endpoint.events.network-*
| where @timestamp > now() - 7 day
| where host.os.type == "linux" and event.category == "network" and event.type == "start" and event.action ==
    "connection_attempted" and not process.name is null and
    not CIDR_MATCH(destination.ip, "10.0.0.0/8", "127.0.0.0/8", "169.254.0.0/16", "172.16.0.0/12", "192.0.0.0/24",
    "192.0.0.0/29", "192.0.0.8/32", "192.0.0.9/32", "192.0.0.10/32", "192.0.0.170/32", "192.0.0.171/32", "192.0.2.0/24",
    "192.31.196.0/24", "192.52.193.0/24", "192.168.0.0/16", "192.88.99.0/24", "224.0.0.0/4", "100.64.0.0/10",
    "192.175.48.0/24","198.18.0.0/15", "198.51.100.0/24", "203.0.113.0/24", "240.0.0.0/4", "::1","FE80::/10",
    "FF00::/8")
| stats connection_count = count(*), unique_agent_count = count_distinct(agent.id) by process.name
| where connection_count <= 5 and unique_agent_count == 1
| limit 100
| sort connection_count, unique_agent_count asc

CLI Error (AuthenticationException): Failed authentication for stryker812:ZXVyb3BlLXdlc3QxLmdjcC5jbG91ZC5lcy5pbzo0NDMkOGM3MWRiN2U4ZWQwNDRmYTllOTQ2MTY0NmVhMzkyMzIkNzE0OWZmY2FjMzg4NDRlMjllYjE5Mzk5ZTU4Y2U5MTY=
Viewing Hunt: 12526f14-5e35-4f5f-884c-96c6a353a544
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json
{
    "author": "Elastic",
    "description": "This hunt identifies low volume external network connections initiated by processes on Linux systems. It focuses on connections attempted by processes that have been seen infrequently (five or fewer connections) and by unique agents. This can help identify potentially suspicious activity that might be missed due to low volume.\n",
    "integration": [
        "endpoint"
    ],
    "uuid": "12526f14-5e35-4f5f-884c-96c6a353a544",
    "name": "Low Volume External Network Connections from Process by Unique Agent",
    "language": [
        "ES|QL"
    ],
    "license": "Elastic License v2",
    "query": [
        "from logs-endpoint.events.network-*\n| where @timestamp > now() - 7 day\n| where host.os.type == \"linux\" and event.category == \"network\" and event.type == \"start\" and event.action == \"connection_attempted\" and not process.name is null and\n    not CIDR_MATCH(destination.ip, \"10.0.0.0/8\", \"127.0.0.0/8\", \"169.254.0.0/16\", \"172.16.0.0/12\", \"192.0.0.0/24\", \"192.0.0.0/29\", \"192.0.0.8/32\", \"192.0.0.9/32\", \"192.0.0.10/32\", \"192.0.0.170/32\", \"192.0.0.171/32\", \"192.0.2.0/24\", \"192.31.196.0/24\", \"192.52.193.0/24\", \"192.168.0.0/16\", \"192.88.99.0/24\", \"224.0.0.0/4\", \"100.64.0.0/10\", \"192.175.48.0/24\",\"198.18.0.0/15\", \"198.51.100.0/24\", \"203.0.113.0/24\", \"240.0.0.0/4\", \"::1\",\"FE80::/10\", \"FF00::/8\")\n| stats connection_count = count(*), unique_agent_count = count_distinct(agent.id) by process.name\n| where connection_count <= 5 and unique_agent_count == 1\n| limit 100\n| sort connection_count, unique_agent_count asc\n",
        "from logs-endpoint.events.network-*\n| where @timestamp > now() - 7 day\n| where host.os.type == \"linux\" and event.category == \"network\" and event.type == \"start\" and event.action == \"connection_attempted\" and user.id == \"0\" and not process.name is null and\n    not CIDR_MATCH(destination.ip, \"10.0.0.0/8\", \"127.0.0.0/8\", \"169.254.0.0/16\", \"172.16.0.0/12\", \"192.0.0.0/24\", \"192.0.0.0/29\", \"192.0.0.8/32\", \"192.0.0.9/32\", \"192.0.0.10/32\", \"192.0.0.170/32\", \"192.0.0.171/32\", \"192.0.2.0/24\", \"192.31.196.0/24\", \"192.52.193.0/24\", \"192.168.0.0/16\", \"192.88.99.0/24\", \"224.0.0.0/4\", \"100.64.0.0/10\", \"192.175.48.0/24\",\"198.18.0.0/15\", \"198.51.100.0/24\", \"203.0.113.0/24\", \"240.0.0.0/4\", \"::1\",\"FE80::/10\", \"FF00::/8\")\n| stats connection_count = count(*), unique_agent_count = count_distinct(agent.id) by process.name\n| where connection_count <= 5 and unique_agent_count == 1\n| limit 100\n| sort connection_count, unique_agent_count asc\n"
    ],
    "notes": [
        "Monitors for network connections attempted by processes that have a low occurrence frequency (five or fewer connections) and are seen by a unique agent.",
        "Excludes common internal IP ranges to minimize false positives.",
        "A separate query is included to specifically monitor low volume network connections initiated by the root user, as these can be particularly indicative of malicious activity."
    ],
    "mitre": [
        "T1071.001",
        "T1071.004"
    ],
    "references": []
}
Generating summary of hunts by integration
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json
Generating hunt summary broken down by integration...
╒════════════════════════╤══════════════╕
│ Integration            │   Hunt Count │
╞════════════════════════╪══════════════╡
│ endpoint               │           87 │
├────────────────────────┼──────────────┤
│ network_traffic        │            1 │
├────────────────────────┼──────────────┤
│ system                 │           13 │
├────────────────────────┼──────────────┤
│ azure                  │            9 │
├────────────────────────┼──────────────┤
│ o365                   │            2 │
├────────────────────────┼──────────────┤
│ windows                │           24 │
├────────────────────────┼──────────────┤
│ aws_bedrock.invocation │            4 │
├────────────────────────┼──────────────┤
│ okta                   │           11 │
├────────────────────────┼──────────────┤
│ aws.cloudtrail         │           24 │
╘════════════════════════╧══════════════╛
Generating summary of hunts by platform
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json
Generating hunt summary broken down by platform...
╒═════════════════════╤══════════════╕
│ Platform (Folder)   │   Hunt Count │
╞═════════════════════╪══════════════╡
│ cross-platform      │            1 │
├─────────────────────┼──────────────┤
│ llm                 │            4 │
├─────────────────────┼──────────────┤
│ macos               │           15 │
├─────────────────────┼──────────────┤
│ azure               │            8 │
├─────────────────────┼──────────────┤
│ linux               │           43 │
├─────────────────────┼──────────────┤
│ okta                │           11 │
├─────────────────────┼──────────────┤
│ aws                 │           24 │
├─────────────────────┼──────────────┤
│ windows             │           32 │
╘═════════════════════╧══════════════╛
Generating summary of hunts by language
Loaded config file: /Users/stryker/workspace/Elastic/detection-rules/.detection-rules-cfg.json
Generating hunt summary broken down by language...
╒════════════╤══════════════╕
│ Language   │   Hunt Count │
╞════════════╪══════════════╡
│ ES|QL      │          121 │
├────────────┼──────────────┤
│ EQL        │           13 │
├────────────┼──────────────┤
│ OSQuery    │           26 │
╘════════════╧══════════════╛
(detection-rules-build) ➜  detection-rules git:(style-fixes) ✗

Mikaayenson
Mikaayenson approved these changes Jul 1, 2025
View reviewed changes
Copy link
Contributor

@Mikaayenson Mikaayenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great work! unit tests, and integration tests lgtm.

@eric-forte-elastic
Copy link
Contributor

eric-forte-elastic commented Jul 1, 2025 •
edited
Loading

Action connector and exception import tested and functional.

❯ python -m detection_rules kibana import-rules -o -e -ac
Loaded config file: /tmp/dac_dr/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

2 rule(s) successfully imported
 - b524fd0c-c581-4f66-a42d-07ef70ac8ffd
 - c70bfcbc-b0d9-43ed-9e5c-5dabbe2ab694
1 exception list(s) successfully imported
 - f9374858-2c4a-48cd-affc-99ae4d5d0d74
1 action connector(s) successfully imported
 - d29c730b-6528-4ce3-a371-e05ee7ad618a

@Mikaayenson Mikaayenson merged commit 1fb60d6 into main Jul 1, 2025
18 of 22 checks passed
@Mikaayenson Mikaayenson deleted the style-fixes branch July 1, 2025 13:20
@traut traut mentioned this pull request Jul 2, 2025
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Mikaayenson Mikaayenson Mikaayenson approved these changes

@shashank-elastic shashank-elastic shashank-elastic approved these changes

@eric-forte-elastic eric-forte-elastic eric-forte-elastic approved these changes

@terrancedejesus terrancedejesus Awaiting requested review from terrancedejesus

Assignees

@traut traut

Labels
backport: auto ci/cd Hunting maintenance Internal changes minor python Internal python for the repository schema
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug] Help Flag Returns Errors [FR] Add Support for Python 3.13
4 participants
@traut @Mikaayenson @eric-forte-elastic @shashank-elastic