Skip to content

Conversation

@Mikaayenson
Copy link
Contributor

@Mikaayenson Mikaayenson commented Apr 29, 2022

Issues

Resolves #1935

Summary

  • Adds a field in the metadata so rule developers know what stack versions the rule has forked
  • Route B: For new forks on the latest stack, will bump the version space 100+ (or whatever passed in buffer_int)
  • Route C: If forking on an old stack, attempts to calculate half the version space remaining between the latest stack and the max previous version number, and errors if there is no space left

Testing

Ran the etc/lock-multiple.sh script after min_stacking a rule.

(detection_dev) ➜  detection-rules git:(main) ./detection_rules/etc/lock-multiple.sh '8.2'
+ set -e
++ git rev-parse --abbrev-ref HEAD
+ CURRENT_BRANCH=main
++ echo 8.2
++ sed 's/,/ /g'
+ for BRANCH in '$(echo $@ | sed "s/,/ /g")'
+ echo 8.2
8.2
+ git checkout 8.2
Switched to branch '8.2'
Your branch is up to date with 'origin/8.2'.
+ git pull
Already up to date.
+ python -m detection_rules dev build-release --update-version-lock

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building package 8.2
 - 12 rules excluded from package
Rule changes detected!
 - 18 changed rules
 - 4 new rules
 - 0 newly deprecated rules
run `build-release --update-version-lock` to update version.lock.json and deprecated_rules.json
Rule changes detected!
 - 18 changed rules
 - 4 new rules
 - 0 newly deprecated rules
Detailed changes:
  A: 69c420e8-6c9e-4d28-86c0-8a2be2d1e78c, new version: 6
  A: 015cca13-8832-49ac-a01b-a396114809f6, new version: 1
  A: e12c0318-99b1-44f2-830c-3a38a43207ca, new version: 3
  A: e7cd5982-17c8-4959-874c-633acde7d426, new version: 3
  A: ec8efb0c-604d-42fa-ac46-ed1cfbc38f78, new version: 4
  A: 4da13d6e-904f-4636-81d8-6ab14b4e6ae9, new version: 2
  A: c02c8b9f-5e1d-463c-a1b0-04edcdfe1a3d, new version: 3
  A: 530178da-92ea-43ce-94c2-8877a826783d, new version: 1
  A: e514d8cd-ed15-4011-84e2-d15147e059f1, new version: 3
  A: b83a7e96-2eb3-4edf-8346-427b6858d3bd, new version: 8
  A: be8afaed-4bcd-4e0a-b5f9-5562003dde81, new version: 4
  A: b5877334-677f-4fb9-86d5-a9721274223b, new version: 3
  B: 2856446a-34e6-435b-9fb5-f8f040bfa7ed, new version: 112
    - previous 7.16.0 saved as version: 11
    - current min_stack updated to 8.2.0
  A: d72e33fc-6e91-42ff-ac8b-e573268c5a87, new version: 5
  A: 93b22c0a-06a0-4131-b830-b10d5e166ff4, new version: 5
  A: 16a52c14-7883-47af-8745-9357803f0d4c, new version: 7
  A: 15a8ba77-1c13-4274-88fe-6bd14133861e, new version: 4
  A: e4e31051-ee01-4307-a6ee-b21b186958f4, new version: 1
  A: 3605a013-6f0c-4f7d-88a5-326f5be262ec, new version: 1
  A: bfeaf89b-a2a7-48a3-817f-e41829dc61ee, new version: 4
  A: c4818812-d44f-47be-aaef-4cfb2f9cc799, new version: 3
  A: ee5300a7-7e31-4a72-a258-250abb8b3aa1, new version: 6
    - min_stack_version added: 7.16.0
Updated /Users/stryker/workspace/Community/Mikaayenson/detection-rules/detection_rules/etc/version.lock.json file
Package saved to: /Users/stryker/workspace/Community/Mikaayenson/detection-rules/releases/8.2
- sha256: b9e59612dfdd728e839522421bfff99aec343c787937523e319be548d85cca59
- 679 rules included
+ git checkout main
M	detection_rules/etc/version.lock.json
Switched to branch 'main'
Your branch is up to date with 'origin/main'.

@Mikaayenson Mikaayenson added enhancement New feature or request python Internal python for the repository v8.3.0 Rules for 8.3.0 labels Apr 29, 2022
@Mikaayenson Mikaayenson self-assigned this Apr 29, 2022
@Mikaayenson Mikaayenson marked this pull request as draft April 29, 2022 21:50
@Mikaayenson Mikaayenson marked this pull request as ready for review May 2, 2022 20:20
@Mikaayenson Mikaayenson marked this pull request as draft May 2, 2022 21:17
@Mikaayenson Mikaayenson marked this pull request as ready for review May 2, 2022 21:22
@terrancedejesus
Copy link
Contributor

terrancedejesus commented May 6, 2022

Tested the outcome of the versions when locked and it appears new version in version lock will be 100+. Not sure if this is the expected outcome but I have detailed my findings and repeatable steps below to reach route B.

Steps to re-create:

  1. clone repository -> checkout this branch
  2. open detection_rules > rules > windows > collection_posh_keylogger.toml
  3. add the following to the metadata
    1. min_stack_comments = "testing"
    2. min_stack_version = "8.2"
  4. run python -m detection_rules dev update-lock-versions
  5. open detection_rules > etc > version.lock.json and see the outcome below

outcome

  "bd2c86a0-8b61-4457-ab38-96943984e889": {
    "min_stack_version": "8.2",
    "previous": {
      "7.13.0": {
        "rule_name": "PowerShell Keylogging Script",
        "sha256": "6bc0cf9d4c533e8088498db20d276e4d852ce7b1be110fce699f99e9854897da",
        "type": "query",
        "version": 4
      }
    },
    "rule_name": "PowerShell Keylogging Script",
    "sha256": "6bc0cf9d4c533e8088498db20d276e4d852ce7b1be110fce699f99e9854897da",
    "type": "query",
    "version": 103

Comment on lines +245 to +246
if exclude_version_update:
buffer_int -= 1
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to make sure we dont also include the rule version autobump

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if excluding version update, shouldn't the buffer be skipped too?

@Mikaayenson Mikaayenson changed the title 1935 fr support forked rules [FR] Support forked rules with 100 version buffer space Sep 19, 2022
@Mikaayenson
Copy link
Contributor Author

Mikaayenson commented Sep 19, 2022

Update Sep 19 2022

Not only was this out of date with the latest code, but also highlights new concerns based on after all of our recent updates (forking all rules, etc.).

In order to actually use this workflow, rule developers would need to:

  1. Go to the old branch (no no) since we've already forked all to 8.3
  2. Make rule changes on that branch (no no)
  3. Run the lock versions script to make sure the previous entry is added and all the versions are correct.
  4. Push from that branch (big no no)

This seems like a rare edge case that we would not want rule developers to do.

I removed the route C option which calculates version space between a forked version and a version not adjacent to the forked version. E.g. (manually edited for example purposes)

  "000047bb-b27a-47ec-8b62-ef1a5d2c9e19": {
    "min_stack_version": "8.4",
    "previous": {
      "7.16": {
        "max_allowable_version": 99,
        "rule_name": "Attempt to Modify an Okta Policy Rule",
        "sha256": "fc9d05639917fdd13a3a474200a618648fe3dbd6fbc059714179e692544d1354",
        "type": "query",
        "version": 9
      },
      "8.0": {. # The plan is not to support this as we would have to add a new old fork in between versions
        "max_allowable_version": 7.16["max_allowable_version"],
        "rule_name": "Attempt to Modify an Okta Policy Rule",
        "sha256": some new hash,
        "type": "query",
        "version": (7.16["max_allowable_version"] - 7.16["version"])  / 2
      }
      "8.3": {
        "max_allowable_version": 199,
        "rule_name": "Attempt to Modify an Okta Policy Rule",
        "sha256": "dedf2a77f86a3ecebeba40e8a1f54e713510e09384f2ca228c8adb9cc6322490",
        "type": "query",
        "version": 100
      }
    },
    "rule_name": "Attempt to Modify an Okta Policy Rule",
    "sha256": "533da2247bc44a3b45e2bcd6a798f9afae92fde1d5a6e8e99c1f0ebfcb9b0e2d",
    "type": "query",
    "version": 200. # automatic 100 bump supported
  },

Copy link
Contributor

@brokensound77 brokensound77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good, but lets hold off merging until more discussions

Mikaayenson and others added 2 commits September 29, 2022 15:02
punt for #2333

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>
Copy link
Contributor

@terrancedejesus terrancedejesus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me.

@shashank-elastic
Copy link
Contributor

Spent a good amount of time catching up with the thread of conversations and the solution implemented. This LGTM

@terrancedejesus
Copy link
Contributor

Tested locally with a rule and made min-stack changes along with changes to the file before updating version lock.

Results
  "07b5f85a-240f-11ed-b3d9-f661ea17fbce": {
    "min_stack_version": "8.5",
    "previous": {
      "8.3": {
        "max_allowable_version": 101,
        "rule_name": "Google Drive Ownership Transferred via Google Workspace",
        "sha256": "1c82ea9b65fada4ec684045bd8b3e5eaa0730b35b41ddef3dd151ff26a9d6be9",
        "type": "query",
        "version": 2
      },
      "8.4": {
        "max_allowable_version": 201,
        "rule_name": "Google Drive Ownership Transferred via Google Workspace",
        "sha256": "d8abcf3567e56575fa27e0adaf27d0068cf6752ef8e3d06bc8984daf6d0e5ded",
        "type": "query",
        "version": 102
      }
    },
    "rule_name": "Google Drive Ownership Transferred via Google Workspace",
    "sha256": "015e684dd3c94cb01e6b581ab436b09786077963b6dfdd39b881cfeb35750344",
    "type": "query",
    "version": 202
  },

Copy link
Contributor

@brokensound77 brokensound77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing looked good, but to verify, run the lock workflow and we can review that after merge

@Mikaayenson Mikaayenson merged commit 7b596c7 into main Oct 14, 2022
@Mikaayenson Mikaayenson deleted the 1935-fr-support-forked-rules branch October 14, 2022 18:45
protectionsmachine pushed a commit that referenced this pull request Oct 14, 2022
protectionsmachine pushed a commit that referenced this pull request Oct 14, 2022
protectionsmachine pushed a commit that referenced this pull request Oct 14, 2022
protectionsmachine pushed a commit that referenced this pull request Oct 14, 2022
protectionsmachine pushed a commit that referenced this pull request Oct 14, 2022
protectionsmachine pushed a commit that referenced this pull request Oct 14, 2022
protectionsmachine pushed a commit that referenced this pull request Oct 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport: auto enhancement New feature or request python Internal python for the repository v8.3.0 Rules for 8.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FR] Support Forked Rules

5 participants