[Bug] Pre-Built Detection Rules Package Versioning

[terrancedejesus]

Overview

Initial Conversation: https://elastic.slack.com/archives/C02USDK55AQ/p1660833389191059

We recently released 8.2.1 and 7.16.4 pre-built detection rule integration packages for 8.4. During 8.4 dev, @brokensound77 @Mikaayenson and myself were tasked with adding three new fields to rules.

The code was developed, tested and implemented which dynamically builds the three fields during the building of the package. This only occurs if the stack version referenced in packages.yaml is version 8.3 or above.

Our release process involves checking out each least-compatible stack for 7 and 8 series that we support and then building the package from there that is then released to Kibana and Integrations.

Locked commit used: #2236

Therefore, our recent process did the following in short:

Kibana Update

1. checkout 8.4
2. checkout locked version commit hash
3. build 8.4 package
4. push updates to Kibana

The Kibana update did receive the rules with the additional fields as expected and tested in the updated PR. This is because we built from the 8.4 branch so the 3 fields were added as they met the minimum stack requirement of 8.3.

OOB Integrations Update

7 Series

1. checkout 7.16
2. checkout locked version commit hash
3. build 7.16 package
4. push package to integrations

8 Series

1. checkout 8.2 (we only build from -2 on latest series)
2. checkout locked version commit hash
3. build 8.2 package
4. push package to integrations

Since we built from 7.16 and 8.2, the additional fields did not get added to the packages and went into production. This is not a bug as the code is working as expected, however, we will need to release an 8.3.1 package (building from 8.3 branch) for these fields to be added to the package.

Versioning
When version locking occurred and was backported, the process started at 7.16 going to main. If a detection rule had changes, a version bump was given and the changes continued to the next branch, doing the lock versions again. Once 8.3 branch was reached, the new fields were added and some rules received a double bump if they had a change during 8.4 dev. If a rule did not have a change, they still received a single bump because the fields were added in 8.3. This caused all rules to receive a version increase.

This causes an issue if we build with 8.3 if we were to release it immediately as there are no changes to the hash and thus no new versions. Thus we need to manually bump the versions and check forked (previous) versions in the version lock file.

Overall, the tasks are as follows:

• Add min_stack_version of 8.3 to all rules | min_stack all rules to 8.3 #2259
• Lock versions from workflow | Lock versions for releases: 7.16,8.0,8.1,8.2,8.3,8.4 #2261
• Adjust version.lock.json file to change current rules to 100
• Add tests for new build-time fields requiring min_stack_version defined | Add test that newly introduced build-time fields for a min_stack for … #2262
• Release v8.3.1 package | https://epr.elastic.co/package/security_detection_engine/8.3.1/

[terrancedejesus]

@brokensound77 @Mikaayenson

From our discussion, is this what we are looking to provide with the version lock file. A couple of things, we did not include the min_stack_version key as these would be hard-forked then and changes would not be backported, SHA256 hashes in the 8.2 previous entries are the same as current, current versions were bumped by one.

Example versions lock file: version.lock.json.txt

Here is the output from doing a build with update version locks.

[+] Building package 8.5
 - 10 rules excluded from package
Rule changes detected!
 - 11 changed rules
 - 0 new rules
 - 1 newly deprecated rules
run `build-release --update-version-lock` to update version.lock.json and deprecated_rules.json
Rule changes detected!
 - 11 changed rules
 - 0 new rules
 - 1 newly deprecated rules
Detailed changes: 
  A: 3115bd2c-0baa-4df0-80ea-45e474b5ef93, new version: 5
    - min_stack_version added: 7.16
  A: 493834ca-f861-414c-8602-150d5505b777, new version: 4
    - min_stack_version added: 7.16
  A: 9c260313-c811-4ec8-ab89-8f6530e0246c, new version: 11
    - min_stack_version added: 8.2
  A: 699e9fdb-b77c-4c01-995c-1c15019b9c43, new version: 5
    - min_stack_version added: 8.0
  A: 0c9a14d9-d65d-486f-9b5b-91e4e6b22bd0, new version: 5
    - min_stack_version added: 8.0
  A: 3f0e5410-a4bf-4e8c-bcfc-79d67a285c54, new version: 4
    - min_stack_version added: 7.16
  A: c5f81243-56e0-47f9-b5bb-55a5ed89ba57, new version: 4
    - min_stack_version added: 7.16
  A: cf549724-c577-4fd6-8f9b-d1b8ec519ec0, new version: 15
    - min_stack_version added: 8.0
  A: 93e63c3e-4154-4fc6-9f86-b411e0987bbf, new version: 15
    - min_stack_version added: 8.0
  A: cad4500a-abd7-4ef3-b5d3-95524de7cfe1, new version: 16
    - min_stack_version added: 8.0
  A: 785a404b-75aa-4ffd-8be5-3334a5a544dd, new version: 15
    - min_stack_version added: 8.0
  A: 68994a6c-c7ba-4e82-b476-26a26877adf6, new version: 15
    - min_stack_version added: 8.0
  A: acbc8bb9-2486-49a8-8779-45fb5f9a93ee, new version: 15
    - min_stack_version added: 8.0
  A: ad3f2807-2b3e-47d7-b282-f84acbbe14be, new version: 15
    - min_stack_version added: 8.0
  A: a99f82f5-8e77-4f8b-b3ce-10c0f6afbc73, new version: 16
    - min_stack_version added: 8.0
  A: 6f435062-b7fc-4af9-acea-5b1ead65c5a5, new version: 15
    - min_stack_version added: 8.0
  A: e555105c-ba6d-481f-82bb-9b633e7b4827, new version: 16
    - min_stack_version added: 8.0
  A: 12a2f15d-597e-4334-88ff-38a02cb1330b, new version: 2
    - min_stack_version added: 8.2
  A: 14de811c-d60f-11ec-9fd7-f661ea17fbce, new version: 3
    - min_stack_version added: 8.2
  A: 65f9bccd-510b-40df-8263-334f03174fed, new version: 2
    - min_stack_version added: 8.2
  A: 764c8437-a581-4537-8060-1fdb0e92c92d, new version: 2
    - min_stack_version added: 8.2
  A: 12cbf709-69e8-4055-94f9-24314385c27e, new version: 2
    - min_stack_version added: 8.2
  A: df7fda76-c92b-4943-bc68-04460a5ea5ba, new version: 2
    - min_stack_version added: 8.2
  A: 2abda169-416b-4bb3-9a6b-f8d239fd78ba, new version: 2
    - min_stack_version added: 8.2
  A: c7908cac-337a-4f38-b50d-5eeb78bdb531, new version: 2
    - min_stack_version added: 8.2
  A: b9666521-4742-49ce-9ddc-b8e84c35acae, new version: 11
    - min_stack_version added: 7.16
  A: d76b02ef-fc95-4001-9297-01cb7412232f, new version: 10
    - min_stack_version added: 8.2
  A: 99dcf974-6587-4f65-9252-d866a3fdfd9c, new version: 4
    - min_stack_version added: 7.16
  A: d7d5c059-c19a-4a96-8ae3-41496ef3bcf9, new version: 3
    - min_stack_version added: 7.16
  A: e26aed74-c816-40d3-a810-48d6fbd8b2fd, new version: 4
    - min_stack_version added: 7.16
  A: 9d302377-d226-4e12-b54c-1906b5aec4f6, new version: 6
    - min_stack_version added: 8.3
  A: 1faec04b-d902-4f89-8aff-92cd9043c16f, new version: 6
    - min_stack_version added: 8.3
  A: 4330272b-9724-4bc6-a3ca-f1532b81e5c2, new version: 7
    - min_stack_version added: 8.3
  A: abae61a8-c560-4dbd-acca-1e1438bff36b, new version: 6
    - min_stack_version added: 8.3
  A: df197323-72a8-46a9-a08e-3f5b04a4a97a, new version: 6
    - min_stack_version added: 8.3
  A: d4af3a06-1e0a-48ec-b96a-faf2309fae46, new version: 4
    - min_stack_version added: 8.3
  A: f9590f47-6bd5-4a49-bd49-a2f886476fb9, new version: 4
    - min_stack_version added: 8.3
  A: c28c4d8c-f014-40ef-88b6-79a1d67cd499, new version: 4
    - min_stack_version added: 8.3
  A: 5c983105-4681-46c3-9890-0c66d05e776b, new version: 4
    - min_stack_version added: 8.3
  A: 59756272-1998-4b8c-be14-e287035c4d10, new version: 4
    - min_stack_version added: 8.3
  A: 1781d055-5c66-4adf-9d60-fc0fa58337b6, new version: 7
    - min_stack_version added: 8.3
  A: 745b0119-0560-43ba-860a-7235dd8cee8d, new version: 3
    - min_stack_version added: 7.16
  A: d4b73fa0-9d43-465e-b8bf-50230da6718b, new version: 3
    - min_stack_version added: 7.16
  A: 138c5dd5-838b-446e-b1ac-c995c7f8108a, new version: 5
    - min_stack_version added: 7.16
  A: b347b919-665f-4aac-b9e8-68369bf2340c, new version: 10
    - min_stack_version added: 8.3
  A: 1781d055-5c66-4adf-9c59-fc0fa58336a5, new version: 10
    - min_stack_version added: 8.3
  A: 1781d055-5c66-4adf-9e93-fc0fa69550c9, new version: 8
    - min_stack_version added: 8.3
  A: 52afbdc5-db15-485e-bc24-f5707f820c4b, new version: 8
    - min_stack_version added: 8.3
  A: 3c7e32e6-6104-46d9-a06e-da0f8b5795a0, new version: 7
    - min_stack_version added: 8.3
  A: ba342eb2-583c-439f-b04d-1fdd7c1417cc, new version: 9
    - min_stack_version added: 8.3
  A: 647fc812-7996-4795-8869-9c4ea595fe88, new version: 10
    - min_stack_version added: 8.3
  A: 46f804f5-b289-43d6-a881-9387cf594f75, new version: 10
    - min_stack_version added: 8.3
  A: 6d448b96-c922-4adb-b51c-b767f1ea5b76, new version: 13
    - min_stack_version added: 8.3
  A: 445a342e-03fb-42d0-8656-0367eb2dead5, new version: 8
    - min_stack_version added: 8.3
  A: 6e40d56f-5c0e-4ac6-aece-bee96645b172, new version: 10
    - min_stack_version added: 8.3
  A: 0b29cab4-dbbd-4a3f-9e8e-1287c7c11ae5, new version: 8
    - min_stack_version added: 8.3
  A: 1781d055-5c66-4adf-9c71-fc0fa58338c7, new version: 7
    - min_stack_version added: 8.3
  A: 1e9fc667-9ff1-4b33-9f40-fefca8537eb0, new version: 4
    - min_stack_version added: 8.3
  A: 1781d055-5c66-4adf-9d82-fc0fa58449c8, new version: 7
    - min_stack_version added: 8.3
  A: cd66a419-9b3f-4f57-8ff8-ac4cd2d5f530, new version: 6
    - min_stack_version added: 8.3
  A: 8c1bdde8-4204-45c0-9e0c-c85ca3902488, new version: 15
    - min_stack_version added: 8.2
  A: 34fde489-94b0-4500-a76f-b8a157cf9269, new version: 13
    - min_stack_version added: 8.2
  A: 00140285-b827-4aee-aa09-8113f58a08f3, new version: 9
    - min_stack_version added: 7.16
  A: f2f46686-6f3c-4724-bd7d-24e31c70f98f, new version: 11
    - min_stack_version added: 8.2
  A: a4c7473a-5cb4-4bc1-9d06-e4a75adbc494, new version: 4
    - min_stack_version added: 7.16
  A: 850d901a-2a3c-46c6-8b22-55398a01aad8, new version: 5
    - min_stack_version added: 7.16
  A: 0f93cb9a-1931-48c2-8cd0-f173fd3e5283, new version: 6
    - min_stack_version added: 7.16
  A: 4630d948-40d4-4cef-ac69-4002e29bc3db, new version: 15
    - min_stack_version added: 8.2
  A: edf8ee23-5ea7-4123-ba19-56b41e424ae3, new version: 9
    - min_stack_version added: 8.2
  A: 8b2b3a62-a598-4293-bc14-3d5fa22bb98f, new version: 7
    - min_stack_version added: 7.16
  A: fd70c98a-c410-42dc-a2e3-761c71848acf, new version: 16
    - min_stack_version added: 8.2
  A: 2856446a-34e6-435b-9fb5-f8f040bfa7ed, new version: 15
    - min_stack_version added: 7.16
  A: fd7a6052-58fa-4397-93c3-4795249ccfa2, new version: 15
    - min_stack_version added: 8.2
  A: fb02b8d3-71ee-4af1-bacd-215d23f17efa, new version: 14
    - min_stack_version added: 7.16
  A: 852c1f19-68e8-43a6-9dce-340771fe1be3, new version: 9
    - min_stack_version added: 7.16
  A: 6839c821-011d-43bd-bd5b-acff00257226, new version: 7
    - min_stack_version added: 7.16
  A: afcce5ad-65de-4ed2-8516-5e093d3ac99a, new version: 14
    - min_stack_version added: 7.16
  A: 54902e45-3467-49a4-8abc-529f2c8cfb80, new version: 10
    - min_stack_version added: 8.2
  A: 97fc44d3-8dae-4019-ae83-298c3015600f, new version: 9
    - min_stack_version added: 8.2
  A: 58c6d58b-a0d3-412d-b3b8-0981a9400607, new version: 8
    - min_stack_version added: 7.16
  A: ee5300a7-7e31-4a72-a258-250abb8b3aa1, new version: 9
    - min_stack_version added: 7.16
Updated /Users/tdejesus/code/src/detection-rules/detection_rules/etc/version.lock.json file
Updated /Users/tdejesus/code/src/detection-rules/detection_rules/etc/deprecated_rules.json file
Package saved to: /Users/tdejesus/code/src/detection-rules/releases/8.5
- sha256: 52cc6b66d3d7c2babc7a67846319e3e8adbdfd4405ac234b4493c57a86fcf21d
- 679 rules included

If I understand correctly, we want to get a PR going with these version lock changes and backport this to 7.16. Then checkout 8.3 branch, checkout the locked versions commit previously used, build the package from 8.3 branch and follow the normal release process to integrations and then package-storage?

Code used:

import json
versions = json.load(open("/Users/tdejesus/code/src/detection-rules/detection_rules/etc/version.lock.json"))
for k in versions.keys():
    previous_obj = {"rule_name": "","sha256": "","type": "","version": 0}
    previous_obj["version"] = versions[k]["version"]
    versions[k]["version"] += 1 # bump version by one
    previous_obj["rule_name"] = versions[k]["rule_name"]
    previous_obj["sha256"] = versions[k]["sha256"]
    previous_obj["type"] = versions[k]["type"]

    if "previous" in versions[k].keys():
        versions[k]["previous"].setdefault("8.2", previous_obj)
    else:
        versions[k].setdefault("previous",{"8.2":previous_obj})

with open("/Users/tdejesus/code/src/detection-rules/detection_rules/etc/version.lock.2.json", "w") as f:
    json.dump(versions,f)

[terrancedejesus]

After further discussion with @brokensound77 and @Mikaayenson we have decided that all rules will need to be hard-forked (add a min_stack_version) to them if a new build-time field is identified during build. This applies to new rules and existing.

A buffer of ~100 will be given to forked rules in regards to versioning so these rules can still be updated whereas the current version starts at +100 therefore any changes that have to go directly to a forked rule will have versioning space available and not overlap the current. Once this space is up, we must deprecate the rule and create a new one.