Set frozen/cold tier for Security ML integrations

[susan-shu-c]

Proposed commit message

Per discussion brought up by @SourinPaul , here is a draft PR illustrating what the packages would look like with the data frozen tier and data cold tier filtered out. The PR also lints/formats these files with elastic-package to make sure the additional field conforms to style guides.

However, we must first discuss with the authors of those respective packages if that is the expected behavior before merging. @ajosh0504 @sodhikirti07 [Edit] We've discussed these changes and gotten approval.

Affected packages:

• ProblemChild
• DGA
• LMD
• DED

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Additional info:

Apoorva - Agree with the suggestion [...] to exclude cold/frozen tier from the module query. We seem to be doing it in our OOTB jobs in Kibana as well, so looks like the way to go to prevent ML jobs from hitting those tiers. Eg: https://github.com/elastic/kibana/blob/main/x-pack/plugins/ml/server/models/data_recognizer/modules/security_network/manifest.json#L17

How to test this PR locally

Related issues

Screenshots

[elasticmachine]

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-11-15T15:38:15.927+0000

• Duration: 362 min 22 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1
Skipped	0
Total	1

Steps errors

Expand to view the steps failures

Google Storage Download

• Took 0 min 0 sec . View more details here

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[ajosh0504]

Thanks for getting this started! Changes look good, but you'd have to update the package versions as well, specifically, you'd add a new entry to the changelog and bump the version in the manifest file. See this PR for reference.

[susan-shu-c]

Re: @ajosh0504
Ah oops, forgot that! I will do it if the changes look good to Kirti for her packages too

[sodhikirti07]

+1 to Apoorva, you'll need to bump the versions in changelog, manifest and for LMD transform fleet version as well! Otherwise, the filters are looking good.

Comments on draft PR addressed, now changing to ready for review

[droberts195]

LGTM

Thanks for adding these extra conditions.

[sodhikirti07]

Changes lgtm!

[susan-shu-c]

/test

[susan-shu-c]

Buildkite's showing this error for DGA:

023/11/15 20:00:39 DEBUG Build zipped package
--
  | 2023/11/15 20:00:39 DEBUG Compress using archiver.Zip (destination: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1700078074357728088/elastic/integrations/build/packages/dga-2.0.1.zip)
  | 2023/11/15 20:00:39 DEBUG Create work directory for archiving: /tmp/elastic-package-1799526892/dga-2.0.1
  | 2023/11/15 20:00:39 DEBUG Validating built .zip package (path: /opt/buildkite-agent/builds/bk-agent-prod-gcp-1700078074357728088/elastic/integrations/build/packages/dga-2.0.1.zip)
  | 2023/11/15 20:00:39 DEBUG POST https://127.0.0.1:5601/api/fleet/epm/packages
  | Error: can't install the package: could not zip-install package; API status code = 429; response body = {"statusCode":429,"error":"Too Many Requests","message":"[parent] Data too large, data for [<http_request>] would be [1029186428/981.5mb], which is larger than the limit of [1020054732/972.7mb], real usage: [935794648/892.4mb], new bytes reserved: [93391780/89mb], usages [inflight_requests=93391780/89mb, request=0/0b, fielddata=288/288b, eql_sequence=0/0b, model_inference=246784104/235.3mb]: circuit_breaking_exception\n\tRoot causes:\n\t\tcircuit_breaking_exception: [parent] Data too large, data for [<http_request>] would be [1029186428/981.5mb], which is larger than the limit of [1020054732/972.7mb], real usage: [935794648/892.4mb], new bytes reserved: [93391780/89mb], usages [inflight_requests=93391780/89mb, request=0/0b, fielddata=288/288b, eql_sequence=0/0b, model_inference=246784104/235.3mb]"}
  | [dga] run_tests_package failed
  | Collect Elastic stack logs
  | 2023/11/15 20:00:46  WARN CommitHash is undefined, in both /opt/buildkite-agent/.elastic-package/version and the compiled binary, config may be out of date.

When I spin up a local stack to test, I can install DGA however:

The other packages are fine as well on Buildkite. Might test again later to see if the tests can succeed.

[droberts195]

Buildkite's showing this error for DGA

It looks like BuildKite is testing with a small ES node (1GB JVM heap) and there's too much going on simultaneously for that size of node. I don't think the changes in this PR caused this.

[susan-shu-c]

Sounds good, won't merge today (Nov 14) end of day, but aiming for tomorrow morning (Nov 15) to try again and merge. But otherwise this should be good.

[elasticmachine]

Package ded - 2.1.1 containing this change is available at https://epr.elastic.co/search?package=ded

[elasticmachine]

Package dga - 2.0.1 containing this change is available at https://epr.elastic.co/search?package=dga

[elasticmachine]

Package lmd - 2.1.2 containing this change is available at https://epr.elastic.co/search?package=lmd

[elasticmachine]

Package problemchild - 2.1.1 containing this change is available at https://epr.elastic.co/search?package=problemchild