Osquery_manager: Upgrade osquery_manager for serverless, pick up ECS 8.10.0

[aleksmaus]

What does this PR do?

Upgrades osquery_manager for serverless specs
Picks up ECS 8.10.0

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Screenshots

Tested following the serverless opt-in packages guide.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-09-29T14:32:49.723+0000

• Duration: 17 min 26 sec

Test stats 🧪

Test	Results
Failed	0
Passed	9
Skipped	0
Total	9

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[aleksmaus]

hmmm, the error in CI

[2023-09-20T16:32:46.066Z] Error response from daemon: manifest for docker.elastic.co/kibana/kibana:8.10.0 not found: manifest unknown: manifest unknown

[2023-09-20T16:32:46.066Z] Error: failed updating the stack images: updating docker images failed: running command failed: running Docker Compose pull command failed: exit status 18

[aleksmaus]

/test

[aleksmaus]

/test

[mrodm]

Please wait to merge this PR @aleksmaus , merging this branch would publish this package as GA with spec v3 but this spec is not GA yet.

We are checking how this validation is performed in elastic-package to try avoid this situation.

Thanks!!

cc @jsoriano @andrewkroh

[mrodm]

Please wait to merge this PR @aleksmaus , merging this branch would publish this package as GA with spec v3 but this spec is not GA yet.

We are checking how this validation is performed in elastic-package to try avoid this situation.

Thanks!!

cc @jsoriano @andrewkroh

@aleksmaus you could proceed with this PR, just be aware that the spec v3 is not GA yet and there could be still changes in that version (e.g. more validation rules to be applied).

Just as a note, remember to add the capabilities if this package has some special requirement.

Sorry for the inconveniences!

[aleksmaus]

@aleksmaus you could proceed with this PR, just be aware that the spec v3 is not GA yet and there could be still changes in that version (e.g. more validation rules to be applied).

Thanks!
Any advice what to do about this error in CI?

[2023-09-20T16:32:46.066Z] Error response from daemon: manifest for docker.elastic.co/kibana/kibana:8.10.0 not found: manifest unknown: manifest unknown

I tried to re-kick the PR build, didn't help

[mrodm]

@aleksmaus you could proceed with this PR, just be aware that the spec v3 is not GA yet and there could be still changes in that version (e.g. more validation rules to be applied).

Thanks! Any advice what to do about this error in CI?

[2023-09-20T16:32:46.066Z] Error response from daemon: manifest for docker.elastic.co/kibana/kibana:8.10.0 not found: manifest unknown: manifest unknown

I tried to re-kick the PR build, didn't help

mmm that kibana image was deleted, that's why the testing fails.
I think one option would be to update the kibana version condition to be ^8.10.1 @aleksmaus . The disadvantage in this case is that this version would not be available for the current 8.10.0 clusters, but I guess they would need to update to 8.10.1 due to the security issue too.

This has already been applied in at least 3 packages:

 $ git grep "\^8.10.1" | grep manifest
awsfirehose/manifest.yml:  kibana.version: "^8.10.1"
okta/manifest.yml:  kibana.version: ^8.10.1
security_detection_engine/manifest.yml:  kibana.version: ^8.10.1

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚
Classes	100.0% (0/0)	💚
Methods	25.0% (1/4)	👎 -75.0
Lines	100.0% (0/0)	💚 31.522
Conditionals	100.0% (0/0)	💚

[andrewkroh]

This looks good.

I saw a few more fields that could be changed over to use external: ecs if you wanted.

packages/osquery_manager/data_stream/result/fields/base-fields.yml:1:3 data_stream.type
packages/osquery_manager/data_stream/result/fields/base-fields.yml:4:3 data_stream.dataset
packages/osquery_manager/data_stream/result/fields/base-fields.yml:7:3 data_stream.namespace
packages/osquery_manager/data_stream/result/fields/base-fields.yml:10:3 @timestamp

[andrewkroh]

integrations/packages/osquery_manager/data_stream/result/fields/osquery.yml

Lines 2 to 3 in 5b44161

	title: Osquery result
	description: Fields related to the Osquery result

These two fields, title and description on a "group", can be removed. That are not used for anything in packages or Fleet (source).

[aleksmaus]

This has already been applied in at least 3 packages:

 $ git grep "\^8.10.1" | grep manifest
awsfirehose/manifest.yml:  kibana.version: "^8.10.1"
okta/manifest.yml:  kibana.version: ^8.10.1
security_detection_engine/manifest.yml:  kibana.version: ^8.10.1

I tried to update to ^8.10.1 the build still fails in CI

[2023-09-28T20:03:45.058Z] 2023/09/28 20:03:44 DEBUG running command: /var/lib/jenkins/workspace/est-manager_integrations_PR-7894/bin/docker-compose -f /var/lib/jenkins/workspace/est-manager_integrations_PR-7894/.elastic-package/profiles/default/stack/snapshot.yml --ansi never -p elastic-package-stack pull --quiet

[2023-09-28T20:03:50.180Z] Error response from daemon: Get "https://registry-1.docker.io/v2/": EOF

[2023-09-28T20:03:50.180Z] Error: failed updating the stack images: updating docker images failed: running command failed: running Docker Compose pull command failed: exit status 18

script returned exit code 1

[andrewkroh]

/test

[andrewkroh]

That last failure was 18 hours ago which was during a docker hub outage. That might have been the cause. Retrying.

[elasticmachine]

Package osquery_manager - 1.10.0 containing this change is available at https://epr.elastic.co/search?package=osquery_manager