Allow routing for integrations that are not input packages #6340

gsantoro · 2023-05-26T12:21:09Z

What does this PR do?

This PR add permissions to reroute logs to logs-*-* for integrations that are not input packages.

The full list of packages to edit is:

kubernetes.container_logs
system.syslog
activemq.log
auditd.log
aws.cloudwatch_logs
aws.ec2_logs
kafka.log
docker.container_logs

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Relates Allow routing for integrations that are not input packages #6255

Screenshots

elasticmachine · 2023-05-26T12:36:04Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-06-13T12:11:58.706+0000
Duration: 52 min 27 sec

Test stats 🧪

Test	Results
Failed	0
Passed	531
Skipped	4
Total	535

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-05-26T12:36:13Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`22/22`)	💚
Files	95.833% (`23/24`)	👎 -4.167
Classes	95.833% (`23/24`)	👎 -4.167
Methods	84.645% (`441/521`)	👎 -4.244
Lines	90.563% (`12466/13765`)	👎 -9.437
Conditionals	100.0% (`0/0`)	💚

tommyers-elastic

CSP integration changes look good to me - only thing on my mind is the future - someone looking at this file without the context of routing rules etc, they might be able to look up what these settings do, but there is some implicit coupling to other parts of the system here that i think would be good to call out 'at the source'. can we comment what the settings are for?

packages/activemq/changelog.yml

gsantoro · 2023-05-26T14:00:39Z

about

CSP integration changes look good to me - only thing on my mind is the future - someone looking at this file without the context of routing rules etc, they might be able to look up what these settings do, but there is some implicit coupling to other parts of the system here that i think would be good to call out 'at the source'. can we comment what the settings are for?

I think that we could add a comment like

# Ensures agents have permissions to write data to `logs-nginx.*-*`

like the one you can see at here

packages/activemq/changelog.yml

felixbarny · 2023-05-31T17:37:34Z

I've updated the versions so that it increments the minor version, not the bug fix version now.

felixbarny · 2023-06-01T07:13:12Z

Can we bump the minor version of the integration rather than the bugfix, as this is adding features we want to follow the semantic versioning.

Done.

Please test any of the packages on an older version of the stack, while we can assume that fleet should ignore those settings, try to take a look at the oldest version of the stack any of these packages support (8.x, 7.x?), and see if that is still the case. If not, please bump the min kibana version of that package.

This will need to wait until Giuseppe is back. @joshdover, since you implemented the flags in fleet, how big do you think is the risk that packages with these flags cause issues when used in older stack versions?

I assume these settings are now documented?

These settings are documented in the package spec: https://github.com/elastic/package-spec/blob/e29dd918bc5a6e81cb0b36ae8a3c4b4738f1d68e/spec/integration/data_stream/manifest.spec.yml#L304-L309

I was a bit unsure why each package has to define the fact that they are allowed to have dynamic settings,

While dynamic dataset/namespace is enabled by default for input packages, regular integrations need to opt-in for each data stream. That's because not all integrations will want to open up the API key permissions. The regular API key permissions are very narrowly scoped to the exact dataset and namespace, for example logs-nginx.access.default. The dynamic_dataset flag results in the dataset part to be a wildcard in the API key permissions. So if a package sets both dynamic_dataset and dynamic_namespace, the API key permissions are expanded to logs-*-*.

will this then show dataset settings in the UI, or is it something else?

No, this will not show in any UI.

gsantoro · 2023-06-13T09:12:34Z

I have tested now the rerouting of logs for the following use cases:

docker.container_logs in 8.9.0-SNAPSHOT Elastic stack
docker.container_logs in 8.8.0 Elastic stack

gsantoro · 2023-06-13T12:11:41Z

/test

elasticmachine · 2023-06-13T12:13:20Z

💚 Build Succeeded

Buildkite Build
Commit: 22eb873

History

💚 Build #21 succeeded 22eb873

cc @gsantoro

elasticmachine · 2023-06-13T13:49:56Z

Package activemq - 0.10.0 containing this change is available at https://epr.elastic.co/search?package=activemq

elasticmachine · 2023-06-13T13:54:34Z

Package auditd - 3.9.0 containing this change is available at https://epr.elastic.co/search?package=auditd

elasticmachine · 2023-06-13T13:58:56Z

Package aws - 1.44.0 containing this change is available at https://epr.elastic.co/search?package=aws

elasticmachine · 2023-06-13T14:03:23Z

Package docker - 2.6.0 containing this change is available at https://epr.elastic.co/search?package=docker

elasticmachine · 2023-06-13T14:08:55Z

Package kafka - 1.7.0 containing this change is available at https://epr.elastic.co/search?package=kafka

elasticmachine · 2023-06-13T14:13:37Z

Package kubernetes - 1.42.0 containing this change is available at https://epr.elastic.co/search?package=kubernetes

elasticmachine · 2023-06-13T14:17:57Z

Package system - 1.33.0 containing this change is available at https://epr.elastic.co/search?package=system

ruflin · 2023-06-15T10:38:29Z

packages/system/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 1.0.0
 name: system
 title: System
-version: 1.32.0-beta.2
+version: 1.33.0


@gsantoro With this change you accidentally release TSDB which was still in beta!

@gsantoro Are we considering System Integration TSDB as GA now or are we planning to revert the change ?

TSDB in system is now approved to be in GA. #6607

* fix system test for kubernetes integration for k8s v1.27.0 * minor changes from formatting * syslong changes * revert a merge conflict * add permissions to selected list of datastreams * update PR id * fix merge conflict with main * Increment minor version instead of bugfix version * added comments for new settings * update activemq version --------- Co-authored-by: Felix Barnsteiner <felix.barnsteiner@elastic.co>

gsantoro added 7 commits May 24, 2023 17:10

fix system test for kubernetes integration for k8s v1.27.0

a9d0b0b

Merge branch 'main' into feature/k8s_1.27.0

ac6dbd2

minor changes from formatting

4bb4ecb

Merge remote-tracking branch 'origin' into feature/allow_routing

42fbebb

syslong changes

b545962

Merge remote-tracking branch 'origin' into feature/allow_routing

26a3305

revert a merge conflict

3d1070e

gsantoro added the enhancement New feature or request label May 26, 2023

gsantoro requested a review from a team as a code owner May 26, 2023 12:21

gsantoro self-assigned this May 26, 2023

gsantoro requested review from rdner and cmacknz May 26, 2023 12:21

add permissions to selected list of datastreams

62ce182

gsantoro requested review from a team as code owners May 26, 2023 12:25

gsantoro requested review from gizas and tetianakravchenko May 26, 2023 12:25

gsantoro added draft Draft and removed draft Draft labels May 26, 2023

update PR id

c162ee9

gsantoro added 2 commits May 26, 2023 14:45

Merge branch 'main' into feature/allow_routing

4368286

fix merge conflict with main

1f4e99a

tommyers-elastic approved these changes May 26, 2023

View reviewed changes

gizas reviewed May 26, 2023

View reviewed changes

packages/activemq/changelog.yml Outdated Show resolved Hide resolved

gizas approved these changes May 26, 2023

View reviewed changes

leehinman requested changes May 31, 2023

View reviewed changes

packages/activemq/changelog.yml Outdated Show resolved Hide resolved

felixbarny added 2 commits May 31, 2023 19:27

Merge remote-tracking branch 'origin/main' into feature/allow_routing

0a2c483

Increment minor version instead of bugfix version

e894664

leehinman approved these changes May 31, 2023

View reviewed changes

Merge remote-tracking branch 'origin/main' into feature/allow_routing

4dd7eae

gsantoro added 5 commits June 12, 2023 10:31

Merge branch 'main' into feature/allow_routing

6595db6

added comments for new settings

bb9b2fa

update activemq version

cadccb7

Merge branch 'main' into feature/allow_routing

70e0f90

Merge branch 'main' into feature/allow_routing

22eb873

gsantoro merged commit 835c3c7 into elastic:main Jun 13, 2023
4 checks passed

gsantoro deleted the feature/allow_routing branch June 13, 2023 13:29

felixbarny linked an issue Jun 13, 2023 that may be closed by this pull request

Allow routing for integrations that are not input packages #6255

Closed

felixbarny mentioned this pull request Jun 13, 2023

Allow routing for integrations that are not input packages #6255

Closed

felixbarny mentioned this pull request Jun 14, 2023

Enable routing data to different data streams elastic/apm-server#10991

Closed

ruflin reviewed Jun 15, 2023

View reviewed changes

P1llus mentioned this pull request Oct 25, 2023

[UDP/TCP] Add reroute processor support #8294

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow routing for integrations that are not input packages #6340

Allow routing for integrations that are not input packages #6340

gsantoro commented May 26, 2023

elasticmachine commented May 26, 2023 •

edited

Build stats

Test stats 🧪

elasticmachine commented May 26, 2023 •

edited

tommyers-elastic left a comment •

edited

gsantoro commented May 26, 2023

felixbarny commented May 31, 2023

felixbarny commented Jun 1, 2023

gsantoro commented Jun 13, 2023

gsantoro commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

ruflin Jun 15, 2023

ishleenk17 Jun 28, 2023 •

edited

lalit-satapathy Jun 28, 2023

Allow routing for integrations that are not input packages #6340

Allow routing for integrations that are not input packages #6340

Conversation

gsantoro commented May 26, 2023

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented May 26, 2023 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 26, 2023 • edited

🌐 Coverage report

tommyers-elastic left a comment • edited

Choose a reason for hiding this comment

gsantoro commented May 26, 2023

felixbarny commented May 31, 2023

felixbarny commented Jun 1, 2023

gsantoro commented Jun 13, 2023

gsantoro commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

💚 Build Succeeded

History

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

elasticmachine commented Jun 13, 2023

ruflin Jun 15, 2023

Choose a reason for hiding this comment

ishleenk17 Jun 28, 2023 • edited

Choose a reason for hiding this comment

lalit-satapathy Jun 28, 2023

Choose a reason for hiding this comment

elasticmachine commented May 26, 2023 •

edited

elasticmachine commented May 26, 2023 •

edited

tommyers-elastic left a comment •

edited

ishleenk17 Jun 28, 2023 •

edited