[Fleet] Include space ID in namespace piece of integration data stream names

[kpollich]

In order to support data segregation as part of Fleet's space awareness project, Fleet needs to begin including the space ID to which a given policy belongs in documents generated by integrations on that policy.

To accomplish this, Fleet should allow users to include the space ID as part of the namespace value in their integrations' data streams. Generally, the namespace part of a data stream's name is intended as a "user configurable arbitrary grouping" (see docs), so in order to preserve this intent, the space ID should be included as a prefix, and a use should still be able to provide a custom namespace.

For example, if a user configures a policy in space123 with a custom namespace of my_namespace, the resulting data streams for their integrations should follow the form {type}-{dataset}-space123.

Importantly, this behavior can break existing usages of the namespace value. e.g. if a user has ILM policies or processors that rely on a certain namespace value we would break these components by changing the namespace once a policy moves to a new space. Hence, this functionality should be opt-in.

Implementation

• Add a checkbox to opt-in to space-aware namespaces in Fleet settings
• Include the space ID as a prefix for the namespace value generated in compiled agent policies
• Ensure the generated namespace prefix cannot contain invalid characters (e.g. -)
• Add documentation about this functionality

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

@kpollich @nimarezainia Trying to think that one to be future proof, if we move to agent policies belonging to multiple space, how this will work? we will only restrict the namespace in the one of the namespace.

For example with the spaces: security_global, tenant1, tenant2 how this will work?
In my opinion if I create a policy in tenant1 or tenant2 I expect namespace to be restricted to tenant1* tenant2* but for the security_global space?

[kpollich]

I think we'd have to allow for custom prefixes if we move to a multi-space policy model and rely on the user setting up their own data segregation model.

[nimarezainia]

yes I agree with @kpollich , as an extension to what you suggested @nchaulet , let's set the namespace to the space the Policy was created in but also allow for namespace to be modified.

[nchaulet]

I think we'd have to allow for custom prefixes if we move to a multi-space policy model and rely on the user setting up their own data segregation model.

@kpollich @nimarezainia Do you have any idea where that data segregation model should live?
We could introduce a per space settings saved object, this will mean an user with the settings:All permissions will have to switch to that space go to the settings tabs and here in a new section list the prefix it want to allow (the default behaviour will be no restrictions), it something that will work here?

[nimarezainia]

I think we'd have to allow for custom prefixes if we move to a multi-space policy model and rely on the user setting up their own data segregation model.

@kpollich @nimarezainia Do you have any idea where that data segregation model should live? We could introduce a per space settings saved object, this will mean an user with the settings:All permissions will have to switch to that space go to the settings tabs and here in a new section list the prefix it want to allow (the default behavior will be no restrictions), it something that will work here?

do you mean where in the UI? reference is to the namespace that is defined in Agent Policy correct? so only users that have access to modify the agent policy in that space can modify the settings and modify the namespace.

[nchaulet]

do you mean where in the UI? reference is to the namespace that is defined in Agent Policy correct? so only users that have access to modify the agent policy in that space can modify the settings and modify the namespace.

Yes where in the UI, are we going to limit the authorized namespace for datastreams. If we want to limit agent policy in space_a to write to logs-space_a*, metrics-space_a*, ... where the config for that limitation should live/and who can modify it.

It is more clear?

[nchaulet]

What I propose to do here is to introduce a new API in fleet (and the equivalent in kibana.config)

Acessible to someone with the settings:all permissions

POST kbn:/s/{spaceId}/api/fleet/space_settings
{
 "allowed_prefix_namespaces": ["test1", "test2"]
}

Then when creating/updating an agent policy/package policy we will check the prefix is allowed

Default will be null and will not restrict namespaces prefix

{
 "allowed_prefix_namespaces":  null
}

@kpollich does it make sense to you?

[kpollich]

@nchaulet - Yes having a separate API that allows for configuring the namespace prefixes makes sense to me.

@nimarezainia - This is only necessary if we decide that moving to a model that supports multi-space policies is a priority, correct? We don't need to do this immediately?

[nchaulet]

This is only necessary if we decide that moving to a model that supports multi-space policies is a priority, correct? We don't need to do this immediately?

@kpollich I think having the API allow to let user to define their own namespace prefix strategy, I think even without supporting multi-space priority it could make sense no? And the UI can hide that complexity by only having a checkbox to the current spaceId WDYT?

[kpollich]

I think having the API allow to let user to define their own namespace prefix strategy, I think even without supporting multi-space priority it could make sense no? And the UI can hide that complexity by only having a checkbox to the current spaceId WDYT?

Sure I could see this being useful as a way to cover more use cases for specialized user needs. Maybe this can close some gaps between true multi-space support for policies.

[nimarezainia]

sorry for the delay. Agree with the above that we should enable the user to set their own desired namespace if required.

We do need to consider the multi-space affinity use cases but perhaps not as the first iteration - anyhow the namespace discussion is separate to this.

[nchaulet]

@nimarezainia @kpollich I just recorded a small demo of the API part, I am wondering if it's good enough (I still will add some client validation to improve the user experience, and prefill the namespace value with an allowed value) or if we need an UI for the admin to manage the allowed namespace,

demo_restrict_namespace.mov