Feature flags: Controlled rollout of Fleet features by team

[zhumo]

Goal

We want to allow users to do canary rollouts for their features so that they can test it out for their install base before committing to a full rollout. This allows our users to validate scalability, usability, and impact of various features.

Requirements

Implement the controlled rollout as described in:

• Add controlled rollout proposal #6221
• except exclude platform- and team-based rollout
• This is a premium-only feature
• The override behavior should remain the same. The team ignores the global config. There is no merging between the two.

The config should be structured as:

The override exists at the same position as in the global options. For example, a host_settings.enable_software_inventory should exist in teams, if it exists in global. In this way, the configuration is the same between global and team configs, which is easiest for users to remember.

apiVersion: v1
kind: team
spec:
  team:
    name: Client Platform Engineering
    agent_options: { ... }
    features:
      enable_software_inventory: false # This might override a global setting that is set to true. 

NOTE: host_settings is now replaced by features. We should still support host_settings as an alias to features for backwards compatibility, but in our communications, we should use and recommend features as the canonical way forward.

Related

Children

• Implement controlled rollout of features based on teams #7312

Documentation

• Documentation for controlled rollout

Do later

• Feature flags: Controlled rollout of Fleet features by host #7203
• Feature flags: Controlled rollout of Fleet features by platform #7204

[noahtalerman]

Hey @zhumo heads up, I scheduled a call to discuss the config interface specified in the proposal.

I'd like to make sure we take the time to think through how this functionality behaves when compared to the agent_options part of configuration.

For example, in order to set different agent options for a team (override global options set in config YAML document), a user must specify agent_options in a separate teams YAML document.

This approach differs from the proposed approach for software inventory. If I'm understanding correctly, turning on software inventory for a team (override global settings) is done within the config document.

[noahtalerman]

@zhumo I propose updating host_settings to features in the config (global) and teams YAML documents.

This way, the YAML document clearly communicates that this is location for turning on/off features at the global and team level.

I also propose that the interaction (override) between features at the team level and features at the global level matches the current interaction for agent_options. I think this means that...

• When a new team is created, this team inherits the features specified at the global level.
• Team features override global features. This means that if a change is made to features at the global level, this change is not passed to any teams.

[zhumo]

@noahtalerman

I propose updating host_settings to features in the config (global) and teams YAML documents.

Makes sense to me, but to clarify, we want to still support host_settings as an undocumented alias, so as to support backwards compatibility?

When a new team is created, this team inherits the features specified at the global level.
Team features override global features. This means that if a change is made to features at the global level, this change is not passed to any teams.

I know what you mean here, but I would rephrase it as:

• When a new team is created, the new team takes a snapshot of the global config
• The team ignores the global config and only applies the team config
• This means that if a change is made to features at the global level, this change is not passed to any teams.

WDYT?