Proposal: Freeze v1 Interfaces and Adopt Nested Multi-Runner Configuration

[edersonbrilhante]

The multi-runner module currently exposes many flat variables that are tightly coupled to the EC2 provider. The current multi_runner_config shape is also closely coupled to the EC2 implementation.

As the project adds new compute providers, scale-set orchestration, and alternatives to SSM such as DynamoDB, continuing to add provider-specific flat variables will make the module increasingly difficult to maintain and extend.

The recent work in #5251, #5260, #5299, and #5295 appears to be moving toward provider-neutral boundaries for compute, orchestration, and storage.

Would it make sense to move toward one nested multi-runner configuration model and eventually deprecate both the single-runner configuration and the current flat multi_runner_config variables?

I would also suggest freezing both v1 configuration models now. We should stop adding support for new features to the single-runner configuration and the flat multi_runner_config v1, except for critical bug and security fixes. New capabilities should be implemented only in the nested configuration model.

A possible migration path could be:

1. Keep the existing v1 interfaces available for existing users.
2. Stop adding new feature support to the v1 interfaces, except for critical bug and security fixes.
3. Make the multi-runner configuration the primary interface and deprecate the single-runner configuration in the root module.
4. Translate the legacy single-runner and flat multi-runner inputs into the new nested model.
5. Introduce shared global defaults and lane-specific overrides.
6. Make the nested model the default in a future major version.
7. Eventually remove the legacy single-runner interface and flat multi_runner_config variables.

The nested model could organize configuration by responsibility and avoid adding more variables such as prefix_something for every new feature. Shared settings could be defined once under global, while each entry in multi_runner_config would represent an independent runner configuration.

For example:

global = {
  runner                 = {}
  github                 = {}
  compute_provider       = {}
  orchestration_provider = {}
  storage                = {}
  observability          = {}
}

multi_runner_config = {
  build = {
    runner                 = {}
    github                 = {}
    compute_provider       = {}
    orchestration_provider = {}
    storage                = {}
    observability          = {}
  }

  test = {
    runner                 = {}
    github                 = {}
    compute_provider       = {}
    orchestration_provider = {}
    storage                = {}
    observability          = {}
  }
}

global would contain shared defaults, while each multi_runner_config map entry would be a runner configuration. Lane-specific values could override the global defaults. The exact naming and nesting could be refined with maintainer and user feedback.

Compute, orchestration, and storage providers could then be selected independently per runner lane. Storage may also be better modeled by capability—such as bootstrap, cache, configuration, and metadata—rather than exposing only backend-specific settings. This would allow SSM, DynamoDB, or other implementations to be introduced without coupling the whole configuration model to one backend.

Questions for maintainers and users

• Should the single-runner configuration in the root module be frozen, deprecated, and eventually removed?
• Should the current flat multi_runner_config v1 also be frozen and become a compatibility layer rather than the long-term API?
• Should all new features be implemented only through the nested configuration model?
• Should shared global defaults and lane-specific overrides be part of the design?
• Should each multi_runner_config map entry represent an independent runner configuration?
• Should compute, orchestration, and storage providers be independently selectable per lane?
• What migration and state-address compatibility would be required?
• Should this become the default direction for a future major release?