feat: COS-Dev mesh pt.2

[MichaelThamm]

Tip

We should remove Traefik if ingress is all disabled. We should remove ssc if internal_tls = false. We should remove istio charms if mesh_enabled = false.

Issue

The existing COS-Dev Terraform API for configuring TLS and ingress (via Traefik) used a set of flat, disconnected variables (internal_tls, external_ca_cert_offer_url, external_certificates_offer_url, mesh_enabled). As COS-Dev gains support for a service mesh as an alternative to Traefik-based ingress/TLS, this flat API made it increasingly awkward to express that mesh and reverse_proxy are two mutually exclusive but parallel mechanisms for solving the same problem — securing and exposing COS services.

Solution

Builds on:

• feat: COS-Dev mesh pt.1 #343

This PR refactors the COS-Dev Terraform API to introduce two structured, parallel object variables — reverse_proxy and mesh — replacing a collection of flat primitives. The new API makes the relationship between these two features explicit and consistent.

API Changes

Before:

# TLS
internal_tls                     = true  # or false
external_ca_cert_offer_url       = "admin/my-ca.send-ca-cert"
external_certificates_offer_url  = "admin/my-ca.tls-certificates"

# Mesh (mutually exclusive with TLS)
mesh_enabled = false

After:

internal_tls  = true  # or false
# Reverse proxy (Traefik-based ingress + TLS) — enabled by default
reverse_proxy = {
  enabled = true
  cmr_urls = {
    certificates    = "admin/my-ca.tls-certificates"
    receive_ca_cert = "admin/my-ca.send-ca-cert"
  }
}

# Service mesh (Istio-based) — disabled by default, mutually exclusive with reverse_proxy
mesh = {
  enabled = false
  cmr_urls = { ... }
}

Key motivations for the redesign:

1. Symmetry between reverse_proxy and mesh: Both variables solve the same underlying problem (TLS termination and ingress for COS), but via different mechanisms. By giving them the same shape — an enabled flag and a nested cmr_urls object — the API makes this relationship self-documenting.

2. reverse_proxy is now explicitly opt-out: Previously, Traefik was always deployed and only conditionally integrated. Now it is gated behind reverse_proxy.enabled (default true), allowing users to fully disable it with enabled = false. This is the prerequisite for deploying COS with mesh only, or with neither.

3. CMR URLs moved into the structured map: external_ca_cert_offer_url and external_certificates_offer_url were standalone top-level variables. They are now nested under reverse_proxy.cmr_urls.receive_ca_cert and reverse_proxy.cmr_urls.certificates, co-locating all reverse proxy configuration in one place. The existing co-validation (both must be set together, or neither) is preserved.

4. Mutual exclusivity is enforced: A Terraform validation rule prevents mesh.enabled and reverse_proxy.enabled from both being true simultaneously. Both can be disabled.

Additional cleanups bundled in this PR

• loki_bucket, mimir_bucket, and tempo_bucket are consolidated into a single s3_buckets object map.
• istio_beacon and istio_ingress module references are fixed (dashes → underscores) and their Juju charm data sources, channels, and revisions are properly wired up.
• All CMR mesh offers are now gated behind var.mesh.enabled (they were unconditionally created before).
• Test files are reorganised: mesh_enabled.tftest.hcl is replaced by mesh.tftest.hcl and reverse_proxy.tftest.hcl to cleanly separate the two features.

Testing Instructions

Run the Terraform tests:

cd terraform/cos-dev
terraform test

The new test files cover:

• reverse_proxy.tftest.hcl — Traefik deployed when enabled (default), not deployed when disabled.
• mesh.tftest.hcl — Istio deployed when mesh enabled, not deployed by default; validation error when both mesh and reverse_proxy are enabled simultaneously.

[adhityaravi]

One thing we might need to look into is, the beacon and ingress charms will error out without the istio-k8s charm. You mentioned in a different PR that the istio-k8s doesnt belong here.

If thats the case, shouldnt we atleast validate it? or is the idea that we let the charms error out and let the admin figure it out?

[adhityaravi]

I dont think we need to offer the require_cmr_mesh side of the CMR relation. We just need to offer the provide_cmr_mesh and the requirer will consume it with its own requirer side of the relation.

[MichaelThamm]

If thats the case, shouldnt we atleast validate it? or is the idea that we let the charms error out and let the admin figure it out?

Good point, since there is no integration that istio-k8s offers, we have no way of validating this in COS' Terraform module, so we would need to resort to local-exec which we want to avoid if possible.

[adhityaravi]

A general API concern:

mesh and reverse_proxy are not mutually exclusive concepts. Istio still uses an edge proxy istio-ingress to route the traffic. I feel like we are creating an artificial boundary between mesh and reverse_proxy with this design which might be confusing networking wise.

You can theoretically still deploy just istio-ingress-k8s and use it purely for ingressing without have the service-mesh relations. which would be the same as reverse_proxy enabled.

[adhityaravi]

A rough idea to consider - reverse_proxy and mesh arent mutually exclusive but rather different depths at which COS as a product offers to model the network for the users

level 0 - bring your own networking. We model 0 networking items
level 1 - edge or ingress. we do everything in level 0 + we model the networking at the app edges ie we ingress the traffic, and additionally terminate TLS at the COS boundary (external) and optionally at the app boundaries (internal) (the cert provider is also needed for istio-ingress)
level 2 - mesh. we do everything in level 1 + we model the cross app traffic with authorization policies ie we also control which apps can talk to which other apps etc

This can be a single input var object called a networking_model or something?

This would also help us phase traefik completely out of COS without touching the API (however far in the future that might be or never but having an option is always nice) because we can still provide pure edge routing with istio-ingress without enabling the service mesh features of beacon.