Support charmcraft extensions in Scenario auto-load

[tonyandrewmeyer]

When Scenario autoloads the metadata, it does so via charmcraft.yaml (or the older metadata.yaml, actions.yaml, config.yaml). However, if an extension is being used (such as for the 12-factor apps) then there is additional metadata provided by the extension (for example, see the code in charmcraft).

For the moment, if you're writing tests for these, then you need to explicitly provide the metadata rather than use autoload (for example, you could copy it from the packed charm).

12-factor apps are probably less likely to have additional unit tests, but it would be nice if we did make this easier to do.

It seems likely that we'd need to either run charmcraft to get the generated metadata, or import charmcraft to figure it out. I don't think we would want to try to manually maintain a copy of what metadata each extension adds.

[tonyandrewmeyer]

Possible approaches, in no particular order:

1. Scenario could import the relevant code from charmcraft. This could be done once per test run and then used based on the metadata. This would make charmcraft a dependency (plus lots of indirect dependencies). We'd need to figure out versioning somehow.
2. We could store the relevant information in Scenario itself. This would be like (1) but we could have some regularly scheduled automation that extracted the relevant data and wrote it into a place that would be installed with Scenario and then used from there. Vendoring, basically, but with some automation to keep up-to-date and massage into a more useable format. This has the same versioning issues as (1).
3. Scenario could run expand-extensions. This would need to be done for each charm, but could be cached. This would make the charmcraft CLI tool an optional dependency. For versioning, I think we would rely on the installed version of charmcraft being the one that the tests want.
4. We could convince the 12-factor team that the data should be stored in the paas-charm package (potentially in a Python class, which would then also make using our loading code simpler). Scenario could learn how to load it from there. Versioning would not be an issue (or at least, not any more than it was already). This means more work for that team, maintaining the data in the charmcraft extension code, and in the charm code that uses it, and a new location of the raw data.
5. We could add support for a charmcraft-expanded.yaml (or similar) that charm developers can generate once (charmcraft expand-extensions > charmcraft-expanded.yaml) and commit to their repo (it'd be simple enough to have automation to ensure it's kept up-to-date).
6. Similar to (5), have some automation (pre-commit or whatever) that generates the full YAML and saves that into charmcraft.yaml (and then no need for the charmcraft extension system).

For 4:

extensions:
  - django-framework

requires:
  postgresql:
    interface: postgresql_client
    optional: false

config:
  options:
    log-level:
      type: string
      default: info

import paas_charm

class MyConfig(paas_charm.DjangoCharmConfig):
    my_extra_thing: str = "hello"

class MyCharm(paas_charm.DjangoCharmBase):
    def __init__(self, framework):
        config = self.load_config(MyConfig)

[benhoyt]

Without thinking too deeply, I prefer option 3 -- "vendoring, basically", with the script running once a month or so before release. Could have a team discussion next week?

[tonyandrewmeyer]

Without thinking too deeply, I prefer option 3 -- "vendoring, basically", with the script running once a month or so before release. Could have a team discussion next week?

I had a good conversation with Dima about this today also. Some steps:

• Look over these charms and see whether any of them have anything in the charm other than what's provided by the 12-factor system, and if so what they do about unit tests.
• Ask the charmers (somewhere) what they do about unit tests when using the 12-factor system - maybe none of them have tests because it's too hard to do.
• Draft a PR with a vendor solution to see how complex/large it is, maybe with just one extension to start with.

We're leaning towards something simple (like the vendoring solution, maybe without the automation, because this doesn't change that often) and extending if there's a need in the future. It's interesting that (based on the GitHub query above, which may not capture everything) it seems like Flask and Django are really the only extensions being used at the moment.

[tonyandrewmeyer]

• httprequest-lego-operator has no charm unit tests. It has a handler for config-changed and pebble-ready that copies some extra files for git. It also adds 4 extra actions (in an actions module) and a collect-status handler for checking for required config (used copying the files).
• support-portal has no charm unit tests. It configures Traefik (on every juju hook). I think the 12-factor charms support Traefik, so I'm not sure if this is old or extra or something else.
• forms.canonical.com has no charm unit tests. It configures Traefik (like support-portal) and also sets a hard-coded workload version.
• university.canonical.com has no charm unit tests. It configures Traefik (like support-portal) and also sets a hard-coded workload version on start.
• cs.canonical.com has no charm unit tests. It sets a hard-coded workload version on pebble-ready and update-status.
• documentationacademy.org has no charm unit tests. It's pretty old - using xiilib for example - and I think probably if it was brought up-to-date the framework would actually do everything required and there wouldn't be any custom charm code.
• github-runner-webhook-router has no charm unit tests. There's an extra action, and some ingress setup.
• dashboard has no charm unit tests. There's one extra action.

[tonyandrewmeyer]

Also these chamrs (base, not bases).

[tonyandrewmeyer]

Also these chamrs (base, not bases).

Of those, the following charms meaningfully extend paas_charm base classes:

Custom actions

Repo	Base	What it adds
canonical/commitment-tracker	paas_charm.go	4 actions (create-token, delete-token, list-tokens, bootstrap-maintainer-token) for API token management via Pebble exec; auto-bootstraps maintainer token on start; requires postgresql
canonical/github-runner-operators (planner-operator)	paas_charm.go	2 actions (enable-flavor, disable-flavor); provides planner relation with full credential lifecycle (create/delete auth tokens, sync flavors); overrides restart() to reconcile relation data; overrides _create_app() for OpenTelemetry env vars; custom COS dir; requires rabbitmq, tracing, postgresql
canonical/hockeypuck-k8s-operator	paas_charm.go	3 actions (block-keys, rebuild-prefix-tree, lookup-key) via separate actions.Observer; overrides restart() to open reconciliation port and push admin GPG key; overrides is_ready() to block multi-unit; overrides _create_charm_state() to inject admin key fingerprint; installs gnupg via apt; traefik-route observer; requires postgresql, traefik-route
canonical/job-manager	paas_charm.fastapi	2 actions (generate-access-token, revoke-access-token) via HTTP calls to workload; overrides _on_config_changed and _on_pebble_ready; custom _setup_environment() to create dirs and push clouds.yaml into container; requires postgresql, s3, rabbitmq
canonical/kernelfactory.canonical.com (engine)	paas_charm.flask	2 actions (generateexchange, generatetoken) that exec flask CLI commands in the container; requires postgresql, rabbitmq, redis
canonical/telemetry-management-service	paas_charm.go	1 action (get-latest-authorization-model-id) that queries OpenFGA HTTP API; adds charm-tracing; requires openfga, postgresql, tracing

Overridden methods / custom app logic (no custom actions)

Repo	Base	What it adds
canonical/gatus-k8s-operator	paas_charm.go	Overrides restart() with config validation guard (blocks restart if config invalid); resolves Juju secrets for Mattermost webhook URL; injects custom env vars via extra Pebble layer; requires postgresql
canonical/github-runner-operators (webhook-gateway)	paas_charm.go	Overrides _create_app() to inject OpenTelemetry environment variables (Prometheus metrics exporter, OTLP traces); requires rabbitmq, tracing
canonical/open-graph-images-generator	paas_charm.go	Overrides _create_app() with a CustomApp subclass that strips the APP_ prefix from environment variable keys
canonical/directory-api	paas_charm.go	Overrides _create_app() with CustomApp that strips APP_ prefix from env vars; uses nginx-route instead of built-in ingress (disables _on_ingress_ready/_on_ingress_address_changed); requires nginx-route, redis
canonical/message-queue-service	paas_charm.go	Adds ops.tracing.Tracing for charm-level tracing; requires postgresql, charm-tracing
canonical/telemetry-connector	paas_charm.go	Adds ops.tracing.Tracing for charm-level tracing; requires postgresql, charm-tracing

nginx-route ingress override pattern

These charms replace the built-in PaaS ingress with nginx-route and disable the default ingress handlers.

Repo	Base	What it adds
canonical/careers.canonical.com	paas_charm.fastapi	Replaces built-in ingress with nginx-route; requires nginx-route, postgresql
canonical/directory.canonical.com	paas_charm.flask	Replaces built-in ingress with nginx-route
canonical/specs.canonical.com	paas_charm.go	Replaces built-in ingress with nginx-route; requires nginx-route, postgresql
canonical/wikijs-charm	paas_charm.go	Disables ingress handlers (nginx-route setup commented out); requires postgresql

The 8 canonical/portal sub-applications (academy, cerberus, console, distributor, hermes, janus, ktesios, zeus) also only add _on_start to set workload version — no real functional extension.