refactor(platform)!: make sandbox environments explicitly composed

[johntmyers]

User Story

As an OpenShell platform operator, I want sandbox environments to be assembled explicitly from an image and the capabilities my workload needs, so that OpenShell core can run arbitrary compatible images without silently imposing a stale toolchain, network policy, provider catalog, or product workflow.

As a product builder, I want versioned reference compositions that I can copy and adapt, so that I can start from a working example without turning that example into an implicit platform default or compatibility promise for every image.

Problem Statement

OpenShell's current first-run path carries development-oriented defaults from an earlier stage of the project. Those defaults intentionally supported a narrow set of use cases so the project could iterate quickly:

• the default image and bare image shorthand resolve to the separately maintained OpenShell Community catalog;
• the community base image contributes an image-baked sandbox policy and assumes a particular shell, user, filesystem layout, and installed toolchain;
• when no policy is supplied, the supervisor discovers one from the image or creates and persists a core-authored restrictive fallback;
• fifteen provider profile YAML files are compiled into the release binary and selected by default alongside user-managed profiles; and
• several of those profiles bind network access to executable paths that happen to exist in the community image but are absent or different in normal third-party images.

Those choices served development velocity, but they now place assumptions for a narrow set of workflows in the platform's critical path. They also allow separately valid components to become invalid when composed. A policy authored before credential provenance enforcement can contain an L4 endpoint that is valid on its own, while an attached credentialed provider causes the same host and port to require inspected traffic.

Issue #2998 exposed this through the old image policy. Its pypi and copilot rules contain L4-only GitHub endpoints. When a GitHub provider is attached, gateway-derived credential provenance marks matching github.com:443 and api.github.com:443 endpoints as credentialed. The resulting effective policy is unsafe even though the image policy and provider profile were each accepted through different lifecycle paths.

The disk YAML is not wholly unvalidated: the supervisor parses it and performs intrinsic policy validation before loading it. The missing invariant is validation of the exact image, user, provider, credential, and middleware composition before workload activation. The gateway cannot see an image policy during the initial create request; the supervisor discovers and syncs it later through a path that deliberately skips the credential-aware admission gate. The workload can therefore start while the network proxy correctly denies affected connections at runtime.

PR #3129 addresses the resulting edit deadlock by allowing existing-sandbox edits to inherit previously admitted violations while rejecting newly introduced ones. That makes an invalid sandbox repairable, but it does not make the first activated configuration valid and does not make the affected traffic work. Python or uv access to the PyPI profile's overlapping GitHub endpoints remains denied; ordinary pypi.org and files.pythonhosted.org traffic is not part of that collision.

Patching the community image policy or permanently grandfathering invalid effective policies treats the symptoms. OpenShell core is becoming a platform on which products and organization-specific environments are built. Its stable contract should preserve the ability to iterate quickly through ready-to-run examples while moving narrow use-case assumptions out of the platform's implicit defaults.

Impact / Why This Matters

Today, operators can receive a sandbox that appears ready but cannot perform expected work because its effective configuration was never admitted as a complete unit. Unrelated policy edits can then fail on inherited content, and connection-time denial surfaces an architectural configuration error as an application failure.

Provider profiles also appear portable because they are built in, even when their binary selectors assume community-image paths such as /sandbox/.venv, /app/.venv, /usr/lib/node_modules, or /bin/bash. Changing the sandbox image can silently make a profile inert or unusable. Broadening those profiles to match every possible image would weaken binary-scoped least privilege.

The current workaround is to patch shared defaults, add exceptions, or reverse-engineer the implicit composition. That is insufficient because the defaults will drift again, third-party images remain unconstrained, and a general-purpose platform cannot know the correct tools, permissions, credentials, or middleware for every product.

Making composition explicit removes those assumptions from the critical path while preserving approachable, ready-to-run examples outside the platform defaults.

Proposed Design

1. Keep the core runtime neutral

Issue #3116 replaces the community image default and shorthand with a version-qualified official Alpine image and preserves explicit OCI image selection. Complete that work as a dependency of this direction, including the shell, identity, filesystem, network-namespace, and supervisor prerequisites required to run an unmodified Alpine image.

The default image supplies an operating-system userspace only. It does not promise Git, Python, curl, an agent runtime, skills, provider-specific binaries, or a workload access policy.

2. Remove implicit workload policy

OpenShell does not ship or select a project-authored workload access policy for a sandbox. Selecting the default image must not discover network endpoints, provider behavior, middleware, or product-specific filesystem permissions from that image.

When the operator supplies no workload policy or provider profile, the externally observable result is zero workload network grants and no activated middleware. OpenShell may retain the minimum internal, fail-closed runtime substrate necessary to launch and supervise an unprivileged process, but that substrate is a platform safety mechanism rather than a reusable or featureful "default policy." It must not contain provider endpoints or tool-specific permissions.

Explicit third-party images may contain policy material if that remains a supported composition source, but choosing such an image does not bypass validation. Policy-source precedence and provenance must be deterministic and visible, and the exact composed generation must pass the activation gate in #3145.

3. Make provider profiles import-only

Remove the compiled built-in provider profile catalog and stop selecting a builtin profile source by default. A new gateway starts with its user-managed profile source available but may have an empty profile catalog.

Before creating a profile-backed provider, an operator imports the desired profile at platform or workspace scope, or configures an explicit external/interceptor profile source. Existing profile lint, import, export, update, revision, and source-precedence behavior should form the basis of this workflow.

The YAML files currently under providers/ remain versioned, linted examples. Documentation shows how to obtain, review, copy, modify, and import them. They are not embedded into release binaries and are not represented as universally compatible defaults. Each example states its expected client binary identities, compatible reference image or installation layout, credential behavior, endpoint access, and smoke test.

Removing built-ins also requires removing built-in ID reservation so operators can import familiar IDs such as github and pypi. CLI credential suggestions and command-to-provider inference must use the gateway's active imported catalog or become explicit; they must not retain a hidden local copy of the removed catalog.

This change removes embedded profile definitions, not necessarily every protocol implementation. Refresh strategies, credential brokers, inference backends, or metadata emulators that are genuine platform capabilities may remain implemented in core, but their activation must require an explicitly imported profile/configuration. Type-specific behavior that YAML cannot currently describe—such as Google Cloud metadata and project/region environment projection—must either become expressible in the profile contract or remain a deliberately documented adapter. It must not masquerade as generic profile data.

Apply the same classification test to middleware. Gateway interceptors are already opt-in. The compiled openshell/regex supervisor middleware is currently installed automatically but remains inert until policy selects it; maintainers must explicitly classify it as a stable platform primitive or move it to the examples/extension layer. In either case, no middleware configuration activates by default.

4. Provide explicit reference compositions

Keep "bundle" as a working term until the artifact model is designed. Initially, a reference composition may be a versioned example directory and orchestration script rather than a new API resource.

A reference composition declares or links all inputs needed for one use case:

• an explicit image reference or Dockerfile;
• the sandbox policy and its intended scope;
• provider profile files plus their import scope;
• provider and credential setup instructions without embedded secrets;
• optional supervisor middleware configuration and implementation;
• optional gateway interceptor configuration and implementation;
• setup, smoke-test, expected-denial, and cleanup commands; and
• compatibility metadata, including tested OpenShell version, image digest, architectures, and resolved executable paths.

Provide at least these ready-to-run examples in the core repository:

1. a minimal Alpine sandbox with no network capability, demonstrating the neutral platform baseline;
2. a small Git image plus an imported GitHub profile, demonstrating read-only git ls-remote and a denied write; and
3. a small Python image plus an imported, image-adjusted PyPI profile, demonstrating a public metadata read and installation/download boundaries.

At least one additional example should demonstrate explicit middleware or interceptor composition. No example is loaded, imported, or activated merely by installing OpenShell.

Do not overload the existing workload-template resource without a separate design. Current templates describe image, environment, resources, and driver configuration, while policies, providers, middleware, and gateway interceptors have different scopes and lifecycles. A first-class bundle resource would need explicit ownership, ordering, atomicity, trust, secret, upgrade, and rollback semantics.

5. Validate the complete composition before activation

Use issue #3145 to establish the lifecycle invariant: OpenShell composes all selected policy sources, imported provider layers, credential bindings and derived provenance, middleware, and relevant interceptor contracts; validates the exact candidate; activates that generation; and only then exposes or starts the workload.

Invalid compositions enter a stable, repairable configuration-error state with no workload egress. Runtime connection-time enforcement remains fail closed as defense in depth. Creation, restart, provider attach/detach, policy replacement/update, profile revision changes, middleware changes, and applicable interceptor changes must converge on the same invariant.

The explicit local Rego/data override path also needs a defined outcome. It cannot currently receive gateway-derived credential provenance. It must either participate in the same composition/admission exchange or reject combination with provider credentials; a warning followed by partially inactive credential controls is not a valid composition mode.

Issue #3092 remains complementary: it covers validation and normalization at policy transport and runtime-loading boundaries, while #3145 covers composition and workload activation ordering.

PR #3129 should be evaluated as a migration and repair mechanism for already-admitted legacy configurations. Once #3145 defines an atomic repair workflow with the workload stopped, decide whether differential grandfathering is still needed for upgrade compatibility or should not become permanent policy semantics.

6. Publish a coordinated migration

Treat this as a coordinated pre-0.1.0 breaking change under issue #2565.

The migration guide must cover:

• the Alpine default and removal of community shorthand;
• removal of the community image's policy and tools from the implicit path;
• the zero-grant behavior when no workload policy/profile is selected;
• exporting or obtaining example profiles and importing them before provider creation or sandbox use;
• adapting binary selectors to the selected image without broad wildcarding;
• behavior of existing provider instances whose profile is absent after upgrade;
• replacement of CLI auto-inference or credential suggestions that formerly depended on compiled profiles;
• converting current tutorials and examples into explicit reference compositions; and
• diagnosing and repairing a composition rejected by the pre-activation validation gate.

Missing profiles, incompatible binary identities, and invalid compositions fail with actionable bounded diagnostics rather than silently falling back to a built-in definition.

Acceptance Criteria

• refactor(sandbox)!: retire community images and default to Alpine #3116 delivers a neutral, version-qualified Alpine default and removes OpenShell Community image resolution from supported runtime, deployment, test, documentation, and skill paths.
• The default image contains no OpenShell-authored sandbox policy and supplies no implicit workload network endpoints, provider behavior, middleware activation, or agent/tool assumptions.
• Creating a sandbox without workload capabilities results in a documented zero-network-grant, no-middleware posture while retaining only the minimum fail-closed platform substrate required to supervise an unprivileged process.
• Policy origin and precedence are deterministic for explicit sandbox policy, supported image policy, gateway-global policy, provider-derived layers, and runtime substrate; no image policy bypasses the complete activation gate.
• No provider profile YAML is compiled into OpenShell release binaries, and the default gateway profile catalog consists only of explicitly configured/imported sources.
• A gateway with no imported profiles starts successfully with an empty user-managed catalog.
• Operators can import the example profiles using their canonical IDs at platform or workspace scope; former built-in ID reservation does not block them.
• Provider creation, credential discovery suggestions, command auto-inference, and sandbox attachment resolve only against the active gateway catalog and fail actionably when a required profile is absent.
• Core-specific provider adapters that remain are inventoried and clearly separated from imported profile data; profile schema gaps are resolved or documented as explicit adapters.
• Built-in middleware implementations are explicitly classified as platform primitives or moved to examples/extensions, and no middleware or interceptor configuration is active by default.
• Every YAML file under providers/ is treated as a linted example and documents compatible image/tool assumptions, credential scope, least-privilege behavior, and a smoke test.
• The repository includes explicit, versioned reference compositions for neutral Alpine, GitHub read-only, and PyPI read workflows, plus at least one middleware or interceptor example; none activate automatically.
• feat(sandbox): validate effective policy before workload activation #3145 prevents workload activation and egress until the exact composed policy/provider/credential/middleware generation validates, and provides a stable repair workflow for rejected configurations.
• Local Rego/data override mode either participates in the same credential-aware admission pipeline or fails closed when combined with provider credentials.
• Validate and normalize SandboxPolicy before supervisor activation #3092 ensures equivalent validation and normalization at supported policy-loading and runtime-activation boundaries.
• Runtime enforcement continues to deny unapproved connections even after admission validation succeeds or fails.
• Upgrade documentation covers all removed defaults and gives a complete export/copy/import/customize/run path for existing users.
• Tests prove an empty installation has no implicit provider catalog or workload access, imported examples work only with their declared images, incompatible paths fail closed, and no component is pulled from OpenShell Community.
• Maintainers explicitly decide whether PR fix(server): gate only newly authored uninspected credentialed endpoints #3129 remains necessary as a bounded migration/repair behavior after feat(sandbox): validate effective policy before workload activation #3145, rather than treating it as the root fix.

Suggested Work Breakdown

1. Complete refactor(sandbox)!: retire community images and default to Alpine #3116 and remove community-image consumers without adding new tool/profile assumptions to Alpine.
2. Specify the zero-grant runtime substrate and remove default image-policy discovery from the default path while preserving any deliberately supported explicit image-policy contract.
3. Remove the compiled provider catalog, switch defaults to the user-managed source, and make the CLI resolve suggestions against the gateway catalog.
4. Make canonical profile IDs importable and define upgrade behavior for existing provider instances with missing profiles.
5. Inventory type-specific provider adapters and either express their behavior in the imported profile contract or document their explicit platform boundary.
6. Land Validate and normalize SandboxPolicy before supervisor activation #3092 and feat(sandbox): validate effective policy before workload activation #3145 so the full composition is validated and repairable before workload activation.
7. Convert providers/, tutorials, tests, and selected examples into versioned reference compositions with positive and negative smoke tests.
8. Decide whether a first-class bundle resource is warranted after the example-based workflow establishes concrete scope and lifecycle requirements.
9. Reassess PR fix(server): gate only newly authored uninspected credentialed endpoints #3129 against the completed activation and repair model.
10. Publish and verify the coordinated 0.1.0 migration, then remove remaining community/default-profile compatibility paths.

Alternatives Considered

Patch the community image policy

Updating the old pypi and copilot rules would remove the immediate collision for that image, but it preserves the coupling and cannot prevent the next mismatch between an image, profile, and platform safety rule.

Merge PR #3129 as the complete fix

Differential validation makes inherited invalid policy editable and preserves strict checks for newly introduced violations. It is useful as a compatibility or repair mechanism, but the workload still starts with an invalid effective configuration and the proxy still denies affected connections. It does not replace validation before activation.

Keep built-in profiles and continuously broaden their binary paths

This creates an unbounded compatibility promise across arbitrary images and encourages broad binary patterns. Importable examples make assumptions reviewable and editable without weakening core policy.

Keep built-ins but disable them by default

This removes automatic selection but still embeds a product catalog, reserves its identifiers, drives CLI inference, and implies a release compatibility surface. Import-only profiles create a cleaner platform boundary.

Require a first-class bundle API immediately

Images, policies, providers, middleware, and interceptors have different ownership and lifecycle scopes. A premature aggregate resource could hide important ordering and trust decisions. Start with explicit, versioned reference compositions and design an API from demonstrated workflows.

Remove all built-in provider adapters and middleware implementations

Removing profile definitions does not automatically require removing generic refresh, inference, metadata, or middleware mechanisms. Keep only capabilities deliberately classified as platform primitives, require explicit activation, and track any remaining product-specific adapter separately.

Agent Investigation

The investigation confirmed the architectural diagnosis with a narrower statement about validation:

• image policy YAML is parsed and intrinsically validated on disk today;
• sandbox creation cannot include that unseen image policy in its candidate;
• supervisor discovery sync intentionally bypasses the credential-aware admission gate to avoid a crash loop;
• gateway delivery composes and credential-stamps the policy but reports rather than rejects the unsafe combination;
• readiness and workload launch are not gated on admission of that exact effective generation; and
• the proxy correctly denies the resulting uninspected credentialed connection at runtime.

For #2998, the GitHub provider's credential scope overlaps L4 github.com and api.github.com endpoints in the old image's pypi/copilot rules. PR #3129 admits edits that inherit those violations but leaves runtime enforcement unchanged. The collision affects the overlapping GitHub endpoints; it does not make all PyPI endpoints unusable.

The existing code already has much of the import-first foundation: user-managed profile storage and scope, lint/import/export/update commands, configurable authoritative source sets, and opt-in gateway interceptors. The principal gaps are compiled/default profile selection, CLI dependence on the local built-in catalog, type-specific behavior not represented by profile YAML, image-policy lifecycle ordering, and the absence of a cross-component composition artifact.