Archive: RFC: External Backend Manifests (Backend Plugin Specification)

[abn]

Proposal size: Major feature (spans multiple well-scoped PRs).
Prototype: https://github.com/abn/lemonade/tree/feat/external-backend-system

This RFC covers how an out-of-tree inference engine is described, discovered, launched, and managed by lemond. The companion Sandbox RFC owns process confinement, and the boundary between the two is stated in section 12. Section 21 states the scope of the first implementation and what it leaves out.

1. Why this exists

Lemonade ships a fixed set of backends. If you want to run an engine Lemonade does not build, your options today are to rebuild the C++ server or to wait for a release. Both are wrong for the cases people actually have: a community llama.cpp build with extra kernels, a vendor nightly, a container image pinned to a digest, an internal engine that only exists in your network.

This RFC makes the engine describable. A small JSON manifest says which executable to run and what it can do. lemond reads the manifest at runtime, starts the engine as a subprocess, waits for it to answer a health probe, and forwards the fixed Lemonade routes to it. Nothing in the C++ tree changes, so a new engine arrives as a file, not a rebuild.

The design keeps one hard line: a manifest describes an engine, it does not program the server. There is no scripting and no request rewriting. If an engine needs more than a manifest can say, it graduates to a built-in backend, and that path stays open.

2. User stories

As an operator, I want to register a third-party or containerized inference engine (custom llama.cpp build, vLLM, an NPU runner) with lemond as a declarative manifest, so I can run the engine that fits my hardware without rebuilding the C++ server.

As an engine or fork developer, I want to publish my engine, or a fork of an existing one, as a self-describing manifest, so it is discoverable, installable, and reviewable in Lemonade.

As a user with a fork or repackaged build of an already-built-in backend, I want to register it as a variant_of that reuses the built-in's handling, so it runs without re-declaring argv or touching C++.

3. High-level design

An out-of-tree engine has one of two shapes:

• Manifest-only passthrough. lemond orchestrates the engine directly from the manifest's command and args. This covers most engines, including containerized podman and docker recipes.
• variant_of. The recipe is a binary drop-in for a named built-in. lemond asks the built-in for the argv it would have used and runs that against the external binary, so the manifest declares binary provenance and deltas only.

Both run the engine as an out-of-process subprocess. In-process dlopen plugins are permanently out of scope.

A backend never reads /sys or /proc to learn about its host. Centralized hardware and model facts are resolved by lemond and handed to the engine as launch-time tokens. A queryable fact channel (an engine asking for facts at runtime) belongs to a future adapter RFC.

An engine that outgrows both shapes graduates to a built-in WrappedServer. An out-of-process adapter tier is a possible expansion and would be its own RFC.

4. The manifest contract

The manifest is validated against the published JSON Schema, pinned by a lock file. Its JSON Schema major is api_contract_version, so there is no second version number to keep in sync.

Validation is strict, and the parser rejects rather than ignores. A manifest is rejected, with a message naming the offending field, when it:

• omits a required field (recipe, display_name, api_contract_version, capabilities, platforms);
• carries an unknown field at any object level (additionalProperties: false);
• names an unknown capability in capabilities, capability_enable_args, or endpoints;
• declares a newer api_contract_version than the running server supports;
• uses an unknown token in a command, arg, env value, or stop command;
• declares both extends and variant_of;
• is a variant_of binary pinned by hash without a sha256.

Silent acceptance of an unknown field or capability is a contract violation, not a tolerance.

4.1 Identity

• recipe: stable id, ^[a-z0-9][a-z0-9_-]{1,63}$. It must not collide with a built-in recipe or with another external recipe found at a higher-priority path.
• display_name: human-readable name.
• api_contract_version: "1" for this major.
• extends (optional): declarative inheritance, see section 6.
• variant_of (optional): procedural inheritance, see section 7.

4.2 Execution block

Each platform block carries command, args, an optional stop_command and stop_command_args, an optional working_dir, and an env object. A variant_of block also carries binary, argv_extra, and reserved_args, and may override the top-level source, sha256, and version_policy. Tokens are resolved when the process is spawned, not when the manifest is read.

Normative token vocabulary (v1). The vocabulary is fixed for the contract major and additive across majors. Unknown tokens fail at discovery.

Group	Tokens
Runtime	{port}, {host}, {pid}, {log_level}, {recipe}, {model_name}
Checkpoints	{checkpoint:NAME}, {checkpoint_relative:NAME}, {resolved_path}, {model_relative_path}, {model_dir}, {exe_dir}, {hf_cache}, {cache_dir}
Hardware	{rocm_arch}, {cuda_arch}, {arch_alias}, {target_device}, {hip_visible_devices}, {cuda_visible_devices}, {rocr_visible_devices}, {ggml_vk_visible_devices}, {ze_affinity_mask}
Options	{ctx_size}, {batch_size}, {ubatch_size}, {threads}, {cache_type_k}, {cache_type_v}
Custom	{custom:NAME}, {custom:NAME:-DEFAULT}, {custom_args}
Env	{env:NAME}, {env:NAME:-DEFAULT}

Resolution and injection rules.

• A token resolves into a single argv element or env value. The text is never passed through a shell. The primary process is spawned with execvp; there is no shell string for command or args.
• A resolved value that begins with - is rejected regardless of argv position, unless it is a validated negative number (-0, -12, -1.5), which the existing option parser already accepts. This covers positional tokens such as {model_name} as well as option values.
• {custom:NAME} resolves any key present in recipe_options, including one inherited through extends. custom_options only controls CLI and config exposure; it is not a whitelist for {custom:} resolution.
• {custom_args} expands a user string or JSON array into separate argv elements using the existing shell-quote-aware parser, then validates each against reserved_args. An exact match or a leading <flag>= is rejected.
• {env:NAME} reads the parent environment through a default-deny allowlist. LEMONADE_* names and secret-shaped names (*API_KEY*, *TOKEN*, *SECRET*, *PASS*, *AUTH*) are never resolvable, so a secret cannot be placed on a command line. The allowlist is implementation-defined and documented. The v1 default is empty plus credential-free cache and CA variables; proxy URL variables are excluded because their values may embed credentials.
• The token-resolution allowlist is distinct from the Sandbox RFC's child-process env_allowlist. The former chooses which host values a manifest may interpolate; the latter chooses which parent variables a child inherits and is enforced by the Sandbox RFC.
• stop_command and stop_command_args are executed as an argv vector. A manifest that needs a container runtime supplies podman or docker as the command, not a shell pipeline.

4.3 Lifecycle fields

• health_probe: type is http, tcp, or process. http probes endpoint for expected_status; tcp checks that {port} is connectable; process treats the child surviving past spawn as ready. timeout_seconds and poll_interval_ms bound the probe. A child that exits during the probe fails fast instead of waiting out the timeout.
• requested_ports: how many ports the engine needs, default 1. v1 defines the single-port contract ({port}); multi-port is a reserved extension and any value other than 1 is rejected.
• reserved_args: flags a user's {custom_args} must not override. Per-block reserved_args are unioned with the top-level list, with the block taking precedence.
• model_management: lemond_managed (default) or self_managed. In v1, self_managed means lemond does not pre-download model weights for the recipe. The model-inventory and readiness RPC is deferred (section 19), so self_managed implies nothing else.

4.4 Governance

slot_policy is standard, exclusive_npu, coexist_by_type, or unmetered, and maps directly onto the existing router slot policies. default_accelerator selects the platform block when the request names no device.

4.5 Extension fields

These were kept from the prototype as additive surface. They do not open routes.

• endpoints: remaps a declared capability's nonstandard engine path onto Lemonade's fixed route, for engines whose internal path differs from the OpenAI shape. The value is an engine-side path.
• custom_options: declarative CLI and config knobs the recipe exposes, surfaced in the CLI and config file.
• downsize_endpoint: engine-side path invoked on soft-idle downsize.

4.6 Arch overrides and aliases

The platform matrix is host OS x accelerator. Hardware arch is a further axis, handled two ways so nothing is duplicated.

• arch inside a platform block is a map of arch pattern to a partial block. The block stays the common config and each entry names only its deltas. Keys are exact arch strings (gfx1151, sm_90) or globs (gfx115*); exact wins, otherwise the first matching glob. Merging is field-wise: a present field replaces the base field and env merges key-wise. The only pattern syntax is a glob, so this stays declarative.
• arch_aliases at the top maps an arch (or glob) to a short alias. The {arch_alias} token resolves through it, so a name, tag, or path that varies by arch is written once.

Selection order for a load: OS, then accelerator (model device, else default_accelerator, else detection), then the matching arch override. The same merge runs at install time, so the artifact the CLI fetches and the argv lemond launches cannot disagree. When an artifact exists only inside an arch override, install needs the arch: lemonade backends install-external --arch gfx1151.

5. Capabilities

Membership in the capability set is a canonical registry. A capability maps to exactly one deployment mode and one or more fixed Lemonade routes. A manifest never opens a route: an operation outside the declared set is answered with the existing unsupported_operation envelope.

Manifest capability names are not the same as the server's deployment-mode labels. The mapping is explicit and normative:

Manifest capability	Kind	Deployment mode label	Lemonade routes
chat_completion	core	chat	POST /v1/chat/completions
completion	core	chat	POST /v1/completions
embeddings	core	embeddings	POST /v1/embeddings
reranking	core	reranking	POST /v1/rerank (plus /reranking, /reranker aliases)
transcription	core	transcription	POST /v1/audio/transcriptions
image	core	image	POST /v1/images/generations, /edits, /variations
tts	core	tts	POST /v1/audio/speech
responses	extension	chat	POST /v1/responses
streaming_transcription	extension	transcription (variant_of only)	Realtime WS /realtime
classification	extension	classification	POST /v1/classify
audio_generation	extension	audio-generation	POST /v1/audio/generations
model_3d	extension	3d	POST /v1/3d/generations
slots	plumbing	none	GET /v1/slots, POST /v1/slots/{id}
tokenize	plumbing	none	POST /v1/tokenize

The seven core capabilities are the closed set defined by this specification. The five extensions are additive and kept from the prototype. New capability kinds are reviewed core changes; a manifest cannot introduce one.

streaming_transcription is variant_of-only in v1. The Realtime WS route needs protocol framing that manifest passthrough cannot express, and protocol translation is an explicit non-goal.

capability_enable_args attaches argv when the capability's deployment mode is the one the model loads in. A model deploys in exactly one mode, so every declared capability that shares that mode is enabled together. A manifest that needs finer activation than that does not fit this surface.

Gating is per route, not per interface. chat_completion and completion share the chat mode but are distinct routes, and WrappedServer bundles both in ICompletionServer, so the external path consults the declared capability set at the route through has_capability, not through dynamic_cast.

6. extends

extends inherits declarative data only. It merges the base manifest's recipe_options and custom_options into the child, with the child's keys taking precedence where they overlap. It never supplies execution: the child still provides its own platform blocks. Capabilities and platform blocks are not inherited. A cycle is rejected. If the base cannot be resolved, the child is dropped with a recorded reason. extends and variant_of are mutually exclusive.

This is narrower than inheritance across the whole manifest. If you want a fork that reuses a built-in's argv construction, that is variant_of; if you want shared default option values, that is extends.

7. variant_of

A variant_of recipe's binary is wire-compatible with a named built-in. lemond builds the argv it would have handed the built-in and runs that against the external binary. The manifest reduces to:

• binary provenance: source (optional), sha256, version_policy;
• per-platform binary, argv_extra, and reserved_args deltas.

The value must name a built-in backend, and specifically one that exposes a launch plan. The supported values today are llamacpp, whispercpp, and sd-cpp. Any other value is rejected at load with a message naming them: an unknown recipe, an external recipe, or a built-in that has not yet grown a launch plan.

version_policy defaults to pinned, which requires sha256. roll_forward is an explicit opt-out for authors tracking a moving artifact. source is fetched and hash-verified by the CLI; lemond never downloads executables. A variant_of without source uses a user-provided binary at the declared binary path.

source, sha256, and version_policy may also be set inside a platform block, overriding the top-level values for that block. That is what lets a project that publishes one artifact per platform or accelerator be described in a single manifest. At install time the CLI selects the host OS block and fetches its effective source, verifying its effective hash unless its effective policy is roll_forward. If the manifest offers more than one artifact for the host OS, the CLI requires --accelerator and names the choices rather than guessing.

This is distinct from the *_bin config override. _bin is a global, mutable, single-slot change with no versioning and no separate lifecycle. variant_of is per-recipe, pin-verified, and separately supervised.

Reuse needs a seam. Built-in argv construction is inline in each WrappedServer subclass's load(). A launch-plan step lets a backend produce (executable, argv, environment, working_dir) without spawning; variant_of then re-targets the executable and appends argv_extra. llamacpp, whispercpp, and sd-cpp expose that step; the other built-ins return a clear error until they do.

variant_of is about backend type, not runtime. It answers one question: which built-in's argument handling does this binary follow? It says nothing about how the binary is obtained or what launches it. A container image is a runtime choice that can hold any backend, so a containerized fork of llamacpp is described as a passthrough manifest (the runtime owns the image and the argv), not as variant_of. Combining the two, reusing a built-in's argv while wrapping it in a container runtime, is a provisioning concern and is not part of this surface.

8. Discovery and integrity

Descriptor directories, highest priority first:

1. User config: $XDG_CONFIG_HOME/lemonade/backends/ or %APPDATA%\Lemonade\backends\.
2. User cache: <cache>/lemonade/backends/ or %USERPROFILE%\.cache\lemonade\backends\.
3. System: /usr/share/lemonade-server/backends/, /usr/local/share/lemonade-server/backends/, /Library/Application Support/Lemonade/backends/, /etc/lemonade/backends/, or %ProgramData%\Lemonade\backends\.

A descriptor in a higher-priority directory shadows the same recipe lower down. A recipe that collides with a built-in is rejected rather than shadowed.

Validation on POSIX rejects symlinked descriptor files, files with any group or world write bit (mode & 0022), user files not owned by geteuid(), system files not owned by root, and ancestor directories that are group or world writable without the sticky bit. On Windows it rejects owner SIDs outside the process user or administrators and DACLs that grant write to Everyone or Users.

lemond never downloads executables or model weights. Container images should be pinned by digest, because a mutable tag can change under you. This is a recommended best practice, not an enforced rule: the image reference lives inside the free-form command and args, which lemond does not parse. Enforcing digest pinning is a future enhancement that needs a first-class container field (image, runtime, devices, mounts) the server turns into runtime argv (section 19).

Descriptors that fail validation are rejected with a clear message and recorded in the registry's rejected list. They are never silently loaded. In this build the rejected list is not yet logged or exposed over an endpoint, which makes a misconfigured manifest harder to diagnose than it should be (section 21).

lemond scans the discovery directories when it first needs an external recipe, and /system-info enumerates them on first use. Once an external backend has been created, a background watcher re-scans every 2 seconds, so manifest edits are picked up without a restart.

9. Execution and lifecycle

At load time, lemond selects a platform block, resolves tokens, spawns the process, probes for readiness, and only then marks the model ready. On unload it runs the stop command and stops the child.

Platform selection, in order:

1. The requested device recipe option, when the manifest has a block with that exact key.
2. default_accelerator, when the manifest declares one and the block exists.
3. Runtime detection: metal on macOS, otherwise cuda when a CUDA architecture is detected, then rocm, then vulkan, gpu, and cpu.
4. The first accelerator key in the block, as a last resort.

If the manifest has no block for the host OS, the load fails with a clear error.

For a passthrough recipe, command and args are resolved into argv and env. For a variant_of recipe, lemond calls the built-in's launch plan, substitutes <cache>/external/<recipe>/<binary> for the executable, appends argv_extra, and merges the plan's environment with the manifest's env. A missing binary fails the load and names install-external.

The health probe runs with the configured timeout and poll interval. A child that dies during the probe fails fast. If the probe type is http, an after-ready watchdog keeps checking the endpoint. health_probe: process treats survival past spawn as ready, which is the weakest option and only fits engines that expose no port.

On unload, the stop command runs first (as an argv vector, with tokens resolved from the load-time snapshot), then lemond stops the child. On soft-idle downsize, lemond calls the manifest's downsize_endpoint if it declares one.

10. Management and consent

Models reference a recipe through user_models.json (recipe plus recipe_options). The management surface is lemonade backends (list, install, uninstall) and the external subcommands:

lemonade backends install-external <recipe> [--yes]
lemonade backends uninstall-external <recipe> [--yes]

These commands operate locally and do not require the server or an admin API key. Before doing anything, they print the manifest's provenance (display name, variant_of, source, sha256, version_policy, and declared capabilities) and this disclosure:

WARNING: this backend runs as an external subprocess. No process
         sandbox is enforced by this RFC; run only manifests you trust.

Confirmation is interactive. --yes skips the prompt. On a non-interactive terminal without --yes, the command refuses rather than assuming consent.

install-external downloads and verifies a variant_of binary. It picks the artifact for the host platform; if the manifest offers more than one artifact for this host OS, it requires --accelerator and names the choices. For a passthrough manifest with no source, there is nothing to download, and the command reports that the binary is user-provided. uninstall-external removes the manifest and the install directory, and refuses to remove a system descriptor (remove those with the package manager).

lemonade backends --all lists discovered recipes, and external recipes appear in GET /v1/system-info with "is_external": true.

When the Sandbox RFC adds the grant block as an additive contract change, grants and the re-consent-on-widening rule move there and are surfaced through the same consent flow.

11. Versioning and compatibility

Schema major equals api_contract_version. The server accepts manifests at or below its supported major and rejects a newer one with a clear message rather than partially honoring it. Evolution is additive: per-major schema files, a lock file pinning each released major, load-time shims that upgrade older documents to the latest internal form, and retention of every prior major. Because the versioned schema is additionalProperties: false, the only additive-within-a-major surfaces are the explicitly open extensions bag and recipe_options. A new first-class field, capability name, or token requires a new major. The v1 schema is released: false until it is published; editing it before then requires a reviewed lock refresh.

12. Security boundaries

Two things sit on opposite sides of a line, and conflating them is the mistake this section exists to prevent.

What this RFC does. It validates descriptors and their file permissions, restricts tokens to a closed vocabulary, gates {env:} through a default-deny allowlist, rejects leading-dash token values, enforces reserved_args, and keeps lemond from downloading executables or model weights. The CLI hash-verifies the binaries it does download.

What this RFC does not do. It does not confine the child process. There is no filesystem confinement, no ambient-environment scrubbing, no egress policy, and no sandbox grant block. An external backend runs as a subprocess with lemond's own privileges, so a manifest can run any command the lemond user can run. Treat every manifest as trusted code. Those protections belong to the companion Sandbox RFC, which also owns sandbox status and grant consent.

One gap inside the boundary: manifest env keys are literal and are not filtered against loader variables such as LD_PRELOAD or LD_LIBRARY_PATH. The allowlist governs the {env:} token, not the env block's keys. Until the Sandbox RFC lands, a manifest that you choose to install can set any environment value it likes on its own child.

13. Worked examples

The three worked examples below follow.

13.1 Passthrough: a user-installed Vulkan engine

{
  "recipe": "llamacpp-vulkan-custom",
  "display_name": "llama.cpp Server (Vulkan Custom)",
  "api_contract_version": "1",
  "capabilities": ["chat_completion", "completion"],
  "slot_policy": "standard",
  "health_probe": {
    "type": "http",
    "endpoint": "/health",
    "expected_status": 200,
    "timeout_seconds": 60,
    "poll_interval_ms": 200
  },
  "platforms": {
    "linux": {
      "vulkan": {
        "command": "/opt/lemonade/bin/llamacpp/vulkan/llama-server",
        "args": [
          "-m", "{checkpoint:main}",
          "--host", "{host}",
          "--port", "{port}",
          "-c", "{ctx_size}",
          "-t", "{threads}"
        ],
        "stop_command": "kill",
        "stop_command_args": ["-9", "{pid}"],
        "env": {
          "GGML_VK_VISIBLE_DEVICES": "{custom:vk_device:-0}"
        }
      }
    }
  }
}

You install and manage the binary yourself. lemond runs it with the tokens resolved to the current model and port. There is nothing to download and no source field.

13.2 Passthrough: a digest-pinned container

The containerized dflash-rocm example (abridged here to the parts that matter) runs a container with podman, mounts the model directory read-only, and pins the image by digest:

{
  "recipe": "dflash-rocm",
  "api_contract_version": "1",
  "capabilities": ["chat_completion", "completion"],
  "health_probe": {"type": "http", "endpoint": "/health", "timeout_seconds": 300, "poll_interval_ms": 500},
  "platforms": {
    "linux": {
      "rocm": {
        "command": "podman",
        "args": [
          "run", "--rm", "--name", "lemonade-{recipe}-{port}",
          "--device", "/dev/kfd", "--device", "/dev/dri",
          "--group-add", "video", "--group-add", "render",
          "-v", "{model_dir}:/models:ro",
          "--network", "host",
          "ghcr.io/luce-org/lucebox-hub:rocm-7.2@sha256:9f3a...c2",
          "/opt/lucebox-hub/server/build/dflash_server",
          "/models/{checkpoint_relative:main}",
          "--host", "{host}", "--port", "{port}"
        ],
        "stop_command": "podman",
        "stop_command_args": ["rm", "-f", "-t", "0", "lemonade-{recipe}-{port}"]
      }
    }
  }
}

The stop command is not a shell pipeline, it is podman rm. The image digest is the pin; a mutable tag would not be. Nothing enforces the digest: it is a convention the author follows and a reviewer can check by reading the args.

13.3 variant_of: a llama.cpp fork or nightly

{
  "recipe": "llamacpp-rocm-nightly",
  "display_name": "llama.cpp ROCm Nightly (community fork)",
  "api_contract_version": "1",
  "variant_of": "llamacpp",
  "capabilities": ["chat_completion", "completion", "embeddings"],
  "source": "https://example.invalid/llamacpp-rocm/nightly/llama-server-linux-rocm.tar.gz",
  "version_policy": "pinned",
  "sha256": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "platforms": {
    "linux": {
      "rocm": {
        "binary": "llama-server",
        "argv_extra": ["--gpu-layers", "{custom:gpu_layers:-0}"],
        "reserved_args": ["--port", "--host", "-m"]
      }
    }
  }
}

install-external fetches the archive, verifies the hash, extracts it under <cache>/external/<recipe>/, and lemond launches that binary with the built-in's argv. argv_extra adds only the fork's own flags, and reserved_args stops a model option from overriding the flags lemond already builds.

13.4 variant_of: one artifact per platform

A fork that publishes a separate archive for each accelerator sets the provenance inside each platform block instead of at the top level. The top-level source stays unset, and every block carries its own URL and hash:

{
  "recipe": "llamacpp-per-platform",
  "display_name": "llama.cpp Fork (one artifact per accelerator)",
  "api_contract_version": "1",
  "variant_of": "llamacpp",
  "capabilities": ["chat_completion", "completion"],
  "platforms": {
    "linux": {
      "rocm": {
        "binary": "llama-server",
        "source": "https://example.invalid/llamacpp-fork/linux-rocm.tar.gz",
        "sha256": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
        "argv_extra": ["--gpu-layers", "999"],
        "reserved_args": ["--port", "--host", "-m"]
      },
      "cuda": {
        "binary": "llama-server",
        "source": "https://example.invalid/llamacpp-fork/linux-cuda.tar.gz",
        "sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222"
      },
      "cpu": {
        "binary": "llama-server",
        "source": "https://example.invalid/llamacpp-fork/linux-cpu.tar.gz",
        "sha256": "sha256:3333333333333333333333333333333333333333333333333333333333333333"
      }
    },
    "windows": {
      "cuda": {
        "binary": "llama-server.exe",
        "source": "https://example.invalid/llamacpp-fork/windows-cuda.zip",
        "sha256": "sha256:4444444444444444444444444444444444444444444444444444444444444444"
      }
    }
  }
}

On a Linux host with both a ROCm and a CUDA GPU, lemonade backends install-external llamacpp-per-platform cannot choose for you and fails with the available accelerators; --accelerator rocm (or cuda, or cpu) selects one. On a single-artifact host the choice is unambiguous. lemond then runs the block that matches its own platform selection, so the installed artifact and the launched argv agree.

13.5 Arch overrides and aliases

Two DRY shapes: one for a name that changes by arch, one for structure that changes by arch.

Image name by alias. One block, one alias table, no duplicated block:

{
  "recipe": "toolbox-arch-alias",
  "arch_aliases": { "gfx1151": "strix-halo", "gfx1150": "strix-point" },
  "api_contract_version": "1",
  "capabilities": ["chat_completion"],
  "platforms": {
    "linux": {
      "vulkan": {
        "command": "podman",
        "args": ["run", "--rm",
                 "docker.io/kyuz0/amd-{arch_alias}-toolboxes:vulkan-radv",
                 "llama-server", "-m", "/models/{checkpoint_relative:main}", "--port", "{port}"]
      }
    }
  }
}

On a gfx1151 host the image resolves to ...amd-strix-halo-toolboxes...; on gfx1150 it becomes ...amd-strix-point-toolboxes.... The naming rule lives in arch_aliases once. If the arch is unmapped, {arch_alias} fails loudly instead of substituting an empty string.

Per-arch artifacts by override. The base block holds the shared config; each arch names only its artifact and deltas:

{
  "recipe": "llamacpp-arch-overrides",
  "variant_of": "llamacpp",
  "arch_aliases": { "gfx1151": "strix-halo" },
  "api_contract_version": "1",
  "capabilities": ["chat_completion"],
  "platforms": {
    "linux": {
      "rocm": {
        "binary": "llama-server",
        "argv_extra": ["--device-name", "{arch_alias}"],
        "arch": {
          "gfx1151": {
            "source": "https://example.invalid/gfx1151.tar.gz",
            "sha256": "sha256:5555555555555555555555555555555555555555555555555555555555555555",
            "argv_extra": ["--mtp"]
          },
          "gfx115*": {
            "source": "https://example.invalid/gfx115x.tar.gz",
            "sha256": "sha256:6666666666666666666666666666666666666666666666666666666666666666"
          }
        }
      }
    }
  }
}

install-external llamacpp-arch-overrides --arch gfx1151 fetches the gfx1151 artifact; lemond merges the same entry when it launches. The gfx115* entry covers every other gfx115x arch from one declaration. Both examples are also shipped under the example manifests.

14. Scenarios

These are the concrete cases those pieces enable. Each one maps to fields defined in section 4.

A community llama.cpp build on Vulkan. You compiled llama-server yourself for a GPU Lemonade does not cover. You write a passthrough manifest with a linux.vulkan block, point command at the binary, and declare chat_completion and completion. Drop it in ~/.config/lemonade/backends/. It appears in lemonade backends --all, and once a model is registered against the recipe, requests flow. No rebuild.

A containerized engine pinned to a digest. You want an engine you cannot build locally, and you want it to run identically every time. A passthrough manifest with podman run and an image digest gives you that. {model_dir} mounts the model, {checkpoint_relative:main} points the container at it, and {recipe} and {port} keep the container name unique. lemond health-probes the port and later runs podman rm.

A fork of a built-in. Your whisper.cpp fork adds a flag. You do not want to describe the whole argv; you want the built-in's careful argument handling against your binary. A variant_of: "whispercpp" manifest with source, sha256, binary, and argv_extra does that. The CLI verifies the download, and lemond only ever launches what the CLI installed.

An engine that manages its own weights. An internal engine pulls its own model files out of band, so Lemonade must not try to download them. You set model_management: "self_managed" and register the model; lemond skips the pre-download and the completeness check. A deeper readiness handshake is deferred, so the engine still has to answer the health probe.

An engine that serves more than chat. A local server exposes embeddings, reranking, and a classify route. You declare all of them, and lemond gates each route on the capability you declared. A request to a route you did not declare comes back as unsupported_operation, not a confusing proxy error.

15. Maintenance plan

CI. JSON-Schema validation of manifests, the schema-lock test, registry unit tests (ownership and permissions, discovery priority, extends merging, token resolution, version rejection), and a live server integration suite on Linux x86_64 with a fake engine. Container smoke tests run where podman is available.

Human. macOS and Windows path validation runs through the existing cross-platform CI. New-engine recipes are community-submitted and triaged; they are never merged by automation alone. Adding a capability name or a token is a reviewed contract change (section 5).

Long term. The capability set and token vocabulary are additive and reviewed. Every new schema major is frozen in the lock file and kept.

16. Risks

• Manifest-as-scripting-language creep. Closed capability contracts, launch-time-only substitution, reserved_args, and graduation to a built-in keep the manifest from becoming a program.
• Malicious descriptor placement. Ownership and permission validation, a default-deny {env:} allowlist, no binary downloads by lemond, and hash pinning all raise the cost. They do not confine the process, which is the Sandbox RFC's job.
• The variant_of argv reuse is a real refactor across built-in backends. It lands incrementally; the initial implementation exposes launch plans for llamacpp, whispercpp, and sd-cpp.
• Diagnosability. Rejected descriptors are recorded but not surfaced, so a broken manifest looks like a missing recipe. This is the top follow-up.
• Ecosystem acceptance. variant_of plus the declarative surface answer the "declarative cannot cover existing backends" concern. 1:1 idempotency with built-in backends stays an explicit non-goal.

17. Breaking changes

None for existing users. The change is additive. One behavior change: descriptor files in the search paths are now validated strictly, so a non-compliant descriptor is rejected with a clear error instead of being silently loaded.

18. Non-goals

• 1:1 behavioral idempotency with built-in C++ backends.
• Arbitrary new public endpoints.
• A scripting language inside the manifest.
• Binary distribution infrastructure.
• In-process plugins.
• The out-of-process adapter tier.
• The marketplace.
• Container runtime management: pulling images, detecting Podman or Docker, and advising on installation. The runtime owns those.
• A standard container interface (declared devices, network, supported models).

19. Deferred items

• The sandbox grant block, enforcement, and sandbox status (Sandbox RFC).
• The adapter tier: a queryable fact channel, a lease and VRAM broker, and protocol translation.
• The marketplace.
• Deep self_managed support: the model-inventory RPC and readiness handshake.
• Host driver and kernel pre-checks against the engine, and model-format declarations. Arch selection itself is covered by the arch overrides in section 4.6.
• Multi-port requested_ports semantics beyond the single-port v1 contract.
• Request-time behavior of nonstandard engines (custom error shapes, non-SSE streaming) surfaces at runtime, not at validation.
• A provisioning layer that fetches an artifact by shelling a command or over HTTP and can wrap a variant_of argv in a container runtime. Today provisioning is a single HTTPS download with a pinned hash; anything richer needs its own RFC.
• Future enhancement: first-class container support in the schema (image, runtime, devices, mounts), with digest pinning required by the schema. Until then, digest pinning is a recommended best practice the author and reviewer honor.

20. Design decisions

#	Decision	Why
D1	The sandbox block is removed from RFC01.	Declaration without enforcement is the drift the split prevents.
D2	Extensions are kept as a superset over the core set.	They preserve the working prototype surface and stay additive and route-safe.
D3	Consent is scoped to provenance and disclosure until the Sandbox RFC.	There are no grants to consent to yet; the flow stays minimal.
D4	{env:VAR} is allowlist-only and default-deny.	Ambient secrets must not reach a command line.
D5	Capability gating is per route.	chat_completion and completion are distinct routes; the bundled interface is too coarse.
D6	Reuse the routing schema-lock machinery.	One versioning story; frozen majors become mechanical.
D7	Clean-room rewrite rather than porting the prototype.	The prototype predates main's generated-descriptor and mode-contract rework.
D8	An open extensions bag allows in-major additions.	A strict schema would otherwise make every addition a major bump.
D9	streaming_transcription is variant_of-only in v1.	Realtime WS needs protocol translation, an explicit non-goal.

21. Scope of the first implementation

This RFC lands as a series of additive changes. Built-in backends and existing configs are untouched.

The first implementation covers:

• Manifest schema and strict validation: required api_contract_version, unknown field, unknown capability, unknown token, provenance conditional, and extends/variant_of exclusivity rejection.
• Discovery with POSIX ownership and mode checks, Windows DACL checks, path priority, and rejection recording.
• The full token vocabulary, allowlist-only environment tokens, the negative-number exemption, reserved-argument enforcement, and {custom_args} expansion.
• Route-level capability gating across the router, including responses.
• Passthrough execution: spawn, http/tcp/process health probes with fail-fast, platform selection, two-stage unload, and endpoint and downsize forwarding.
• variant_of through the launch-plan seam, provided by llamacpp, whispercpp, and sd-cpp, with per-platform and per-arch source/sha256/version_policy overrides, arch_aliases with the {arch_alias} token, CLI-side hash-verified install, and --accelerator/--arch selection when a recipe publishes one artifact per platform or arch.
• extends merging recipe_options and custom_options with cycle detection.
• self_managed skipping the pre-download and completeness check.
• capability_enable_args applied per deployment mode, and downsize_endpoint invoked on soft-idle downsize.
• External recipe_options and custom_options registered so {custom:} and {custom_args} resolve from model options.
• Management: /system-info listing, the discovery watcher, and lemonade backends install-external / uninstall-external with a provenance and no-sandbox disclosure.

Not in the first implementation (none of these contradict the schema):

• variant_of covers only built-ins that expose a launch plan. The others return a clear error until they do.
• requested_ports > 1 is rejected; v1 is single-port.
• The deeper self_managed readiness RPC is deferred. The flag only suppresses auto-download and completeness checks.
• Rejected descriptors are recorded but not yet logged at startup or exposed over an endpoint.
• Image upscale sits outside the image capability.
• The sandbox grant block and all process confinement belong to the companion Sandbox RFC.

22. Proposals this RFC resolves

Two open proposals shaped this surface. Both are served, in part, by what is described here, and each is explicitly bounded.

#3583, "RFC: user-defined list of backend forks." The proposal asks for a named list of alternate builds of an existing backend and a way to pick one per model. variant_of is the mechanism. Each fork is a small manifest that names the built-in it is wire-compatible with (variant_of: llamacpp), where the binary comes from (source, sha256, version_policy), and only the argv deltas (argv_extra, reserved_args). A model picks a fork by naming its recipe, and lemond reuses the built-in's argv construction, so adding a fork needs no C++ and no rebuild. Two differences from the exact CLI in the proposal are deliberate: selection is by recipe at model registration rather than a --backend flag on load, and every fork is pin-verified and separately supervised. The *_bin config override the proposal currently relies on is unaffected by this RFC.

#3585, "Support for OCI backends." A containerized engine is a passthrough manifest: command is podman or docker, args are run ... image@sha256:..., and the runtime pulls and caches the image itself. The digest is the pin, the runtime is the isolation boundary, and the dflash-rocm example is exactly this shape. A container-only engine such as HaloGen fits as a passthrough recipe with its own argv. A container image that happens to be a fork of a built-in, such as a llama.cpp or whisper container, also fits as passthrough, not as variant_of, for the reason in section 7: variant_of is about backend type, not runtime.

What this RFC does not add for #3585, and why:

• Image pull, runtime detection, and "install Podman" guidance are the runtime's job, not the manifest's. A provisioning step that shells the runtime or downloads an artifact is a different concern and belongs in its own RFC.
• A standard container interface (declared devices and network, suggested models) goes with the sandbox and adapter work.
• Process confinement is the companion Sandbox RFC.

The split to hold onto: variant_of answers which built-in's argument handling a binary follows, and runtime answers how that binary is obtained and launched. Keeping them separate is what lets one mechanism describe a host fork and the other describe a container without either pretending to be the other.

23. References

• The JSON Schema for the external backend manifest (contract major 1) and its lock file (normative)
• The schema-versioning README and the schema-lock test (versioning precedent)
• The external backends user guide and maintainer guide
• The worked example manifests
• Discussion #3585, OCI backends
• Issue #3583, user-defined list of backend forks
• PR feat(server,router): add dynamic custom backend discovery and process isolation #2880, dynamic custom backend prototype (reference)
• The Lemonade Backend Gateway: Research and design output based on various maintainer feedback, testing and verification
• Companion: RFC Backend Sandbox Architecture and Permission Policy

[abn]

Testing external backends: Donato's Strix Halo toolboxes

This branch adds **external backends**: a JSON manifest describes an out-of-tree engine, and `lemond` discovers it, launches it as a subprocess, health-checks it, and forwards its OpenAI routes to it. No C++ rebuild for a new engine.

This walkthrough is the OCI-container use case from #3585. You run one of Donato's kyuz0/amd-strix-halo-toolboxes containers as a Lemonade backend. The container is a hermetic userspace: the host supplies only the GPU device nodes, and the runtime pulls and isolates the image. Lemonade does not wrap it again.

What you are exercising: manifest discovery, platform (OS x accelerator) selection, container launch and teardown, health probing, and route proxying.

These images are built for Strix Halo (gfx1151). On other AMD GPUs the container starts and loads the model but fails at the first GPU op (ROCm error: device kernel image is invalid). That is an engine build mismatch, not a Lemonade error.

Prerequisites

• Linux with an AMD Strix Halo GPU (gfx1151).
• podman (or docker; this guide uses podman).
• Build tools per the repository README (CMake, Ninja, a C++17 compiler, libcurl, openssl).
• Network access to pull the container image and the test model.

1. Get and build the branch

git clone https://github.com/abn/lemonade.git lemonade
cd lemonade
git checkout feat/external-backend-system
./setup.sh
cmake --build --preset default --target lemond lemonade

2. Create the scratch dirs and the manifest

The manifest must exist before the server starts. Put it under XDG_CONFIG_HOME and make it owner-only.

export TEST=/tmp/lemonade-toolbox-test
mkdir -p "$TEST"/{cache,config,xdg/lemonade/backends,hf}

Write the file below to $TEST/xdg/lemonade/backends/toolbox-llamacpp.json, then chmod 600 it. It declares two platform blocks; lemond picks the one matching your detected hardware. --security-opt label=disable is required on SELinux hosts so the container can read the model mount.

{
  "recipe": "toolbox-llamacpp",
  "display_name": "kyuz0 Strix Halo llama.cpp toolbox",
  "api_contract_version": "1",
  "capabilities": ["chat_completion", "completion"],
  "health_probe": {
    "type": "http", "endpoint": "/health", "expected_status": 200,
    "timeout_seconds": 300, "poll_interval_ms": 500
  },
  "platforms": {
    "linux": {
      "vulkan": {
        "command": "podman",
        "args": [
          "run", "--rm", "--init", "--name", "lemonade-{recipe}-{port}",
          "--device", "/dev/dri", "--group-add", "video",
          "--security-opt", "seccomp=unconfined",
          "--security-opt", "label=disable",
          "-v", "{hf_cache}:/models:ro", "--network", "host",
          "docker.io/kyuz0/amd-strix-halo-toolboxes:vulkan-radv",
          "llama-server", "-m", "/models/{checkpoint_relative:main}",
          "--host", "127.0.0.1", "--port", "{port}", "-c", "{ctx_size}"
        ],
        "stop_command": "podman",
        "stop_command_args": ["rm", "-f", "lemonade-{recipe}-{port}"]
      },
      "rocm": {
        "command": "podman",
        "args": [
          "run", "--rm", "--init", "--name", "lemonade-{recipe}-{port}",
          "--device", "/dev/dri", "--device", "/dev/kfd",
          "--group-add", "video", "--group-add", "render",
          "--security-opt", "seccomp=unconfined",
          "--security-opt", "label=disable",
          "-v", "{hf_cache}:/models:ro", "--network", "host",
          "docker.io/kyuz0/amd-strix-halo-toolboxes:rocm-10.0",
          "llama-server", "-m", "/models/{checkpoint_relative:main}",
          "--host", "127.0.0.1", "--port", "{port}", "-c", "{ctx_size}"
        ],
        "stop_command": "podman",
        "stop_command_args": ["rm", "-f", "lemonade-{recipe}-{port}"]
      }
    }
  }
}

If the image does not accept llama-server as its command, check with
podman run --rm docker.io/kyuz0/amd-strix-halo-toolboxes:vulkan-radv which llama-server
and, if needed, add "--entrypoint", "llama-server" before the image and remove the bare llama-server token.

The image is named explicitly because these toolboxes exist for Strix Halo (gfx1151) only. When a recipe must name a different image or artifact per hardware arch, a platform block accepts an arch map of arch glob to partial overrides, and a top-level arch_aliases map resolves the {arch_alias} token; the shipped docs/examples/external_backends/toolbox_arch_alias.json shows an image name that varies by arch. This doc keeps the concrete image so the Strix Halo path is unambiguous.

3. Start an isolated server

Use a scratch directory and a non-default port so you do not touch any Lemonade you already run.

XDG_CONFIG_HOME="$TEST/xdg" HF_HOME="$TEST/hf" \
  ./build/lemond "$TEST/cache" "$TEST/config" --port 13350 &

Confirm it can see the recipe:

curl -s http://127.0.0.1:13350/api/v1/system-info \
  | python3 -c 'import sys,json; print("toolbox-llamacpp" in json.load(sys.stdin)["recipes"])'

4. Register a small model, load, and chat

Custom models must be named user.*. If you have another Lemonade running, add --no-discovery so the CLI talks to this server.

./build/lemonade --no-discovery --port 13350 pull user.toolbox-270m \
  --recipe toolbox-llamacpp \
  --checkpoint main unsloth/gemma-3-270m-it-GGUF:gemma-3-270m-it-UD-IQ2_M.gguf

./build/lemonade --no-discovery --port 13350 load user.toolbox-270m

curl -s http://127.0.0.1:13350/api/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"user.toolbox-270m","messages":[{"role":"user","content":"In one word, say OK."}],"max_tokens":8}'

A short OK-style completion means the full path works: discovery, platform pick, container launch, health probe, and proxy. On a gfx1150 host the load instead fails with a ROCm kernel error inside the container; see the note above.

5. Confirm lifecycle

podman ps --filter name=lemonade-            # the engine container is running
./build/lemonade --no-discovery --port 13350 unload
podman ps -a --filter name=lemonade-         # empty: the container was removed
kill %1                                       # stop the test server

6. Pin the image by digest (recommended)

Tags move; digests do not. podman pull docker.io/kyuz0/amd-strix-halo-toolboxes:vulkan-radv, then
podman inspect --format '{{index .RepoDigests 0}}' docker.io/kyuz0/amd-strix-halo-toolboxes:vulkan-radv,
and replace the tag in the manifest with the printed ...@sha256:.... Digest pinning is a best practice, not yet enforced.

What to report

Please note what worked and what did not, with the OS, kernel, GPU (gfx arch), podman --version, the selected platform block, and the relevant lemond log lines. A container that fails to launch, a health probe that times out, or an unexpected route response are all useful. Known limits: this is Linux only, container image pulls belong to the runtime, and confinement beyond the container is out of scope for this branch.

[abn]

Update: This is an archive for anyone who wants to build on it, or parts of it. Other in-flight RFCs have momentum and priority. I'm not maintaining or pushing this beyond this point - fork and take what you want. Happy to answer questions if someone picks it up.