RFC: Support for OCI backends, especially the HaloGen, ROCmFPX, and NathanW containers from kyuz0 (Donato)

[jeremyfowers]

Proposal size

Major feature (spans multiple well-scoped PRs)

Updates

1. Clarify that this proposal is Linux-only since that is what Donato's toolboxes support.
2. Committed the decision to replace our DS4 native binaries with the toolbox. (Also I am going to add an architecture section to this, stay tuned.)
3. A detailed specification for this feature has been opened for review at docs: add the container backends spec #3697. Let's consider this discussion closed, and move the conversation to that PR.

User story

Engine innovation is happening fast on Strix Halo, with the emergence of llama.cpp forks like DwarfStar4, ROCmFPX, NathanW, along with new engines like Halogen. Users want to try out new optimizations before they land in mainline core engines like llamacpp. Devs also need a way to do apples-to-apples benchmarking and agentic tests across engines/backends.

The core proposal here is to enable OCI containers on Linux to serve as Lemonade engines/backends, and then to leverage this capability to support new experimental engines in a way that is both agile and secure. Future work may include defining a standard interface for containers that makes these Lemonade integrations even easier in the future.

User stories enabled:

• First class UI/UX for installing new experimental engines via container, for example lemonade install halogen:rocm
• First class UI/UX for managing models: lemonade pull Qwen3.8-27B-ROCmFP4
• Benchmark across core and experimental engines: lemonade bench Qwen3.8-27B-GGUF Qwen3.8-27B-ROCmFP4
• Evaluate agentic performance lemonade launch pi -m MODEL --agent-args="-p TEST PROMPT" across multiple core and experimental backends.
• Do all of the above with confidence, since the experimental engine is confined to a container.

Note: Windows support is not in scope because the toolboxes we're taking a dependence on are Linux-only.

High-level design

All of the following is done in collaboration with @kyuz0, whose recent video on Qwen Flash Next inspired this RFC!

Container source

There are two major problems with adding any new experimental backend to Lemonade today:

1. How can we be sure that running it is safe and secure for Lemonade's userbase?
2. How will we maintain the backend over time?

This proposal attempts to address both by leveraging @kyuz0's work. He already maintains a Docker Hub with curated containers for popular experimental backends, including a lot of backends that Lemonade doesn't support yet. He is also a security professional and pulling a backend from him is a lot more credible than getting it from a relatively unknown GitHub.

OCI containers as backends

All of our WrappedServers today assume that they are launching native processes. New features:

• Add formal support for pulling and launching an OCI containers by calling the appropriate Podman/Docker command.
• Advise the user, via "actions required" and error messages, on how to install Podman if needed.
• Allow any backend for any engine can be a container. Examples:
  • Halogen would be a container-only engine with 1 backend
  • NathanW would be a container backend to the established llamacpp engine "recipe"

New Engines

I am proposing to add the following new engines. The first PR will introduce both an engine and the general OCI support, while subsequent PRs should just incremental add 1 engine per PR.

My proposed order:

1. DwarfStar4: convert our existing DS4 support from our own native build to use @kyuz0's container instead. (if we ever decide to promote DS4 to a Lemonade core backend we can revive our native build). This let's us try out the OCI process without simultaneously trying out a new backend.
2. NathanW: as a fork of llama.cpp that can run GGUF, this is relatively low risk as well. It would show up as a new llamacpp backend.
3. ROCmFPX: this one only runs its own specialized GGUF files, so it is the next biggest lift.
4. Halogen: leaving this one for last to give it time to open-source.

Breaking changes

Support for our native DS4 binaries is replaced by the container, so we will need to auto-migrate people (help them delete their native installs).

Maintenance plan

This work will include automated GitHub actions that pull new container versions and curated models from @kyuz0. He has already set up actions that automatically update his containers when the upstream backends have releases. Our CI will validate that container-lemonade integration still works, so we are expecting minimal manual maintenance.

In the future, it would be great to define a container interface standard that would make maintenance even easier. For example, if containers support an endpoint that lists their supported/suggested models, then Lemonade could read that to auto-update a supported models list.

Risks

The security risks of supporting experimental projects should be mitigated by containers and by @kyuz0's expertise.

[jeremyfowers]

BTW rocmfpx is our number 1 most upvoted feature request, with 15 votes: #2089

[charlie12345]

I love this idea! so many great projects

[Geramy]

@jeremyfowers are we proposing OCI over nono or whatever the other one was that @abn had suggested before? Is there a reason for that also? I feel like expecting Podman / docker might be a hard pill to swallow. Perhaps maybe we do nono as fallback if no Podman / docker?

[Geramy]

It isn’t either-or. My understanding is that nono is a solution to native binaries, while containers solve a different problem. These are also not plugins, this RFC is for in-tree support of containers vetted by a security expert.

I feel like expecting Podman / docker might be a hard pill to swallow.

It’s for people who would be accessing these things via container anyways, Donato’s toolboxes are very popular. People who dont want containers and want to stick to mature native backends can keep using mainline llamacpp.

Okay that makes sense, so this is a use case scenario for podman / docker support in lemonade. I love that its great!

[Geramy]

PS. my understanding is that the nono.sh PR got stuck in discussion… if you want nono.sh please help unstuck it!

yeah I better!

[abn]

@jeremyfowers #2880 was originally created to support both binary and container based solutions. You can see "Testing & Verification Instructions" in the followup comment for examples.

Nono came up because sandboxing was flagged as a mandatory prerequisite before progress could be made on that. And then we went down the sandboxing path and #3212 was created and is still awaiting testing and feedback. I personally don't think sandboxing should be prerequisite for the #2880.

Also note that there were other feedback around the complexities that backend wrapping logic today handles, and there also were other contentions iirc.

That said I am all for any solution that enables the adoption of and orchestration of externally provided/maintained backends. Especially if it's integrated with oci runtimes natively.

[jeremyfowers]

I personally don't think sandboxing should be prerequisite for the #2880.

This is worth revisiting. Sandboxing is important if we force something onto people. But if they are installing plugins of their own free will they should probably understand what the plugin is?

[jeremyfowers]

Very related to #3583 btw

[iswaryaalex]

great to define a container interface standard that would make maintenance even easier

+1 on the standard container interface.
For eg, if it declares required devices and whether it needs network, alongside supported models, Lemonade could launch each engine at least privilege automatically instead of applying one permissive profile to all of them.

[Geramy]

Would be nice to add firewalling in here some day too. Perhaps we could make a lemonade virtual switch that lemonade directly controls over the default? That would give us more control later without effecting the existing systems architecture?

[Rose22]

this fork of llamacpp is the combined effort of some of the popular strix-halo llamacpp devs: NathanW, Laurent (AgentionAI), rocmfpx, and others: https://github.com/halo-box/strix-llama.cpp

it supersedes them all (though not everything from the other forks is merged yet) because this is all the fork devs essentially combining all the forks into one central strix halo-focused llamacpp fork

so, please add support for it!

[iswaryaalex]

This is really cool @Rose22

[jeremyfowers]

I would definitely consider it! But @kyuz0 would need to support it here first (at least to fit into this plan): https://github.com/kyuz0/amd-strix-halo-toolboxes

[kyuz0]

@Rose22 I am ware of that fork, my understanding was this wasn't ready yet but I'll speak to the maintainers to see where they are at.

[indistinctTalk]

Loving the interest in broader backend support but for users already running lemonade in a container, how is this expected to work? Passing in the docker socket?

[storm1er]

Using DinD and not DonD, docker socket does not need to be mounted.

https://github.com/docker-library/docker/blob/master/Dockerfile-dind.template

https://forums.docker.com/t/is-that-possibile-use-the-docker-in-docker-without-mounting-the-docker-socket-from-host-machine-into-the-container-using-ubuntu-22-04-base-image/139252/2

But it MUST BE crafted into the lemonade docker image for it to work :/

[indistinctTalk]

For context on why I'm raising, I'm currently running lemonade via a helm chart I've crafted (ugly vibe-code smashed together) which is doing the needfuls for implementing custom backends. At face value it seems like the RFC might complicate things for me, but as a single user I don't think my experience should hold much weight. 😅

[PhilWheat]

@indistinctTalk I was the one asking below about Windows support, I completely understand your angle. I don't actually run it under Windows but have and a big selling point of Lemonade is that it simply runs on whatever you're running. I can understand if not everything can be supported everywhere, but each time there's an "only supported on this platform" branch, that story gets a bit weaker.

[indistinctTalk]

Yeah I think it would be a loss if lemonade-orchestrated containers were to become the only way to support custom backends, but I'm not sure that's being proposed. Maybe I need to reread the RFC though. 😂

[jeremyfowers]

if lemonade-orchestrated containers were to become the only way to support custom backends, but I'm not sure that's being proposed

That's not what's being proposed :) All that is being proposed is that Donato makes some great toolboxes and people should be able to use them via Lemonade if their system supports the toolbox.

I can understand if not everything can be supported everywhere, but each time there's an "only supported on this platform" branch, that story gets a bit weaker.

@PhilWheat I understand this, and there will always continue to be non-container core backends that work cross-platform. The constraint we're addressing here is that maintaining a core backend to our highest quality standard is a large amount of work and doesn't scale. How do we also engage the bleeding edge of innovation? This RFC is one answer to that; I haven't seen any achievable alternative yet.

[PhilWheat]

How will this impact Lemonade on Windows? WSL OCI container support seems to be a future thing and the referenced toolbox containers aren't windows compatible as far as I know.

[jeremyfowers]

I clarified on the RFC that this is Linux-only because the upstream dependence on Strix Halo Toolboxes is Linux-only :(

Out of scope for this RFC, but if someone wanted to bring more functionality to Windows maybe @charlie12345's vLLM-for-Windows would be a good one?

[iswaryaalex]

@jeremyfowers the RFC doesn't say, and I think it needs to, about Windows support for custom backends like this. Podman and Docker means WSL which is a whole new topic

We can also just call it out in this RFC as Linux only feature

[jeremyfowers]

#3585 (reply in thread)

[Geramy]

Windows can support docker without WSL2 as well. Like someone can install docker on windows and we as lemonade get the same docker sock library. Also keep in mind docker on Mac requires a different level of API, its not localhost:5948/docker or w.e its a sock I believe. I'll provide some specifications let me know where.

[admcpr]

While technically windows can support docker without WSL2 it's not really feasible because you're going to need some other Hyper-V Linux VM running docker engine or docker VMM for that to work which is not going to provide a happy support path.

[superm1]

The core proposal here is to enable OCI containers on Linux to serve as Lemonade engines/backends

So I don't really want to pigeon-hole us that OCI containers are only as engines/backends. We could totally become an orchestrator for containers too. Think like managing a Hermes OCI container or Claw OCI container. We then become the glue that "links" the OCI containers for backends to OCI containers for frontends.

I don't think it necessarily needs to be done immediately as it's a big scope creep from the original RFC, but I want to make sure this design doesn't preclude it and make it more difficult to accomplish.

[superm1]

Love the idea of managing a hermes container! But yeah that’s its own RFC.

Is there anything specific in the design above that needs fixing to avoid precluding? Or anything you need me to specify in more depth so that you can check it?

I don't think this spec is detailed enough to say one way or another.

So at the level of this spec you should basically make it a bullet along the lines of don't name configuration options anything that wouldnt scale. ie rather than "container" maybe "container-engine".

Don't assume a container starting/stopping API is always for engines.

If you make a more detailed spec we can see if anything falls apart.

[jeremyfowers]

I’ll try to get more detail onto here tomorrow as long as nothing comes up with release prep!

But just to give you an idea, I’m not suggesting any public container start/stop API. Donato’s containers would be started deep in the code by the existing load endpoint and stopped by the existing unload endpoint.

[abn]

Donato’s containers would be started deep in the code by the existing load endpoint and stopped by the existing unload endpoint.

Couple of things. The container lifecycle itself is actually managed outside of the client that ran it. The closest thing to control you can get is to run the container in the foreground as you would the llama.cpp server. Also food to remember might me the container image lifecycle itself you'll have to adjust pull policy etc as well according to the behavior you want.

The command examples in #3594 might be useful.

Edit: also remember there are security considerations here dependending on the distro and the user's security posture. For example you'd need to configure seccomp profile. For systems with SELinux enforcing you will need to ensure mounts (model paths) are correctly managed, similar for app armor iirc. You can see the comment here for examples running Donato's containers.

[superm1]

To add to @abn's comments; the container will be registered with software that isn't lemonade. For example podman or docker will be what's managing it. A user/software can start/stop it outside lemonade. So lemonade needs to interact with that software.

This has non-obvious implications.

• Lemonade will need to monitor if the container is alive when you think it should be (for example if it crashed or was stopped by other software).
• Lemonade will have to check the hash is what it expects every time it starts the container. If not; pull the right hash.
• Lemonade will have to make sure that the launch arguments are what is expected and if not stop and start it with the right arguments.

[jeremyfowers]

@superm1 aren't all of these true of the sub processes Lemonade already launches from native backend binaries? Is there any new consideration specific to containers?

[superm1]

This work will include automated GitHub actions that pull new container versions

I don't see anything in this design that talks about the security of these containers. Here's the areas that I think need to be discussed.

1. We need to be prescriptive about how the containers are launched and what parameters can be passed in to launch arguments. IE if someone can just bind mount a path into the container from a lemonade setting that's too dangerous. I would vote no arguments, and we need to add them on case-by-case basis later.
2. Nothing discusses the pinning. All containers must be pinned to a hash (not a tag).
3. What registry will we support pulling from?
4. How will you decide what GPU and NPU is passed into the container?
5. How do we manage the lifecycle of the container? We might look at https://ramalama.ai/ for inspiration in this RFC.

[Geramy]

Right, we currently dont have multi-gpu or single npu support in this fashion, I believe this is a important piece also.

[jeremyfowers]

Quick answers:

• Only Donato’s registry, in the scope of this RFC. Nobody else’s.
• Only GPU, not NPU.
• I would be fine with single GPU only in this scope.

[superm1]

Only Donato’s registry, in the scope of this RFC. Nobody else’s.

I'll admit I don't know where Donato's registry is. The spec needs to be prescriptive.
IE; we trust Donato's registry host? Or do you mean his account on an existing public registry like Github.io or Docker.io. Whichever it is needs to be clarified.

Only GPU, not NPU.
I would be fine with single GPU only in this scope.

One GPU is fine, but how do we know which GPU to pass in? Existing backends that run on the host have access to all GPUs. Containers need to have specific GPUs passed in.

[jeremyfowers]

I'll admit I don't know where Donato's registry is

https://hub.docker.com/r/kyuz0/amd-strix-halo-toolboxes/tags

It's the #1 worldwide recommendation for new users getting started with Strix Halo (Lemonade is usually #2 😉 )

The spec needs to be prescriptive.

I wasn't intending this to be a spec, it's intended to be a request for comment on the yes/no question "do we want to support Donato's toolboxes?" Is this a question that you require to be answered before you would look at a PR? The PR would obviously have all such details.

[superm1]

@kenvandine can snaps start and stop OCI containers? What runtime can be used? Or this is going to be exclusively to native packages?

[kenvandine]

It would be work to keep it in the sandbox but we could potentially manage containers outside our sandbox with an interface, but then that requires more configuration on the host. It's great that right now its just a simple snap install and everything works, no drivers, no dependencies, etc.

[superm1]

In that case should snap be excluded from the spec?

[wenis]

Disclosure: this comment was written by Nick's AI assistant. Nick (halogen's author) asked me to review the RFC and the thread and reply with comments; the facts below come from halogen's repositories and release records, and Nick reviewed and approved this before posting. Commitments in it ("will", "happy to") are his.

Thanks for the RFC. On the ordering: the license is the gate, and Nick is moving it. A few notes that may make halogen the easy integration rather than the last one, since it already ships as a container and most of what @superm1 listed is how it works today.

What the images already are

• ghcr.io/peonist-ai/halogen-flash-server (Qwen3.8-Flash-Next) and ghcr.io/peonist-ai/halogen (Qwen3.8-27B): one long-running process per container, releases tagged, the digest published with each release. Nick's own release check is an anonymous pull by digest compared against the gated release candidate.
• No launch arguments. Configuration is HALOGEN_* environment variables only, and an invalid value refuses to start before the model loads.
• /health reports loading vs ready, with the engine and API versions and whether they match; /v1/models; chat and completions on both images. The Flash image also serves /v1/responses and /metrics, and carries a Docker healthcheck. The 27B image will be brought to parity before any integration lands.
• The images contain no weights. A container can fetch them itself or take a mounted directory, so lemonade pull can own the download and the engine container can run with no network.
• The release gate runs against the image, not a development build: an identity gate (output byte-identical to the previous release on fixed inputs), endpoint smokes, and a leak gate on the binary actually in the layer.

One launch requirement that has to be declarable

ROCm's HSA runtime, inside a container, needs --ipc=host; without it the process dies in about two seconds. shm_size does not substitute (measured at five sizes). Beyond that: /dev/kfd, /dev/dri, and the render/video groups (--group-add keep-groups on podman; explicit GIDs on docker). seccomp=unconfined is not needed; it was dropped from the Flash image after a user asked (#8), and the 27B image will follow. So "no user-passed arguments" works, provided a container can declare what it needs and Lemonade applies exactly that and nothing more.

The container interface standard: halogen can be the first image to carry it

Proposal: the declaration lives on the image as OCI labels, so Lemonade reads it before starting anything, e.g.

org.lemonade.devices = /dev/kfd,/dev/dri
org.lemonade.ipc     = host
org.lemonade.network = none
org.lemonade.port    = 8000
org.lemonade.env     = HALOGEN_CK,HALOGEN_MAX_TOKENS_DEFAULT,...   (settable keys)
org.lemonade.models  = <JSON: served models, files, sizes, sha256>

with the same models list served on an endpoint once running, for the auto-update idea in the maintenance plan. That covers @iswaryaalex's least-privilege launch and the "read supported models from the container" idea in one place. Nick is happy to implement it first and adjust to whatever shape the maintainers settle on.

Source of truth

A strong preference: Lemonade pulls the vendor's signed image by digest and @kyuz0 vets it, rather than rebuilding it inside a toolbox. Every published halogen number is per image; a rebuild with a different ROCm is a different engine (ROCm 10 alone moves prefill about +3% and is not bitwise with 7.14). Vetting a pinned digest is the stronger guarantee, it matches the pinning requirement above, and it scales to the other engines in this RFC.

License

Apache 2.0, both engines, weeks rather than months. Nick is also offering to help on the DS4 conversion PR, since that builds the plumbing this rides on.

Linux-only is fine; halogen targets gfx1151 by construction.

[wenis]

First, I want to let everyone know this project, and especially this RFC, is important to me. Me showing up with an AI generated post is a bad first impression. I will be more sensitive to the preference of the maintainers if they do not want AI generated comments in the future. The way I work now will never be the same and I will be respectful of the rules.

This is the human version of Nick, not Son of Anton.

I have a couple thoughts.

I am a massive fan of simplicity and removing technical barriers for users. Donato's toolboxes made it very easy for me to get an environment setup on my Strix Halo when I first started out, so if someone could try out halogen simply by running lemonade install halogen (I only care about ROCm) that would be an amazing UX.

Halogen (flash) can be picky about memory as it pins 68GB in a continuous block, then eats another 35 in KV cache. It's a memory hog at shipped defaults, regardless of what rocm-smi might tell you. I don't know how lemonade handles the idea of "minimum system requirements". But communicating to the user that it might blow up is probably wise. Halogen-flash-server was designed for a headless 128GB system and I don't want anyone to blame lemonade if halogen misbehaves.

Also, Donato's toolboxes as a single trusted source makes sense, so please disregard the suggestion above... and agree on the container registry or interface long term.

Regarding security, an issue came up when Halogen was first published because I had 0 trust and a docker-compose file that was too permissive. Well that and compiled binaries. I'm not an expert in security but containers are inherently more secure and principle of least privilege should apply. I can only attest that --ipc=host is needed to run the Halogen container in podman on Fedora. If there's a more secure way to access to the host, please let me know.

[jeremyfowers]

No worries! We're really glad you're here, @wenis :)

if someone could try out halogen simply by running lemonade install halogen (I only care about ROCm) that would be an amazing UX.

Let's do it!

I don't know how lemonade handles the idea of "minimum system requirements".

We can make sure the system has enough total RAM available, but we don't currently have a way to check for continuous available blocks. @superm1 is there a way to implement that? If not, we can handle it through an actionable error message like "Halogen requires 68GB of free available RAM to load, please make sure that is available and try again."

I don't want anyone to blame lemonade if halogen misbehaves.

The best way to avoid this is for you to be involved and give us tips like these! We're off to a good start.

Regarding security

IMO the best single thing would be to wait until Halogen is OSS before we add it. That way an independent security scan could attest that a specific tag's hash is safe for users. It will take us some time to add the basic container support and handle the easier containers, so hopefully you wont be waiting for us too long.

[superm1]

is there a way to implement that?

You mean like forcing an OOM? No thanks. Show a message.

[jeremyfowers]

@superm1 I'm asking if there's a way to check if an OOM would happen before attempting it in this specific situation.

[superm1]

I mean you can look at /proc/meminfo to see how much is free, but that's a slippery slope with things like swap and MEXT. The kernel will move some things around for you to swap.

[storm1er]

Off-topic about your AI generated post

Details

Me showing up with an AI generated post is a bad first impression

Don't worry, on my side there were a lot of green flag there:

• The message did start with a good disclaimer.
• You didn't let the AI respond wildly to everything.
• You did respond after and used what the AI pre-worked for you.
• You corrected and made up your own mind on what the subject was.
• You didn't edit the AI message to make it yours nor removed it like nothing happened.

That's a genuine use of AI to me. The only bad point I would make is the rules are the rules, if maintainer(s) said no ai generated comment, then just make sure your agent follow it from now on.

[ramkrishna2910]

In regards to testing the backends itself (not just the integration with lemonade) is it enough to rely only on external verification? Given the hardware we have in the dev lab we could either enable testing of the container backends itself or if it help we can onboard @kyuz0 to use the resources in our dev lab for CI.

[kyuz0]

Late to the party, my thoughts:

• I'm in favour of Lemonade pinning containers, and having CI pipelines that test them before release. It's something I've thought about myself but I simply do not have enough hardware to be able to do this in a timely fashion for all containers.
• In theory, every container should go through a performance and stability gate to identify regressions with core models, optionally also a benchmark like Terminal Bench Mini to identify inference bugs/regressions.
• There is indeed a significant security risk, this supply chain is inherently risky. Isolating containers from the network point of view would help mitigate immediate data exfiltration opportunities, but it also means all models must be downloaded outside of the containers and then made available by mounting specific folders (which is already happening for most toolboxes in AI Toolbox Cockpit)

[jeremyfowers]

I simply do not have enough hardware

We have a lot of hardware! There is a cloud of Strix Halo and Radeon servers we've been building specifically to support open source projects like this. Happy to discuss on an email, call, etc. any way we can support.

Isolating containers from the network point of view would help mitigate immediate data exfiltration opportunities

Let's definitely do this

all models must be downloaded outside of the containers

This is already the plan, Lemonade's model manager will be responsible for downloads and each container will only mount 1 model file/folder.

[Geramy]

Yeah it would be nice to have all containers point to the hosts model directory. So they can be accumulated in one spot and not duplicated :)

[abn]

You can delegate this to the model manager to derive the checkpoints and mount them with the appropriate flags. Remember ro,Z for allowing SELinux systems to mount the files.

You can see a reference of this in #3594 's implementation branch.

[pwilkin]

Okay, I'm pretty late to the party given that I've poked @jeremyfowers to move on with this, but here's how I imagine this works.

Since with containers, you can have a standard HTTP port where the API lives and you can redirect it to any port you want, this will not be a problem. As for the protocol, we're probably aiming for OpenAI-compatible where applicable and proposing our own protocol where there isn't an industry standard (3D-gen probably?).

The only thing this leaves is a format for the containers to describe their configuration. I assume this means each container is shipped with some sort of metadata that describes the input variables, preferrably with some standarization so that things with similar meanings for different engines (such as backend choice, eg. Vulkan vs ROCm) are passed on the same way. And in line with the security discussion, I do think it should be maximum isolation, i.e. the container has no access to the parent filesystem except for mounting the single folder that contains the model images (HF cache?).

[Geramy]

why don't we just making a hugging face of engines???? lol but seriously.
We have a standard config file you upload it and then there is a lemonade pull button.
Of course that's not a phase 1 thing but, I mean :O ???? Oh and we call it lemonface.

[pwilkin]

Yeah, that was my main fantasy here as well :)

[abn]

We have a standard config file you upload it and then there is a lemonade pull button.

That was the original idea that Jeremy wanted the plugin working group to address. The challenge with this is how do you handle various platforms and hardware seamlessly. As an example see the example in 13.4 of #3594.

[pwilkin]

I've looked at those earlier discussions, but my dominant impression from those is that they're dominated by AI-generated manifests, which are really hard to parse for me. I feel like LLMs are very bad at synthetic work like this because it requires an ability to weigh priorities which, to me, those documents seem to lack, so everything is bundled into one huge clump of fine-grained requirements.

The big benefit of containers is that they can easily bypass issues with system configuration and hardware issues basically boil down to driver support. So for me at least, you can short-circuit the entire standards discussion to a set of simple filters (supported hardware, supported modalities, supported models), agreeing on the API protocol (OpenAI-compatible with a Lemonade-defined extension for non-OpenAI outputs largely settles this, so you just need the extra API for status checks, can extend later) and the container behavior (such as "how does it access models"). I'm not saying this is trivial, but I don't believe it requires such a huge DSL as those suggested in the linked threads.

[abn]

so everything is bundled into one huge clump of fine-grained requirements.

That version is an accumulation of a month and a bit of work fielding a whole heap of requirements and changes from various folks. So yeah, it is a flushed out requirements, not the typical RFC because most of it was worked out prior to the current form of the RFCs. It is not an output of "hey AI please produce requirements" 😉.

The point I wanted to make was, if we go down a binary path we end up in a world were multiple cases need to be considered and handled declaratively. The original version of that proposal was in fact a container, with the weak contract that it provides OpenAI API - nothing more, nothing less. Where it gets interesting, is when you need to consider what devices get mounted, what files need to be mounted via model manager. The problem is that multiple engines do not conform to standard cli contracts either. Even if you take llama.cpp forks, some need additional options, some need extra checkpoints so on. IIRC dflash was one such example. On top of this, there is also how you determine what capabilities the engine provides/exposes even when it is OpenAI API compatible.

And it is not a DSL, it is only a schema to be precise. And I do not necessarily think this is more complex than what is already being used as recipes.

{
  "recipe": "dflash-rocm",
  "api_contract_version": "1",
  "capabilities": ["chat_completion", "completion"],
  "health_probe": {"type": "http", "endpoint": "/health", "timeout_seconds": 300, "poll_interval_ms": 500},
  "platforms": {
    "linux": {
      "rocm": {
        "command": "podman",
        "args": [
          "run", "--rm", "--name", "lemonade-{recipe}-{port}",
          "--device", "/dev/kfd", "--device", "/dev/dri",
          "--group-add", "video", "--group-add", "render",
          "-v", "{model_dir}:/models:ro",
          "--network", "host",
          "ghcr.io/luce-org/lucebox-hub:rocm-7.2@sha256:9f3a...c2",
          "/opt/lucebox-hub/server/build/dflash_server",
          "/models/{checkpoint_relative:main}",
          "--host", "{host}", "--port", "{port}"
        ],
        "stop_command": "podman",
        "stop_command_args": ["rm", "-f", "-t", "0", "lemonade-{recipe}-{port}"]
      }
    }
  }
}

All this said, and to be unambiguous, I am all for having a solution that just lets me drop-in a subset of containers that "behave" so that it can be used as a backends.

[neuhaus]

I support this proposal very strongly.

Please add https://github.com/gufo-org/gufo to the list (near the top). Leading performance and open source.

Also please ensure compatibility with distributed inference setup (tp=2 w/ RDMA)

[invzz]

this proposal would make my life much easier

[ramkrishna2910]

Can we document an exit plan for experimental engines? For example what gets an engine promoted, demoted or removed.

[jeremyfowers]

@Geramy has a PR open adding a guide for that, feel free to contribute there.

[jeremyfowers]

Onwards to implementation!