RFC: Support for Open Models in Coop and Osprey

[julietshen]

Context

Part of ROOST's mission is to enable pluralist online safety where each organization that's using our tools gets to decide what is harmful and how to take action on it. Coop and Osprey are both designed to be fully configurable and customized, with no prepopulated rules or policies; the ROOST model community curates detection models that allow organizations to apply powerful AI to assist with classification and automation.

Our vision is to support open models (including, but not limited to the ones in the ROOST Model Community) in Coop and Osprey. Today both tools plug into hosted APIs cleanly, but every open model still needs custom integration code, which puts sovereign and non-English-capable models out of reach for most adopters. This request for comment is a PRD on that, proposing a third project that provides infrastructural support for both vs building it twice in different places. Three questions we would most like feedback on:

• Should this be a third project, or would you rather see this built into each tool?
• Should the modelfile be a portable artifact people can share, with the UI as the primary way to create one, or should configuration live entirely in the tool?
• The proposal assumes the main thing blocking adopters from open models is per-model integration friction. If the real constraint for you is inference infrastructure, cost, or expertise, we want to know.

Thank you @juanmrad whose initial ideas grew into this proposal and to @taobojlen for his early feedback and thoughts

PRD: Open Models and ROOST (Pigeon)

TLDR;

In order to support open weight AI models in Coop and Osprey, a new project is needed to provide the integration infrastructure. Pigeon lets platforms add open source AI models to Coop and Osprey as easily as adding an API-based integration. Both tools can connect to safety models today, but only API services plug in cleanly. Open models, which platforms can run themselves and tailor to their language and community, still require custom code for each one. Pigeon replaces custom code with a small description file, so adding a new open model is a configuration step vs a software project.

North Star

An adopter goes to Hugging Face, finds an open model that fits their language and norms, registers it in Pigeon, and enables it for their org. A new signal or callable model function is then available in Coop or Osprey.

Background

Safety classification runs disproportionately on a small number of dominant, hosted APIs (OpenAI Moderation, Hive) and a handful of English-prioritized classifiers. These tools work well for the language and speech norms they were trained on, and noticeably less well outside them. Adopters serving local communities, operating across multiple languages, or working under regulatory regimes that require local data and model control are stuck with tools that don't fit their users.

Coop and Osprey are the open source layer that should make those models usable. Today they can't. Coop and Osprey have plugin support for hosted APIs, but lack the open-model modelfiles that make open source AI plug and play. The result is that adopters who want plural, sovereign, non-English-capable safety models can't reach them through ROOST tooling today.

Pigeon’s purpose is to close that gap while remaining an optional module that lets ROOST adopters use open models (classifiers and general purpose models) inside Coop and Osprey with the same effort as adding a hosted API today. Pigeon is one of two AI-related ROOST projects in scope for 2026. It is the narrow slice that delivers integration value first; broader agentic work depends more deeply on the Data Abstraction Layer (DAL) and the Safety Taxonomy.

Why (’User problem’)

Target Users

Primary: the engineer at a small-but-mighty organization. They've decided they want a specific open model and want it wired into their Coop or Osprey signal pipeline without writing substantial custom code.

Secondary: a RMC model developer who wants their model to have a low-friction path to ROOST interoperability through a Pigeon modelfile.

Goals

1. Adopters get stable, versioned moderation signals or model functions that can be safely consumed by rules engines in Coop or Osprey
2. A Coop or Osprey adopter can plug in an open model with a config change, not by writing a new integration package.
3. Adding support for a new open model is a modelfile task, not a code change in Coop, Osprey, or Pigeon itself.
4. RMC-published models have a documented path to becoming a Pigeon modelfile.

Non-Goals

1. A general-purpose inference gateway for users who aren't using Coop or Osprey. (General-purpose models are in scope; serving users who are not on Coop or Osprey is not.)
2. A new LLM proxy. Wherever existing OSS proxies solve sub-problems, Pigeon will wrap them.
3. Model hosting
4. Model evaluation or benchmarking. Handled by the separate eval workstream.
5. Fine-tuning, training, or labeling tools.
6. Multi-step agent reasoning.
7. Non-text modalities (audio, image, video). v1 is text-only. The modelfile spec should be designed so it doesn't preclude extending to audio classifiers like the Roblox voice safety model joining the RMC, but actual multimodal support will be handled in v2.

What ('How to solve')

MVP

Pigeon would be a new repository containing three things:

1. The versioned modelfile YAML spec that documents what a modelfile declares and how it gets loaded.
2. An Osprey Python package that has a plugin that exposes each enabled modelfile as a callable function in Osprey rules, plus optional registration as Osprey's LLM provider.
3. A Coop npm package that exposes each enabled model as one or more signals in Coop's rule builder, via Coop's existing integration model.
4. A light UI and management API for registering, validating, enabling, and versioning modelfiles per org, without a redeploy.

A modelfile is the YAML file that tells Pigeon how to use a particular model. It names the model, the inference runtime it expects (Ollama, vLLM, Hugging Face Inference, a self-hosted endpoint), the prompt to send, how to parse what comes back, the labels it produces, and the languages it's validated for. Adding a new model to Pigeon means writing a YAML file, not releasing software. For classifier and BYOP models, the modelfile declares the labels and the parser. For general purpose models, it declares the output instead as free text or a schema the parser validates.

The Osprey and Coop packages read the same modelfiles and once a modelfile is authored, it works in either tool. Modelfiles also declare a format that tells Pigeon how to package the input for the model. Common cases are chat (the runtime applies the model's tokenizer template, covering CoPE-B / Llama-3 / etc), chat-harmony (Pigeon handles the Harmony framing gpt-oss-safeguard expects), and classifier (raw input in, scored labels out). New formats can be added as named entries in the spec.

v2

Modelfiles take content as their primary input and support an optional context field for enrichment values like account age or channel history, which LLM-style prompts can reference. The DAL provides additional context so that classifications can reason about the same enriched events Coop's review interface displays, without breaking modelfiles that only use the content field. A modelfile likewise declares the labels it emits and, optionally, how those map onto a canonical safety taxonomy. Rules written against taxonomy labels become portable across any modelfile that declares the mapping. This allows adopters to swap one model for another without changing rules or downstream review or investigation behavior.

Decisions about what to do with a classification (the violative / needs review / needs classification framework on the Taxonomy roadmap) are made by rules, not by Pigeon. Pigeon emits classifications or completions; rules combine them with other signals into dispositions. As the Taxonomy work matures, this is also where the longer-term answer to cross-model confidence comparability lives.

V2 introduces a visual block-programming style of mapping a model’s output to a safety taxonomy. It applies to fixed-label classifiers only (completion-mode and BYOP models have no fixed label set to map from).

How should it work (’Requirements’)

Osprey

Osprey rules call modelfiles the same way they call any other rule function, and the call shape depends on the model type. Fixed-taxonomy classifiers return a label and score directly. BYOP models (gpt-oss-safeguard, Zentropi's CoPE-B) additionally accept a policy argument. General-purpose models run in completion mode: the rule passes a prompt and gets back the model's completion, free text or fields the parser extracts into individual features, for agentic or assistive steps like drafting a triage note or extracting fields from a report. In all three cases, adding a new modelfile changes the YAML, not Osprey code.

A general-purpose modelfile can also be registered as Osprey's LLM provider for AI-assisted features such as natural-language query building. This is the same completion-mode model, surfaced two ways: as a callable function in rules, and as the deployment's assistive provider. Registering a provider is opt-in via config, because Osprey allows only one LLM provider per deployment, and an adopter who already has one should not be forced to swap.

Coop

In Coop, Pigeon installs as one integration. Each enabled modelfile becomes one or more signals in the rule builder, configured per-org through Coop's existing integrations interface. The adopter installs one package and gets every modelfile they enable, instead of installing a separate package per model.

Adding a new Hugging Face model is a modelfile-authoring task. The Pigeon CLI scaffolds a draft from a model URL by reading the model card and walking the author through prompt and parser choices. The author registers the modelfile through the Pigeon API or UI. Once it is validated, it shows up in the signal library in Coop.

Risk Assessment and Mitigation

• Non-standard formats are ongoing maintenance investments. Harmony, and any future custom format, is code that Pigeon would need to maintain. Each one needs tests, updates when the upstream format evolves, and clear documentation for adopters.
• Once modelfiles ship, the schema is hard to change.
• Each Coop signal can only carry one value: a yes/no, or a single number. A model that scores content across multiple categories at once (like "80% harassment, 30% spam, 10% threat") has to be split into one separate signal per category, because Coop has no way today to hold them all together in a single signal.
• Open safety models still have the same English bias as closed APIs.
• Supply chain risk for open models - grabbing a model off HF can mean pulling arbitrary downloaded weights, models that require trust_remote_code to run, and model cards containing prompt-injection or malicious instructions that the CLI scaffolder reads and acts on while authoring a modelfile.
  • Pigeon does not vet or curate models as adopters own the decision to run a given model. Prioritize RMC-vetted models and the CLI treats model card content as untrusted text to display, never as instructions to act on, and surfaces when a model requires arbitrary code execution to load so the adopter is choosing it knowingly.

Success Metrics and KPIs

• Number of Coop and Osprey deployments running an open model via Pigeon within 6 months of launch.
• Median time from "I want to try this HuggingFace model" to "the model is producing signals in my pipeline."

Open Questions

• Granting that open runtimes have largely converged on a common chat API (Ollama, vLLM, LM Studio, Hugging Face TGI), do we target that interface directly, or wrap a proxy like LiteLLM?
  • Wrapping LiteLLM gets us a DB-backed config and an admin UI that applies changes without restarts, a guardrail registry browsable by use case, and a compliance playground that tests a policy against traffic and reports precision, recall, and false-positive rate. That would cover provider abstraction, the management UI we want, and much of completion mode. The tradeoff is a heavy external dependency.
• Which named formats belong in the MVP?
• One rule function vs. one per model in Osprey coverage.

Glossary of Concepts

Example YAML

name: example-classifier
version: 1
model:
  id: org/example-safety-classifier
  runtime: vllm            # ollama | vllm | hf-inference | endpoint
mode: signal               # signal | completion
format: chat               # chat | chat-harmony | classifier
policy_argument: true      # BYOP: the rule supplies the policy at call time
prompt:
  user: "{{policy}}\n\n---\n\n{{content}}"
parser:
  type: json
  labels_path: $.labels
  score_path: $.confidence
labels: [violating, borderline, non_violating]
languages_validated: [en]

• Open model: an AI model released with the neural network's numerical parameters (the "weights") so anyone can download and run it locally.
• General purpose model: an open model not trained as a fixed-taxonomy classifier, used for agentic or assistive tasks
• Moderation signal: single normalized output from a classifier that Coop or Osprey can act on, carrying a label or category and a score.
• Completion: the free-form or structured output a general-purpose model returns when called, as opposed to a fixed safety label
• Taxonomy mapping: the optional declaration in a modelfile that connects a model's own labels to a shared canonical safety taxonomy.
• Signal mode: the way Pigeon runs a classifier or BYOP safety model, normalizing its output into a versioned moderation signal.
• Completion mode: the way Pigeon runs a general-purpose open model, serving it behind a stable interface that Coop and Osprey rules and agentic workflows call. The model returns a completion (free text or fields the parser extracts into individual features) vs a normalized signal.
• Versioned signal: a moderation signal whose meaning is pinned to a specific combination of model version, modelfile version, prompt version, parser version, and taxonomy-mapping version. Because the version is composite, any change to those parts produces a new version rather than shifting what an existing signal means.
• Modelfile: A Modelfile is the blueprint to create and share customized models. Here, it means a YAML file that tells Pigeon how to call a specific model (which inference runtime to use, what prompt to send, how to parse the response, its output, and which languages it's validated for)
• YAML (YAML Ain't Markup Language): is a human-readable data serialization language widely used for configuration files and data exchange.
• Inference: the execution phase where a trained model applies its learned patterns to new, unseen data to generate predictions, make decisions, or create content.
• Hosting: the process of storing, running, and managing models on a server (or local machine) so that users and applications can interact with them via an API or user interface.
• BYOP: Bring Your Own Policy style models where the model’s classification uses a user-provided policy prompt.

[ThatKoffe]

This would be great, however it'd be nice as an optional dependency, say a plugin system. Where you can import models through docker or something similar, to avoid bloating the project.

[verdverm]

I believe this is essentially what the modelfile is, it's an ollama thing I have not come across elsewhere. One really only needs a bit of config, ideally user supplied so it can be customized (i.e. the example yaml near the end of the RFC)

[verdverm]

For Open Question 1, I would target the OpenAI API. This allows users to choose direct model calls or the gateway of their choice (which all implement the same OpenAI API). You'll likely want to cover the Anthropic API endpoints too, something gateways are increasingly doing as well.

An interesting feature in my gateway of choice is "virtual models" (https://gomodel.enterpilot.io/docs/features/virtual-models). This enables me to swap out models or partition traffic without needing to reconfigure my tools.

You might also look at something like https://adk.dev which is pure SDK and handles the normalization of provider responses for you. Any multi-provider agentic SDK should do this, at least any worth the investment of your time. Out source that part for sure, because it is an endless game of edge case whack-a-mole.

I am curious about the pathing syntax in

parser:
  type: json
  labels_path: $.labels
  score_path: $.confidence

I would pick something already in the wild like CEL, something familiar to the people who will be writing this yaml / logic (those words don't belong together imo 🤮 but I won't belabor the point or CUE :)

[taobojlen]

An adopter goes to Hugging Face, finds an open model that fits their language and norms, registers it in Pigeon, and enables it for their org. A new signal or callable model function is then available in Coop or Osprey.

I'd like to see more detail in this document on how the model is actually deployed. Is Pigeon responsible for provisioning inference or for talking to already-provisioned inference? E.g. does Pigeon run on a box with a GPU? Or is Pigeon a control plane that can call out to other inference backends (Modal, Fireworks.ai, self-hosted GPUs)? If so, what does the API between Pigeon <> self-hosted GPUs look like, what needs to run on the GPU boxes, etc.

My 2 cents: we'd probably want a control plane that can call out to multiple deployment backends. But for self-hosted GPUs I'd like to see more about how that'd work.

AFAIK running models directly on a GPU can be somewhat of a faff, especially when containerized, since it's dependent on the details of the model + GPU etc. -- I am not sure whether that's something we can feasibly abstract away fully. (maybe!)

Modelfiles take content as their primary input and support an optional context field for enrichment values like account age or channel history, which LLM-style prompts can reference.

To me, this is leaking consumer-specific details into the inference API. For LLMs I'd lean towards a minimal contract (perhaps a subset of the OpenAI API as @verdverm suggests).

Also, I would move away from the "modelfile" term since that's an existing thing -- and not the best fit for this, IMO. I'd call it something like a "model manifest" instead.

chat-harmony

I had to look up what Harmony was, it's OpenAI's format for gpt-oss: https://developers.openai.com/cookbook/articles/openai-harmony

Is there a risk that we end up with many different formats here to support open models with slightly different APIs? chat-harmony seems to describe a particular model's wire format whereas classifier describes the kind of inference. In other words, does the format key describe what format the model expects or the API that Pigeon exposes?

policy_argument

I think this is an unnecessary key in the model manifest. You can just choose not to put {{policy}} in the prompt. But see my point below, maybe the prompt doesn't need to live in the model manifest.

Some other thoughts:

• AFAIK, Coop and Osprey do not currently have any support for LLMs, right? So I'd suggest we start by focusing on the classifier use case.
• I'd like to see an example of how it'd look to call an LLM with structured outputs. That can be a useful way to use an LLM as a classifier!
• I wonder if the model manifest is the right place to put the prompt. The reason is that the prompt can have a large impact on quality, and so ideally you want it managed alongside robust analytics/QA features -- whereas the manifest seems relatively static. For example, a Coop user should be able to run 2 versions of a prompt concurrently and then compare performance. It seems like it could be a better UX if that prompt is then managed in the same place as the analytics.