RFC: Typed decision models in Instructor

[jxnl-oai]

I'd like to add decision model support to Instructor, starting with Jev via TypeSafe and OpenRouter. Define questions in a Pydantic model, evaluate them against shared context, and get typed answers and probabilities.

This is a proposal; the API and results below are illustrative.

Content moderation example

from typing import Annotated, Literal

import instructor
from instructor.decisions import Choice, Choices, Level, Noul, Score
from pydantic import BaseModel, Field


class Category(Choices):
    """Which policy category best describes the post?"""

    HARASSMENT = Choice(
        description="Abusive content directed at a person",
        examples=["You're worthless and nobody wants you here."],
    )
    HATE = Choice(
        description="Content attacking people for a protected characteristic",
        examples=["People of that religion are inferior."],
    )
    NONE = Choice(
        description="Neither category applies",
        examples=["I disagree with your argument."],
    )


class ModerationDecision(BaseModel):
    category: Category

    policy_violation: Annotated[
        float,
        Noul(
            instructions="Does the post violate the supplied content policy?",
            when_true="The post attacks or harasses a person or group.",
            when_false="The post expresses disagreement without an attack.",
            examples=[
                ("You're worthless and nobody wants you here.", True),
                ("I disagree with your argument.", False),
            ],
        ),
    ]

    severity: Annotated[
        float,
        Score(
            instructions="How severe is the content?",
            scale=(0, 10),
            levels=[
                Level("No abuse", examples=["I disagree with you."]),
                Level("Insult or harassment", examples=["You're worthless."]),
                Level("Explicit threat", examples=["I'm going to hurt you."]),
            ],
        ),
    ]

    format: Literal["text", "image_caption", "other"] = Field(
        description="What kind of content is being reviewed?"
    )

    target: (
        Annotated[
            Literal["individual"],
            Choice(description="Directed at an individual"),
        ]
        | Annotated[
            Literal["group"],
            Choice(description="Directed at a group of people"),
        ]
        | Annotated[
            Literal["none"],
            Choice(description="No person or group is targeted"),
        ]
    ) = Field(description="Who is the post directed at?")


# Set OPENROUTER_API_KEY in your environment.
client = instructor.from_provider(
    "openrouter/typesafe/jev-1.13",
    mode=instructor.Mode.DECISIONS,
)

decision, raw = client.create_with_completion(
    response_model=ModerationDecision,
    context={
        "post": {
            "text": "You're worthless and nobody wants you here.",
            "format": "text",
        },
        "policy": ["No harassment", "No attacks on protected groups"],
    },
)

# Illustrative results, not a live model response.
print(decision.category)          # Category.HARASSMENT
print(decision.policy_violation)  # 0.84, probability of a violation
print(decision.severity)          # 7.0, on the requested 0–10 scale
print(decision.format)            # "text"
print(decision.target)            # "individual"

if decision.policy_violation >= 0.80:
    print("Send to human review")

# Proposed access to the provider's response.
print(raw["answers"]["category"]["probabilities"])
print(raw["answers"]["category"]["confidence"])
print(raw["answers"]["severity"]["score"])  # e.g. 1.4 on Jev's 0–2 scale
print(raw["answers"]["severity"]["probabilities"])
print(raw["answers"]["severity"]["legend"])

All five questions use the same context. create_with_completion returns the validated model and provider response; the shape of raw remains open. The review threshold is illustrative.

Direct TypeSafe access would use the same model with instructor.from_provider("typesafe/jev-latest", mode=instructor.Mode.DECISIONS) and a TypeSafe API key.

Design details

Choice definitions

Support four ways to define a Choice:

• Literal["safe", "unsafe"] for simple options.
• A string-valued Enum for reusable options.
• Choices with Choice(description=..., examples=...) for reusable criteria.
• A union of Annotated[Literal[...], Choice(...)] for inline criteria (Union[...] on Python 3.9).

Enum members default to lowercase values; Choice(value=...) overrides them. Results preserve the declared enum or Literal type. Instructions come from Question(instructions=...), then Field(description=...), then the Choices docstring. Missing instructions fail before the request: a field name is only an identifier.

Noul and Score

Noul returns the probability of true between 0 and 1, without a separate confidence value. when_true and when_false describe the boundary; examples pair input with an expected boolean.

Score levels are ordered descriptions. Jev returns a probability-weighted position across their zero-based indexes. Three levels yield 0–2 by default; scale=(0, 10) maps them to 0, 5, and 10:

scaled = lower + native_score / (number_of_levels - 1) * (upper - lower)

The raw score, probabilities, and legend remain unchanged. A scaled 7 is an intermediate weighted value, not a separately defined level. TypeSafe recommends at least two levels and accepts up to ten, so 0–10 cannot have eleven distinct native levels. Score examples use zero-based level indexes even with a scale. Bounds must be finite and increasing; Pydantic validates the scaled value.

State, structure, and examples

context is a dictionary sent in full as provider state and passed to Pydantic validators. Put strings or lists under a key; omit validator-only data that should not be sent. Instructions may use Jinja templates; context values are state, not templates.

Instructions, Choice descriptions, Noul meanings, and Score levels accept JSON-compatible strings, objects, or arrays. Question carries Choice instructions; Choice, Noul, and Level carry criteria. Choice descriptions may also be None. Alternatively, Noul(criteria={"true": ..., "false": ...}) could accept the native shape, mutually exclusive with when_true/when_false.

Examples are an Instructor convenience, not a native Jev field. Instructor would convert inputs with str(input) and group them by option, boolean, or level as { "meaning": ..., "examples": [...] }. Structured meanings stay intact. Level and Score (input, level_index) examples can be combined; indexed examples follow level examples in order. Without examples, descriptions pass through unchanged.

Validation and results

Reject unsupported types, duplicate options, missing instructions, and invalid examples before the request. TypeSafe documents up to 255 Choice options and 10 Score levels. Missing or malformed answers fail validation; Jev does not generate explanations.

Expose Choice and Score distributions and confidence. Choice confidence summarizes the distribution, not the winning option’s probability. Different distributions can yield the same Score, so applications may need both the distribution and parsed float.

Questions for discussion

1. Are Choice, Choices, Question, Noul, Score, and Level the right names, and should they live under instructor.decisions?
2. Are inline Literal unions useful alongside Choices? Where should descriptions and instructions live when both enum and field metadata are present?

References: TypeSafe API reference, TypeSafe structured questions, TypeSafe Score, and OpenRouter's Jev guide.

[IRONICBo]

Inline unions look useful for one-off options, while Choices should stay the reusable path; field metadata should own question instructions and option annotations should own option descriptions. Choice, Choices, Score, and Level read clearly under instructor.decisions, but Noul may need a discoverable alias; from maintaining Jev Social, I'd also expose probabilities and confidence in a typed sidecar so uncertainty cannot be mistaken for validation failure.