New Feature Proposal: Type constraints

[hellovai]

Stage

• Proposal
• Revew <---
• Approved
• Implementation in progress
• Merged

Validator / Type constraints

Problem

When defining a schema, often times, a type is insufficient to constrain it. Constraints like a regex match on a string, array length, contains, etc are all just as necessary. You may want to ensure an ID exists in your db before doing X or that a string is a substring of Y.

Currently, users have to do some post processing on the class to get validation errors in their native code, then they need to collect those errors and replumb it through the LLM to attempt to fix those errors. This problem can be amplified further when classes / enums have aliases and the raw string the user has is hard to connect back to the data model (for the user), to re-run through another prompt.

Solution

Some syntax in BAML that natively allows the user to apply JSON-schema-like constraints onto fields.

Types of constraints

Constraint	Types
length	array, map, string
regex match	string
eq, ne	all
gt, ge, lt, le	int, float, string
xor	int, float, bool
and, or	int, float, bool
custom function	all
contains	string, array, map
index operator []	array, map, string
min, max	int, float
unique	array	every item is unique (consider using set type?)
default	all	default value to fill if not found
reference	all	when a value references another value

As a general goal, we should attempt to keep the constraint system as flexible as possible, so its possible for users to add their own custom logic, and not be bottled on us shipping features. That means custom function is top priority in the design. Capabilities the user cannot trivially add are also high priority (regex, operators, etc).

Note: Network based constraints are considered out of scope as BAML has no plans to support a socket interface.

Prior work

JSON Schema

{
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer", "minimum": 0},
        "email": {"type": "string", "format": "email"},
        "is_member": {"type": "boolean"}
    },
    "required": ["name", "age", "email"]
}

• Reference fields are not supported
• Some primitive validators for strings are available:
  • date-time, time, date, duration
  • email, idn-email
  • hotename, idn-hostname
  • ipv4, ipv6
  • uuid, uri, uri-reference, iri, iri-reference
  • uri-template
  • json-pointer, relative-json-pointer (/foo/bar - used to point to a JSON object like jq)
  • regex - (a string that itself is a regex string)
• The official spec’s list of validators is: https://cswr.github.io/JsonSchema/spec/grammar/

PKL

class Bird {
  name: String(length >= 3)     
  parent: String(this != name)  //  notice it references another type
}

pigeon: Bird = new {
  name = "Pigeon"
  parent = "Pigeon Sr." 
}

class Project {
  local emailAddress = (str) -> str.matches(Regex(#".+@.+"#))
  // constrain the nullable type's element type
  type: String(contains("source"))?
  // constrain the map type and its key/value types
  contacts: Map<String(!isEmpty), String(emailAddress)>(length <= 5)
}

project: Project = new {
  type = "open-source"
  contacts = Map("Pigeon", "pigeon@example.com")
}

Note: With validators it may be very hard to understand how to navigate our docs. We would likely benefit from making a page per type, that outlines everything about it:

• what it is
• how to use it
• how to test with it
• how to constrain it

See that most things do this:

https://json-schema.org/understanding-json-schema/reference/string

Alternatively, like pkl we could do a single reference page which has everything in one place:

https://pkl-lang.org/main/current/language-reference/index.html#strings (cmd+f is much easier)

[recommended] Option 1: Attribute

We introduce a primitive attribute: @constraint()

This takes in a jinja expr as a parameter where the word this is used to denote that property.

keywords for expr:

this - denotes teh current type

super - references the object in the outer most block. In the case of non-class types, this would be nonsense and unstable.

The second parameter is optionally the error message to show when the constraint fails

Example:

int @constraint(this > 0 and this < 10) // this = int value

class Foo {
  bar int @constraint(this > 0 and this < 10) // this = Foo.bar value
}

Note that due to unions, its important to understand extactly where the @constraint attribute attaches.

class Foo {
  bar (int @constraint(this > 0 and this < 10)) // this = int
}

Note that the above who are different. in the first, the @constraint applies to the field Foo.bar, but in the second, @constraint applies to the int. In practice, they would do the same thing, but this may cause turmoil later.

Does it modify the prompt? No. If we want to support generic functions as constraints, we cannot intelligently modify the prompt. The user is expected to modify the prompt appropriately.

More complicated examples:

Referencing a field

We use super to refer to a different field in the class inside a @constraint.

class Foo {
  password string @constraint(this|length > 10)
  // Remember, second arg is the error message
  // super is used to access the properties of the class
  confirm_password string @constraint(this == super.password, "password_match")
}

// You are probably wondering why not Foo.password. Keep reading!

@@constraint is a block-level attribute that applies to the class, and you can refer to all the properties of the class using this

class Foo {
  password string @constraint(this|length > 10)
  confirm_password string
  
  // Block level as well! Remember, second arg is the error message
  @@constraint(this.confirm_password == this.password, "password_match")
}

The difference between these two is the error. The first raises an error with message password_match on class Foo . The second raises an error on Foo.confirm_password .

Referencing a field in a different class:

@ref is used to reference temporary data passed in from parent classes.

Example: Here we define a Deparment class that contains employees. The employee class will validate that the salary follows the department’s salary cap.

class Employee {
  // This is a temporary variable that will be provided elsewhere
  // max_salary won't exist in your python/ts bindings
  max_salary float @ref
  salary float @constraint(this < super.max_salary)
  ...
}

class Department {
  name string
  salary_cap float
  // Employee must now be instanciated with the max_salary parameter, which can come from the current Department class
  employees Employee(max_salary=this.salary_cap)[]
}

The syntax looks like you’re instantiating a class “Employee”, since that’s what we’re used to seeing in other languages. However, it really means that the type Employee has 1 reference passed in: max_salary — which comes from Department.

References bubble up until they are passed in.

What does this mean? If a child class has a @ref (Employee), the parent class (Department) must instantiate that child with that ref. If it doesn’t, then it means Deparment implicitly also has that same ref (max_salary).

We will add a third top-level parent called “Corporation” that passes in the ref:

// If you don't specify max_salary, then Department will have an inferred
// max_salary float @ref property
class Department {
  name string
  salary_cap float @constraint(this <= super.max_salary) // since max_salary is inferred from Employee
  employees Employee[]
}

class Corporation {
  // You can use static values for max_salary as well
  departments Department(max_salary=100000)[]
}

Passing references in functions

Sometimes we want to validate that the output matches a string in the input, like when asking an LLM to generate citations. Or you may just want to constrain the output type of a specific function in a certain way.

// If you don't specifiy max_salary here, you'll get an error as Department has an unbound @ref
function GetDepartment(exec_summary: string) -> Department(max_salary=100000)

You can also use inputs from function parameters

// use super, just like in a class, to access any parameters.
function GetDepartment(exec_summary: string, cap: float) -> Department(max_salary=super.cap)

Note: we highlight functions and types differently in BAML (not in this doc)

When validations fail…

Your BAML function will raise a BamlValidationError similar to what happens when parsing fails.

Validating fields without throwing exceptions

You can make any constraint error a warning that is available in your Python/TS code, by adding a third parameter:

@constraint(<expr>, <message>, error_type="warning")

This will have the following impact on your generated bindings:

// BAML
class Citation {
  quote string @constraint(
	  this in text, "exact_citation_not_found", error_type="warning"
  )
  idx int
  text string @ref
}

The parameter error_type="warning" in @constraint would automatically change the type of quote string to quote BamlValidated<string> in Python / TS, where BamlValidated is the following:

interface BamlValidated<T> {
  value T
  failed_constraints string[] // name of the constraints
}

// In TS:
class Citation {
  quote BamlValidated<string>
  ...
}
// In Python:
class BamlValidated[T](BaseModel):
  value T
  failed_constraints string[]

class Citation(BaseModel):
   quote BamlValidated[string]

Another example (RAG with citations)

class Citation {
  quote string
  idx int
}

class Answer {
  sentence string
  chosen_citations (int @constraint(this in super.citations))[] @description("idx of the citation, if relevant")

  // Not injected into the prompt or a member of the class
  // However, in order to use the Answer type as an output, citation must be set)
  citations int[] @ref
  // alt:
  @@constraint_ref(citations int[])
}

class Response {
	citations Citation[]
	answer Answer(citations=super.citations|map(attribute='idx')[]
	// This would error out, since the super.citation field is a naming conflict,
	// and must be explicitly set.
	answer Answer[]
	// Acceptable you could isntead do if you wanted to forward a paramater in from somewhere else
	answer Answer(citations=super.items)
	items int[] @ref
}

class Citation {
  quote string @constraint(this in text, "exact_citation_not_found")
  idx int
  text string @ref
}

class Response {
	// Since we don't specify the constraint, Response infers that it also has a text string constraint from Citation
	citations Citation[]
	answer Answer(citations=super.citations|map(attribute='idx'>)[]
}

class Answer {
  sentence string
  citation (int @constraint(this in super.citations))[] @description("idx of the citation, if relevant")
  citations int[] @ref
}

// Usage in a function

// This errors out, as now that response is being used it must specify all of its constraints
function AnswerQuestion(question: string, chunks: string[]) -> Response

// Here super means the function and gives you access to all its inputs.
function AnswerQuestion(question: string, chunks: string[]) -> Response(text=super.chunks|join("\n"))

To summarize:

1. @constraint is an attribute that can be attached to any type or class property (not enum value) that has a couple of parameters, block level @@constraint
2. @ref can be added to a class as an invisible, baml-only property that doesn’t actually get shared anywhere but baml. it is purely a ref to some other value.
3. @ref bubbles up, but if theres ever a naming conflict, must be explicitly set
4. The the point of a function output, all @refs must be resolved

More complex examples

What does it look like with many validations? Is it readable? Can I understand it?

class Address {
    street string @constraint(this|length > 0)
    city string @constraint(this|length > 0)
    country string @constraint(this in ["USA", "Canada", "UK"])
    postal_code string @constraint(
        (this|length == 5 and super.country == "USA") or
        (this|length == 6 and super.country == "Canada") or
        (this|matches("^[A-Z]{1,2}[0-9][A-Z0-9]? [0-9][ABD-HJLNP-UW-Z]{2}$") and super.country == "UK"),
        "Invalid postal code for the specified country"
    )
}

class Person {
    name string @constraint(this|length >= 2)
    age int @constraint(this >= 0)
    email string @constraint(this|matches("^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"))
    address Address
    
    @@constraint(
        this.age >= 18 or this.address.country != "USA",
        "Person must be 18 or older to have a USA address"
    )
}

class Book {
    title string @constraint(this|length > 0)
    author string @constraint(this|length > 0)
    isbn string @constraint(
        CustomValidator.validate_isbn(this),
        "Invalid ISBN format"
    )
    publication_year int @constraint(1000 <= this and this <= 2100)
    genres string[] @constraint(1 <= this|length and this|length <= 5)
    
    @@constraint(
        this.publication_year <= datetime.now().year,
        "Publication year cannot be in the future"
    )
}

class Library {
    name string
    books Book[] @constraint(this|length > 0)
    
    total_books int @constraint(this == super.books|length)
    
    @@constraint(
        this.books|map(attribute='isbn')|unique()|length == this.books|length,
        "All books must have unique ISBNs"
    )
}

Employee / department “interdependent classes with reference passing”

class Department {
    name string @constraint(this|length > 0)
    budget float @constraint(this > 0)
    employees Employee(department=this)[]
    
    @@constraint(
        this.budget >= this.employees|sum(attribute='salary'),
        "Department budget must cover all employee salaries"
    )
    
    // This is NOT yet a BAML feature, but adding in to just visualize
    // what BAML looks like with more capabilities for reability
    cost float @compute(this.employees|map('salary')|sum)
}

class Employee {
    name string @constraint(this|length > 0)
    salary float @constraint(this > 0)
    department Department @ref
    
    @@constraint(
        this.salary <= this.department.budget,
        "Employee salary cannot exceed department budget"
    )
}

class Project {
    name string @constraint(this|length > 0)
    budget float @constraint(this > 0)
    department Department @ref
    team_members Employee(department=this.department)[]
    
    @@constraint(
        this.budget <= this.department.budget,
        "Project budget cannot exceed department budget"
    )
}

class Company {
    name string @constraint(this|length > 0)
    departments Department[]
    employees Employee(department=super.departments)[]
    projects Project(department=super.departments, team_members=super.employees)[]
    
    @@constraint(
        this.employees|all(emp => emp in this.departments|flatmap(attribute='employees')),
        "All employees must belong to a department"
    )
}

Disambiguating ‘this’

One of the current issues with the syntax is that “this” refers to different things in different contexts.

“this” in a constraint, maps to value of a field itself.

class Myclass {
	property string @constraint(this|length > 0) // this maps to "MyClass.property"
	property2 (string @constraint(this|length > 0)) // this maps to "string")
}

using “this” when passing references to a type maps to the whole class

class Department {
    name string
    employees Employee(department=this)[] // "this" maps to Department. super is not valid here.
}

What if we support this inside a new hypotethical class property that’s computed at runtime using @compute? this is also referring to Myclass

class Myclass {
  name string
	prop2 string @compute(this.name) // "this" = Myclass
}

We’re ok with the current approach — IDE / extension can help disambiguate what this means, and there are a limited number of contexts.

Should we require block strings around evaluatable expressions?

Or just disambiguate them automatically as only available inside of @constraint? The only issue is that now in @alias we don’t support this.

The best solution may be:

• Rollback global support for unquoted strings . They are only explicitly supported in certain locations.

    salary float @constraint(this > 0) @alias(this.name)

Dynamic Classes

• In TypeBuilder we’d expose a similar .constraint method eventually, but this is lower priority

Questions

• Is this slow?

  • No, its all done in rust, no-ffi boundary

• How do we do regex matching?

  • We need to add a new custom filter to jinja (matches usage e.g. my_var|matches(#"^[\w]+$"#) )
    • Note that /^[\w]+$/ is out of scope as we would need to add a new string

• How do i add multiple constraints

  int @constraint(..) @constraint(..)
  // Or use and / or keywords + operators
  int @constraint(this > a and this < b)

• Can I validate UUIDs?

  • there’s no native one (but you can write a regex). We could create a shorthand like @isUuid

• How could we support custom hooks to do something like a DB check?

  • Maybe we take in a lambda from the user?

• Can users modify the data in the field?

  • No

• Do we do retries or anything clever on schema failures?

  • No, we raise BamlValidationError

• What happens when the expression doesn’t resolve to a boolean?

  • All types can be cast to bool trivially in jinja, but we should show a warning to inform the user

• When does the validator run?

  • Don’t use super:
    • During parsing
    • During streaming
  • Use super:
    • After parsing
    • ONLY after streaming is fully complete
  • Running a validator may modify your data!

• Can users just add their own pydantic validations as well?

  • No, they need to create a new class that inherits from the BAML class, then add their validators on that class.

• Can i do non-1-liners?

  • No, jinja only supports one-liners sadly. We need to advance JINJA to do this with multi-lines

• Can my one-liner be formatted across multiple lines?

  • yes

• Can you transform these validations into zod/pydantic validations?

  • No, they live in BAML. its too hard 😢

Short-hand

Later we can introduce shorthand validators but they are just all syntactical sugar which makes implementing them much easier.

int @between(1, 10) // @constraint(1 < this and this < 10)

class Foo {
  items string[]
  best string @in(super.items) // @constraint(this in super.items)
}

This is out-of-scope because we should implement @constraint first.

Appendix A: What we learned from our citations RAG demo:

https://baml-examples.vercel.app/examples/rag

(and how this validation would work to make it easier)

1. citation sometimes didn’t match source text because LLM
   1. fixed a spelling mistake
   2. uppercasd a word spacex → SpaceX
   3. removed the “[30]” from the actual text (very hard to get it to INCLUDE them with prompt engineering)

What we added to help?

1. error_type="warning"
2. @ref

In the future we can add a custom filter closest_substring_match which does:

// Returns a span in the target that is the closest match to source + the levinshtein's distance
// Ideally, we'd also like to be able to modify the source itself if it matches some minimum threshhold on score
fn closest_substring_match(source: &str, target: &str) -> (&str, int)

class Summary {
  text string @ref
  quote string @constraint(this in super.text)
}

// instead
class Summary {
  text string @ref
  // @no_binding won't generated
  quote_inner string @no_binding
  transformed_quote (string, int) @compute(this.quote_inner|closest_substring_match(this.text)) @no_binding
  quote string @compute(this.transformed_quote.0) @constraint(super.transformed_quote.1 / super.text|length < 0.01)
}

constraint(closest_sustring_match(...))

// shorthand
@contains_closest_match($Citation.eoijsf)

Appendix B: Alternative @ref ideas

One option is to refer to other properties from other classes using some special syntax inside a constraint, instead of having to define @ref’s explicitly and passing them down.

class Book {
  name string
  category string @constraint(this in $Shelf.types) // assume category must match shelf name
}

Here’s all the issues:

• Ambiguous interpretation: How would we indicate Shelf relates to the Book type? What if there are multiple Shelf types in the parent object?
• Limited reusability: If we wanted to use the Book class in a different context (e.g., an online bookstore), the constraint would still be tied to the Shelf class, even if shelves aren't relevant in that context.
• Implementation: The graph gets very complicated to validate the existence of types as we now need to update jinja with a new special token $

option 2:

class Book {
  name string
  shelf string @ref	
}

class Shelves {
  // We could consider inferring `shelf` for Book here
  books Book[] @constraint(
  shelf string @description("name of the shelf")
}

option 3

class Answer {
	some_text string[]
	citation Citation
}

class AnswerVersion2
{
	some_text string[]
	citation (Citation @passRef(some_text=super.some_text|join("")))
}

class Citation {
	 some_text string @ref
	 sentence string @constraint(this in super.some_text)  
}

function MyFunc(input string) -> Result @passRef(some_text=input)

option 4

class Answer {
	some_text string[]
	citation Citation
}

class AnswerVersion2
{
	some_text Blah{ }[]
	citation Citation { some_text }
} 

class Citation {
	 some_text string @ref
	 sentence string @constraint(this in super.some_text)  
}

Appendix C: Integration with Playground / UX / additional tooling

• We helped people visualize prompt, run tests with VSCode, how can our tooling help with this new language feature (if at all)?
  • Test outputs can now include
    • parsing issues
    • failed validations
  • We could potentially have a UI to test your validations (e.g. write out your schemas, and we will give you a “pass” or “fail”.)
    • Tests cases for validating schemas?

Option 2: PKL-like syntax (rejected)

We likely can’t do this as BAML doesn’t support powerful expressions PKL-lang does as a top level ast-node. Once we add ast support for top level expression nodes, we’d be able to consider this, but as of now, its out of scope due to technical reasons. Additionally, we must support jinja as our primary expression resolver (foo + bar , as otherwise we’d have to write that from scratch).

e.g.

local var = "..." // this is not something BAML can do

[sxlijin]

We should distinguish between hermetic user-defined functions and non-hermetic ones: we use validating IDs against a DB as an example, but the proposal doesn't add support for this.

Other issues I see:

• validation is now intrinsically coupled to parsing
  • what should a user do when they encounter a ValidationError?
    • e.g. "received person.salary = 120_000 but person.salary has constraint <= 100_000 - LLM response doesn't make sense
    • should the user be able to do something useful with the fields that passed validation?
• as described, validation decisions can roll up into union discriminant selection
  • has the above impact

[aaronvg]

the validation error should probably include the parsed payload as well either way.

[aaronvg]

Will a stream be cancelled if there's a validation error? Or do we fulfill it to completion?

[aaronvg]

Needs a section on reusing constraints -- is there some way in the future we could use variables? Or a type alias with a constraint?

[anish-palakurthi]

Use the keyword "scope" or "block" instead of super, to avoid misleading users to inheritance concepts.

[sxlijin]

class Foo {
  field1 string @constraint(..., error="warning")
}

in ts now, JSON.stringify will produce

{
  field1: {
    value: "field1-value"
  }
}

[aaronvg]

Needs more details on what a validation error looks like -- for BAMLErrors we expose have the raw llm response. Are we exposing the same thing + the parsed object?

[aaronvg]

Also a more detailed section on why we are coupling parsing with validation