Repository navigation
Files for models
An agent opts in to files for its model: it names the sources of the files (its input fields, its tools'
results, the thread's history) and the views its model gets of each file (the original, a redacted copy,
a transcription). The framework gives safe defaults and the developer decides — an agent without files
sends no byte to its model. Decision: ADR-0036.
import {
Agent,
AnswerFiles,
FileSource,
FileView,
ModelFiles,
} from "@agboss/core";
import { megabytes } from "@agboss/core/units";
@Agent({
name: "invoice-reader-agent",
description: "Reads an invoice and its items",
prompt: "Read the invoice and its items.",
model: "google/gemini-2.5-flash-lite",
files: ModelFiles.of({
sources: [FileSource.Input, FileSource.ToolResults, FileSource.History],
views: [FileView.Original, FileView.Transcription],
maxFileSize: megabytes(5),
maxTotalFileSize: megabytes(10),
maxFiles: 5,
allowedMediaTypes: ["application/pdf", "image/png", "text/csv", "audio/mpeg", "video/mp4"],
answerFiles: AnswerFiles.SeenInVisit,
guardToolResultFiles: true,
}),
})
export class InvoiceReaderAgent {}| Source | Files | Needs |
|---|---|---|
FileSource.Input (default) |
the A2AFileDto fields of the agent's input |
nothing |
FileSource.ToolResults |
the A2AFileDto fields of a tool's output; the tool message stays text, a user message with the parts follows |
nothing |
FileSource.History |
the files of the turns the agent remembers: a turn keeps a descriptor, never bytes | an A2AFileRepository that keeps files ([files.history-needs-store]) |
A history file is read through the guarded path as a copy under the execution that reads it: only a file of a
root execution of the same tenant and thread is read, never another thread's or tenant's. A file that
retention removed is shown as its descriptor, (removed). The copy is an ordinary file of this execution, so
an agent can hand it back in its answer.
| View | What the model reads |
|---|---|
FileView.Original |
the file as a content part (an image, a PDF, audio, video) or as text (text/*, application/json) |
FileView.Redacted |
a copy with every PII kind the agent's policies name blacked out, made by a FileRedactor
|
FileView.Transcription |
a text transcription by a FileTranscriber, masked by the policies' rules like any text |
The default view is Original, and Transcription where PII policies apply to the agent. Every file goes in
every view the agent has, each under its own heading: ### document (from start, file 1 of 2: invoice.pdf, application/pdf, 84 KB, original). A text format is its own transcription (no model call), and masked where
policies apply, in any view.
maxFiles (source files per model call), maxFileSize, maxTotalFileSize (all views of all files of one call)
and allowedMediaTypes are checked before any model call; a file over one is ModelFileLimitError
(step.model-file-limit), never a silent truncation. The limits cascade: the agent's own, its workflow's and
application's defaults.files, the @A2AExpose skills that reach it, @Application({ a2aFiles }), the
built-ins. Every file passes the guarded path once per visit — allowed origins, no private address, no user
info, size, digest, magic bytes of its media type — and every model call of the visit carries the same bytes.
audio/mpeg|mp3 → mp3, audio/wav|x-wav → wav, audio/ogg, audio/flac, audio/aac, audio/mp4|m4a → m4a go
as input_audio; video/mp4, video/quicktime, video/webm, video/x-matroska as video_url. The same
rules as images: the media check (magic bytes), the limits, the start check of the model's input kinds.
Another audio format is [model.unsupported-input].
At start, where the provider reports the kinds it takes (ModelCapabilities.inputs), every kind an agent's
files can produce must be in them — its model, its fallbacks, its input guardrails and judges, the built-in
transcriber's and redactor's models — else [model.unsupported-input]; a defaulted allowedMediaTypes is
narrowed to what the models take. Where the provider cannot say, the call fails with ModelCallError
model.unsupported-input, which is never retried and never falls back.
@PiiPolicy({ files: PiiFileAction.Mask | PiiFileAction.AllowOriginal }), default Mask:
- a transcription is masked by the rules where it is made, a block rule refusing before anything is kept;
- a redacted copy has every kind the rules name blacked out; any failure is
FileRedactionFailedError— never the unredacted file; -
a block rule refuses an image whole by default. Which kinds an image holds is a model's judgement, and a
text in the image can talk a model into silence ("no personal data here"). So where a policy has a block rule,
an image in the redacted view is refused (
PiiRefusedError, the policy's text) before any model is asked.PiiFileAction.RedactWithModelon the policy is the explicit opt-in to let the redaction model decide: a kind a block rule names that it finds refuses the file, an empty, malformed or truncated answer fails closed — and a model that obeys the text in the image lets it through. Model-checked images are a guardrail, not a guarantee: choose a deterministic redactor of your own, orTranscription(masked by formats, which no text in an image can talk down), where a leak is a payment-grade matter; - the original goes to the model only where every applying policy says
AllowOriginal([pii.original-file]otherwise);Redactedunder a project's own detector is[pii.redacted-needs-kinds].
PII never reads bytes: a file part is passed through the mask untouched.
The execution's exact sensitive values (sensitive context fields) go to the redaction model with the kinds only
where it is the same provider as the agent's own model (defaults.files.redactionValues: RedactionValues.SameProvider, the default). "The same" is decided fail-closed: the same provider, or one on the
same base URL with the same key field — never by a name; not when the two models are of different vendors behind
one aggregator (the default OpenRouter serves anthropic/… and google/… alike), and not when the redaction
provider's circuit breaker can fail over to another provider. In doubt it is not the same. Never sends the kinds only,
Always is an explicit decision to send them wherever the redactor runs. A project's own redactor gets the
values only on Always. Values that cannot be sent never leave a gap: where the contexts hold sensitive fields
and the redactor may not be told them the application does not start ([files.redactor-values-withheld], naming
both providers and the opt-in), and at run time a redaction with values it may not send, or with nothing at all
to look for, fails closed (FileRedactionFailedError) — an unchanged re-encode is never passed off as redacted.
A redacted copy is cached for the policies and the exact value set it was made for (one HMAC of both under the
file-cache key, Guards → the PII key hierarchy, and a "kinds only" marker when no
value is sent), in the state of one execution; a masked transcription by its policies' keyed hash.
Both are contracts, one per application, listed in @Application({ providers }); the built-ins run on a cheap
vision model (defaults.files.transcriptionModel, redactionModel, default google/gemini-2.5-flash-lite).
import { FileRedactor, Injectable, type RedactionResult } from "@agboss/core";
@Injectable()
export class PresidioImageRedactor extends FileRedactor {
supports(mediaType: string): boolean {
return mediaType === "image/png"; // asked at start, on the class: answer from the media type alone
}
redact(): Promise<RedactionResult> {
return Promise.reject(new Error("call your service here"));
}
}ModelImageRedactor redacts png, jpeg and webp: the image is turned upright (its EXIF orientation applied),
the model names the regions, a copy without metadata has them painted black, padded 10 %. A multi-frame
image, an image over 40 megapixels (its header says so: nothing is decoded), a model answer that is not strictly valid or names more than 200 regions, any failure — fail closed. All the regions are painted in one buffer: memory is one full-size image, however many regions there are (#1257).
sharp is an optional peer dependency (npm i sharp); the built-in without it is a start error. A type no
redactor supports is [files.no-redactor]. A redacted copy is kept in the file store when one is listed,
else inline in the execution's state up to 64 KB ([files.redacted-needs-store] for limits that allow more).
Spend is billed under files:transcriber / files:redactor (CostCategory.Files), the budget checked
before each call, and a copy is made once per execution for its file, redactor and policies.
Input guardrails and model judges see what the agent's model sees, files included; output guardrails see the
files of the answer (Guards). AnswerFiles.SeenInVisit (default): an answer's file is { file: <n> }, the number in the headings, mapped to the view the model saw; AnswerFiles.Any: the model's descriptors,
a url only https: on allowedFileOrigins, a fileId only one this execution owns. Only an original can be
returned by number: an agent that reads files only as Redacted or Transcription and answers with files is
[files.answer-file-unseen], and AnswerFiles.Any on an agent whose files a policy masks (without
FileView.Original) is [pii.original-file] — the model could hand back the unmasked input.
agboss describe prints each agent's files (input → original (5 files, 5 MB each, 10 MB in all) · answer files: seen-in-visit) and describe --json carries AgentSnapshot.files. The config hash covers the files
settings, each policy's file action and the application's own FileRedactor and FileTranscriber by class name
(#1269: another one is another configuration — a paused run is not resumed blindly under it). A request shows a file as { kind, mediaType, sizeBytes, sha256 } in
events, traces, records and cassettes — never its bytes (Observability); tests script
mockLlm(FileTranscriber) and mockLlm(FileRedactor) (Testing). The examples are invoice-workflow,
kyc-workflow and audio-note-workflow of examples/flow-patterns.
AGBoss
- Home
- Delivery Pipeline
- Architecture
- Configuration
- Environments
- Workflow — the paths
- Path scope
- Workflow as a step
- Parallel — quorums and arbiters
- Routing patterns
- Rollback — declared compensation
- Routers
- Guards
- CLI — agboss
- App — @Application and createApp
- Several instances
- Running on PostgreSQL
- Running on Redis
- Your own store
- Guarantees
- Guarantees across processes
- Payments
- Execution history
- Mesh Hub — agboss hub
- Health checks
- Metrics
- Autoscaling
- Deploying on Kubernetes
- Context and Authorization
- Testing
- The test split
- Suites and Comparisons
- Eval suites — @EvalSuite
- Components
- Naming
- Standard DTOs
- Examples: 01–08, flow-patterns
- Agents — the loop
- Model providers
- Tools
- Knowledge bases (RAG)
- A2A — agent to agent
- FinOps
- Tracing
- Observability
- Turns and evaluation
- Profiles and comparisons
- Pauses — pending, resume by id
- Pause seam
- Decisions (ADR)