Repository navigation
Azathoth 1.5 brings everything the command line does to the service API.
Production and its rollback; the judge rotation, a person's judgment of a
run, what its steps cited, the runs it started and the runs that never
finished; benchmarks, their cases and their runs; optimization, its
experiments and a winner set beside what it would replace; and the rest of
the catalog, a tool's versions, exports, where a model or a tool is used,
and goals: each is a call of azathoth.api, which the command line now
makes with its output unchanged, and which the local editor,
azathoth-ui, calls to do all
of it. Two things are new everywhere: azathoth spend says what an
instance spent, billed and estimated dollars always apart, and workflow diff, tool diff and benchmark diff say what changed between two
revisions. The database schema does not change, as Upgrading From 1.4
below says.
This release changes what the image runs on AWS only by what it adds: the
stack's one-off init task finds its database at schema version 7 and
leaves it there, and its workers run jobs as 1.4's did, each model call's
cost now saying how it is known. Release validation requires a release
that changes what the stack runs to be deployed to an AWS sandbox before it
ships, and no sandbox was available, so this release was not deployed. The
template was read, the worker run against botocore's Stubber and the S3
tools against a stand-in for S3, as for every release; what only a real
deployment proves, that the stack's workers run jobs on real SQS and S3, is
unproven for 1.5.0, as what 1.4.0 and 1.3.0 left unproven still is
(Release Validation).
Production
azathoth workflow rollback <WORKFLOW_ID>returns a workflow's production
to an earlier production revision, the one before the active revision by
default, or the one--tonames. It saves a new revision with the earlier
one's state, exactly as it ran, and refuses, changing nothing, when there
is nothing earlier, the revision is already active, or a step has neither
its fixed model nor an approved substitute still authorized
(ADR 0128).- Every production revision records who made it, told as an import's author
is, and what a rollback restored.workflow promote,workflow productionandworkflow revisionsshow both; a malformed
AZATHOTH_AUTHORstops a promotion before anything is written. Revisions
saved before 1.5 name nobody, and the database schema does not change
(ADR 0128). azathoth.apireaches production:instance.productionreads what a
workflow's production runs and its revisions, newest first, and promotes
and rolls back as the command line does.invocations.listlists a
workflow's invocations newest first, andinvocations.resumecontinues an
incomplete one from its journal. Six new errors carry codes of their own:
workflow_result_missing,experiment_not_found,
candidate_unavailable,candidate_rebuild,nothing_to_roll_back_to
andmodel_not_authorized
(ADR 0128).
Judges and a Run's Evidence
azathoth.apikeeps the judge rotation:instance.judgeslists it in
order, and adds and removes a judge, which must be served now and
authorized.judge add,judge listandjudge removecall it, and
listing and removing still reach no provider
(ADR 0129).azathoth.apitakes and reads a person's judgment of a run:
runs.record_feedbackrecords one immutable good or bad judgment, a bad
one needing a reason and only it taking a corrected output, and
runs.feedbacklists a run's judgments newest first. EveryRunSummary
andRunDetailcarries the run's latest judgment asfeedback, andrun feedbackcalls the API
(ADR 0129).runs.annotationsandinvocations.annotationsreturn the W3C Web
Annotation collection of every citation a run's steps verified, exactly
asrun annotationsandinvocation annotationsprint it, with its
annotations
(ADR 0129).runs.childrenlists the runs a run's steps started, oldest first, with
the step and attempt that started each and how it ended, judged as
runs.getjudges it.run childrenrenders what it returns, so a child
whose declared result is missing now showsfailed, asrun showdoes
(ADR 0129).azathoth run interrupted <WORKFLOW_ID>lists a workflow's runs that
started and never finished, newest first, with how often each was
resumed and the command that resumes it, its invocation's for a run that
executes one.runs.interruptedreturns the same list, andruns.resume
continues an interrupted configured run from its journal under its own
ID, through the same callrun resumemakes
(ADR 0129).- Two new errors carry codes of their own:
content_unavailable, for a
record of citation checks the content store no longer holds, and
run_not_resumable, for a run that finished, executes an invocation, is
still held by another process, or whose journal cannot be resumed
(ADR 0129).
Benchmarks
azathoth benchmark create --name <NAME> --description <TEXT>creates a
benchmark with no cases.azathoth benchmark edit-case <BENCHMARK_ID> <CASE_ID>changes a case's comparison, expected value, description,
citations or label, keeping its input and where it came from, and
azathoth benchmark remove-caseremoves one. Each saves the benchmark's
next revision; a label may be raised but never lowered, and an edit that
makes a case another's twin is refused
(ADR 0130).azathoth.apireads and changes a benchmark's cases:benchmarks.cases
andbenchmarks.caseread them with where each came from, and
benchmarks.create,add_case,update_caseandremove_casechange
them as the command line does.add_caseturns a run into a case,
expecting its latest correction when no value is given, or makes one
from JSON or a file.benchmark cases,case-showandadd-casecall
the API, their output unchanged
(ADR 0130).azathoth.apiruns benchmarks:benchmarks.runandbenchmarks.compare
check everything before any case runs, tell each case as it starts and
finishes, stop between cases when cancelled, keeping the case runs that
finished and recording no execution, and return what they saved.
benchmarks.executionsandbenchmarks.executionread saved runs and
comparisons, with the revision each ran, andbenchmark executionsand
execution-showread through them.benchmark runandcompareexecute
through the same calls. A new error,judge_unavailable, refuses
semantic cases no judge may judge
(ADR 0130).
Optimization
azathoth experiment compare <EXPERIMENT_ID>sets an experiment's winner
beside what promoting it would replace: what production runs now, when
the experiment holds it, or else the workflow's configured candidate. It
names each step whose model changes and how each case went on both
sides, and says what a run cost and took on each, measured from the case
runs the experiment kept. It says what the winner saves per run, in
dollars and seconds and as a share of the baseline, a negative saving
being a cost; projects that over production's invocations of the
workflow in the last 30 days, labelled as a projection; and ends with the
command that promotes the winner
(ADR 0131).azathoth.apireads experiments:experiments.listand
experiments.getread a workflow's experiments, every candidate ranked
with its scorecard, each case's run and evaluation, and the model each
step ran on.workflow experimentsandexperiment showread through
them, their output unchanged
(ADR 0131).azathoth.apioptimizes:optimization.optimizechecks everything
before any candidate runs, tells each generation as it starts and
finishes and each candidate as it finishes, stops between candidates
when cancelled, keeping every experiment it finished, and records every
experiment it names.workflow optimizeoptimizes through the same
function
(ADR 0131).azathoth.apicompares and checks before promoting:
experiments.comparereturns the comparisonexperiment compare
prints, with its savings and projection, andexperiments.promotable
says whether the winner can still be rebuilt from the workflow's latest
revision with the models authorized now, and why not, saving nothing
(ADR 0131).
Spend
azathoth spendshows what an instance spent, from--sinceup to
--until, by day, ISO week or month (--by) as the clock reads in a
time zone (--time-zone, this computer's by default), so a day the
clocks change is as long as it was. Each period's spend is grouped by
workflow, model, provider or source (--group), and--workflowkeeps
to one. Billed and estimated dollars are said apart, never added, and
unpriced calls are counted, never taken as free. It covers the last 30
days through today unless told otherwise, and--output jsonor
--output csvwrites the service API's report or a spreadsheet's rows,
to--output-fileif named
(ADR 0132).- Every model call's cost says how it is known, as
cost_basison its
reply and its step's metrics:billedwhen its provider reported the
charge, as OpenRouter does, andestimatedwhen Azathoth priced it, as
Bedrock's are. A judge's call records it too. Costs recorded before 1.5
read their basis from their provider, and the database schema does not
change. An OpenRouter reply that names no cost is now unpriced, never
free
(ADR 0132). - Spend counts each model call once: a step that started child runs is
counted at its own cost, the children's taken back out, and each child
run for itself, so a report by model puts every dollar on the model that
spent it. Production invocations' runs are their own source, and each
judge's call is judging, on the workflow of the run it judged
(ADR 0132). azathoth.apireports spend:spend.reportreturns aSpendReport,
every period of the range listed, spent in or not, each with its groups
and totals, asazathoth spendprints it, andapi.spend_csvwrites it
as CSV, RFC 4180, its times with their offsets and its amounts unrounded
(ADR 0132).- On Windows, which keeps no IANA time zone database, Azathoth installs
tzdata, soazathoth spendandspend.reportread any time zone there,
UTC included; without it, 1.5.0a5 refused every report on Windows
(ADR 0132).
The Catalog
- The design of the rest of the catalog through the service API: a tool's
versions, exports, where a model or a tool is used, comparing two
revisions, goals, and making a tool from an MCP server, with
import_goaladded to the catalog of actions, and a warning when a
remote token is made
(ADR 0133). azathoth.apibrowses the rest of the catalog:tools.versionslists a
tool's versions, andworkflows.export,tools.exportand
benchmarks.exportreturn aDocumentExport, the text theexport
commands print with a file name to suggest, and a benchmark's label.
models.usageandtools.usagereturn where a model or a tool is
configured, in production and executed, asmodel usageandtool usageprint it. Each of those commands now calls the API, with
unchanged output
(ADR 0133).azathoth workflow diff,tool diffandbenchmark diffprint what
differs between two revisions, a line per change by its JSON Pointer, or
that the two are identical;--output jsonwrites the service API's
RevisionComparison, which each namespace'scompare_revisionsreturns.
Withoutinclude_restricted, a restricted case's values are withheld as
null
(ADR 0133).azathoth.apikeeps goals:instance.goalslists, gets and imports
them, the newimport_goalaction. Importing a goal already kept,
unchanged, now keeps nothing andgoal importsays the goal is
unchanged, where it was refused; another goal with a kept ID is still
refused
(ADR 0133).tools.from_mcpreturns the document of a tool an MCP server serves,
astool from-mcpprints it, importing nothing; a server that cannot be
reached or listed is the newmcp_unavailableerror
(ADR 0133).remote token createwarns on standard error, when it makes a token,
that whoever holds it can do anything the remote's owner can, and from
1.6 through the whole service API
(ADR 0133).
Toward 1.6
- The design Azathoth 1.6 builds: a remote that serves the whole service
API over HTTP,api.connectand--remoteto reach it, and a catalog of
named actions for a remote's policy to decide. Nothing in this release
implements it yet; the editor's 1.5 pages are built to it
(ADR 0127).
Packaging
- Dependabot proposes new digests of the container image's base, and no
longer new Python versions, which wait until CI tests them.
Upgrading From 1.4
Upgrade, and run azathoth init as after every upgrade:
python -m pip install --upgrade azathoth-ai
azathoth initpython -m pip install --upgrade azathoth-ai
azathoth initIf you serve workflows with azathoth-mcp, upgrade it in the same command,
as Installing Azathoth shows: 1.5 of it requires 1.5 of
azathoth-ai.
The database schema does not change: it stays at version 7, and init
has nothing of it to migrate. Documents, commands and the service API keep
what 1.4 promised, growing only by additions, and every command that now
goes through the service API prints what it did, but for the few lines
below.
What an existing installation will notice:
- production revisions saved before 1.5 name nobody as their author, and a
malformedAZATHOTH_AUTHORnow stops a promotion before anything is
written, as it stops an import; run childrenshows a child whose declared result is missing as
failed, asrun showdoes;- an OpenRouter reply that names no cost is unpriced, never free, and costs
recorded before 1.5 read their basis from their provider; - importing a goal already kept, unchanged, succeeds and says the goal is
unchanged, where it was refused; remote token createwarns on standard error what a token can do.
On Windows, Azathoth now installs tzdata, so spend is read in any time
zone there, UTC included.