Skip to content

Azathoth 1.5.0

Latest

Choose a tag to compare

@OMontano OMontano released this 20 Nov 19:24
f3538b8

Azathoth 1.5 brings everything the command line does to the service API.
Production and its rollback; the judge rotation, a person's judgment of a
run, what its steps cited, the runs it started and the runs that never
finished; benchmarks, their cases and their runs; optimization, its
experiments and a winner set beside what it would replace; and the rest of
the catalog, a tool's versions, exports, where a model or a tool is used,
and goals: each is a call of azathoth.api, which the command line now
makes with its output unchanged, and which the local editor,
azathoth-ui, calls to do all
of it. Two things are new everywhere: azathoth spend says what an
instance spent, billed and estimated dollars always apart, and workflow diff, tool diff and benchmark diff say what changed between two
revisions. The database schema does not change, as Upgrading From 1.4
below says.

This release changes what the image runs on AWS only by what it adds: the
stack's one-off init task finds its database at schema version 7 and
leaves it there, and its workers run jobs as 1.4's did, each model call's
cost now saying how it is known. Release validation requires a release
that changes what the stack runs to be deployed to an AWS sandbox before it
ships, and no sandbox was available, so this release was not deployed. The
template was read, the worker run against botocore's Stubber and the S3
tools against a stand-in for S3, as for every release; what only a real
deployment proves, that the stack's workers run jobs on real SQS and S3, is
unproven for 1.5.0, as what 1.4.0 and 1.3.0 left unproven still is
(Release Validation).

Production

  • azathoth workflow rollback <WORKFLOW_ID> returns a workflow's production
    to an earlier production revision, the one before the active revision by
    default, or the one --to names. It saves a new revision with the earlier
    one's state, exactly as it ran, and refuses, changing nothing, when there
    is nothing earlier, the revision is already active, or a step has neither
    its fixed model nor an approved substitute still authorized
    (ADR 0128).
  • Every production revision records who made it, told as an import's author
    is, and what a rollback restored. workflow promote, workflow production and workflow revisions show both; a malformed
    AZATHOTH_AUTHOR stops a promotion before anything is written. Revisions
    saved before 1.5 name nobody, and the database schema does not change
    (ADR 0128).
  • azathoth.api reaches production: instance.production reads what a
    workflow's production runs and its revisions, newest first, and promotes
    and rolls back as the command line does. invocations.list lists a
    workflow's invocations newest first, and invocations.resume continues an
    incomplete one from its journal. Six new errors carry codes of their own:
    workflow_result_missing, experiment_not_found,
    candidate_unavailable, candidate_rebuild, nothing_to_roll_back_to
    and model_not_authorized
    (ADR 0128).

Judges and a Run's Evidence

  • azathoth.api keeps the judge rotation: instance.judges lists it in
    order, and adds and removes a judge, which must be served now and
    authorized. judge add, judge list and judge remove call it, and
    listing and removing still reach no provider
    (ADR 0129).
  • azathoth.api takes and reads a person's judgment of a run:
    runs.record_feedback records one immutable good or bad judgment, a bad
    one needing a reason and only it taking a corrected output, and
    runs.feedback lists a run's judgments newest first. Every RunSummary
    and RunDetail carries the run's latest judgment as feedback, and run feedback calls the API
    (ADR 0129).
  • runs.annotations and invocations.annotations return the W3C Web
    Annotation collection of every citation a run's steps verified, exactly
    as run annotations and invocation annotations print it, with its
    annotations
    (ADR 0129).
  • runs.children lists the runs a run's steps started, oldest first, with
    the step and attempt that started each and how it ended, judged as
    runs.get judges it. run children renders what it returns, so a child
    whose declared result is missing now shows failed, as run show does
    (ADR 0129).
  • azathoth run interrupted <WORKFLOW_ID> lists a workflow's runs that
    started and never finished, newest first, with how often each was
    resumed and the command that resumes it, its invocation's for a run that
    executes one. runs.interrupted returns the same list, and runs.resume
    continues an interrupted configured run from its journal under its own
    ID, through the same call run resume makes
    (ADR 0129).
  • Two new errors carry codes of their own: content_unavailable, for a
    record of citation checks the content store no longer holds, and
    run_not_resumable, for a run that finished, executes an invocation, is
    still held by another process, or whose journal cannot be resumed
    (ADR 0129).

Benchmarks

  • azathoth benchmark create --name <NAME> --description <TEXT> creates a
    benchmark with no cases. azathoth benchmark edit-case <BENCHMARK_ID> <CASE_ID> changes a case's comparison, expected value, description,
    citations or label, keeping its input and where it came from, and
    azathoth benchmark remove-case removes one. Each saves the benchmark's
    next revision; a label may be raised but never lowered, and an edit that
    makes a case another's twin is refused
    (ADR 0130).
  • azathoth.api reads and changes a benchmark's cases: benchmarks.cases
    and benchmarks.case read them with where each came from, and
    benchmarks.create, add_case, update_case and remove_case change
    them as the command line does. add_case turns a run into a case,
    expecting its latest correction when no value is given, or makes one
    from JSON or a file. benchmark cases, case-show and add-case call
    the API, their output unchanged
    (ADR 0130).
  • azathoth.api runs benchmarks: benchmarks.run and benchmarks.compare
    check everything before any case runs, tell each case as it starts and
    finishes, stop between cases when cancelled, keeping the case runs that
    finished and recording no execution, and return what they saved.
    benchmarks.executions and benchmarks.execution read saved runs and
    comparisons, with the revision each ran, and benchmark executions and
    execution-show read through them. benchmark run and compare execute
    through the same calls. A new error, judge_unavailable, refuses
    semantic cases no judge may judge
    (ADR 0130).

Optimization

  • azathoth experiment compare <EXPERIMENT_ID> sets an experiment's winner
    beside what promoting it would replace: what production runs now, when
    the experiment holds it, or else the workflow's configured candidate. It
    names each step whose model changes and how each case went on both
    sides, and says what a run cost and took on each, measured from the case
    runs the experiment kept. It says what the winner saves per run, in
    dollars and seconds and as a share of the baseline, a negative saving
    being a cost; projects that over production's invocations of the
    workflow in the last 30 days, labelled as a projection; and ends with the
    command that promotes the winner
    (ADR 0131).
  • azathoth.api reads experiments: experiments.list and
    experiments.get read a workflow's experiments, every candidate ranked
    with its scorecard, each case's run and evaluation, and the model each
    step ran on. workflow experiments and experiment show read through
    them, their output unchanged
    (ADR 0131).
  • azathoth.api optimizes: optimization.optimize checks everything
    before any candidate runs, tells each generation as it starts and
    finishes and each candidate as it finishes, stops between candidates
    when cancelled, keeping every experiment it finished, and records every
    experiment it names. workflow optimize optimizes through the same
    function
    (ADR 0131).
  • azathoth.api compares and checks before promoting:
    experiments.compare returns the comparison experiment compare
    prints, with its savings and projection, and experiments.promotable
    says whether the winner can still be rebuilt from the workflow's latest
    revision with the models authorized now, and why not, saving nothing
    (ADR 0131).

Spend

  • azathoth spend shows what an instance spent, from --since up to
    --until, by day, ISO week or month (--by) as the clock reads in a
    time zone (--time-zone, this computer's by default), so a day the
    clocks change is as long as it was. Each period's spend is grouped by
    workflow, model, provider or source (--group), and --workflow keeps
    to one. Billed and estimated dollars are said apart, never added, and
    unpriced calls are counted, never taken as free. It covers the last 30
    days through today unless told otherwise, and --output json or
    --output csv writes the service API's report or a spreadsheet's rows,
    to --output-file if named
    (ADR 0132).
  • Every model call's cost says how it is known, as cost_basis on its
    reply and its step's metrics: billed when its provider reported the
    charge, as OpenRouter does, and estimated when Azathoth priced it, as
    Bedrock's are. A judge's call records it too. Costs recorded before 1.5
    read their basis from their provider, and the database schema does not
    change. An OpenRouter reply that names no cost is now unpriced, never
    free
    (ADR 0132).
  • Spend counts each model call once: a step that started child runs is
    counted at its own cost, the children's taken back out, and each child
    run for itself, so a report by model puts every dollar on the model that
    spent it. Production invocations' runs are their own source, and each
    judge's call is judging, on the workflow of the run it judged
    (ADR 0132).
  • azathoth.api reports spend: spend.report returns a SpendReport,
    every period of the range listed, spent in or not, each with its groups
    and totals, as azathoth spend prints it, and api.spend_csv writes it
    as CSV, RFC 4180, its times with their offsets and its amounts unrounded
    (ADR 0132).
  • On Windows, which keeps no IANA time zone database, Azathoth installs
    tzdata, so azathoth spend and spend.report read any time zone there,
    UTC included; without it, 1.5.0a5 refused every report on Windows
    (ADR 0132).

The Catalog

  • The design of the rest of the catalog through the service API: a tool's
    versions, exports, where a model or a tool is used, comparing two
    revisions, goals, and making a tool from an MCP server, with
    import_goal added to the catalog of actions, and a warning when a
    remote token is made
    (ADR 0133).
  • azathoth.api browses the rest of the catalog: tools.versions lists a
    tool's versions, and workflows.export, tools.export and
    benchmarks.export return a DocumentExport, the text the export
    commands print with a file name to suggest, and a benchmark's label.
    models.usage and tools.usage return where a model or a tool is
    configured, in production and executed, as model usage and tool usage print it. Each of those commands now calls the API, with
    unchanged output
    (ADR 0133).
  • azathoth workflow diff, tool diff and benchmark diff print what
    differs between two revisions, a line per change by its JSON Pointer, or
    that the two are identical; --output json writes the service API's
    RevisionComparison, which each namespace's compare_revisions returns.
    Without include_restricted, a restricted case's values are withheld as
    null
    (ADR 0133).
  • azathoth.api keeps goals: instance.goals lists, gets and imports
    them, the new import_goal action. Importing a goal already kept,
    unchanged, now keeps nothing and goal import says the goal is
    unchanged, where it was refused; another goal with a kept ID is still
    refused
    (ADR 0133).
  • tools.from_mcp returns the document of a tool an MCP server serves,
    as tool from-mcp prints it, importing nothing; a server that cannot be
    reached or listed is the new mcp_unavailable error
    (ADR 0133).
  • remote token create warns on standard error, when it makes a token,
    that whoever holds it can do anything the remote's owner can, and from
    1.6 through the whole service API
    (ADR 0133).

Toward 1.6

  • The design Azathoth 1.6 builds: a remote that serves the whole service
    API over HTTP, api.connect and --remote to reach it, and a catalog of
    named actions for a remote's policy to decide. Nothing in this release
    implements it yet; the editor's 1.5 pages are built to it
    (ADR 0127).

Packaging

  • Dependabot proposes new digests of the container image's base, and no
    longer new Python versions, which wait until CI tests them.

Upgrading From 1.4

Upgrade, and run azathoth init as after every upgrade:

python -m pip install --upgrade azathoth-ai
azathoth init
python -m pip install --upgrade azathoth-ai
azathoth init

If you serve workflows with azathoth-mcp, upgrade it in the same command,
as Installing Azathoth shows: 1.5 of it requires 1.5 of
azathoth-ai.

The database schema does not change: it stays at version 7, and init
has nothing of it to migrate. Documents, commands and the service API keep
what 1.4 promised, growing only by additions, and every command that now
goes through the service API prints what it did, but for the few lines
below.

What an existing installation will notice:

  • production revisions saved before 1.5 name nobody as their author, and a
    malformed AZATHOTH_AUTHOR now stops a promotion before anything is
    written, as it stops an import;
  • run children shows a child whose declared result is missing as
    failed, as run show does;
  • an OpenRouter reply that names no cost is unpriced, never free, and costs
    recorded before 1.5 read their basis from their provider;
  • importing a goal already kept, unchanged, succeeds and says the goal is
    unchanged, where it was refused;
  • remote token create warns on standard error what a token can do.

On Windows, Azathoth now installs tzdata, so spend is read in any time
zone there, UTC included.