Repository navigation
Azathoth 1.1.0
Note: the AWS deployment (Running on AWS) has been checked in two halves: CloudFormation created every resource in the template in a real AWS account, and the released image ran its queue worker, the guide's job, the large-file example and a mid-job stop against an emulator of SQS and S3. A run on Fargate itself will follow; any fix it needs ships as 1.1.1.
Azathoth 1.1 runs in production. Programs send it jobs through a queue,
workers on AWS run them against one PostgreSQL database, and a run cut short
by a deploy or a lost machine resumes where it stopped. Workflows compose from
a standard tool library that ships in the package, map a child over a list,
take their inputs by reference, and process data of any size from S3. Models
run on OpenRouter or Amazon Bedrock, and every run can be traced.
Composing Workflows
- A standard tool library ships in the package, installed by
azathoth init(ADR 0095).
Composing workflows needs no checkout of the repository. - Data-shaping tools pick, build, merge and validate JSON and split, join
and fill in text:json_select,json_object,json_merge,
validate_json,chunk_text,join_textandrender_template. - A step input may be a constant, written in the workflow document
(ADR 0094). - Loops are budgeted from their settings:
loop_workflow1.1.0 takes a
budget_seconds, and every process tool is told its own timeout
(ADR 0093). map_workflowruns a child once per item of a list, several at a
time, and returns the results in order
(ADR 0106).- A child records where it came from: the step and run that started it,
and its root, with nesting bounded by the engine
(ADR 0104).
Inputs and Jobs
- An input is a value or a reference, a
file://ors3://URI with a
media type, resolved before the run starts and kept by its digest
(ADR 0096,
ADR 0108). - A versioned job envelope requests an invocation and reports its result,
with an idempotency key that makes a repeated request replay
(azathoth job run,
ADR 0097). - Data larger than an input may be stays in S3:
split_s3_objectstreams
an object into chunk objects, andread_s3_objectreads one back
(ADR 0110).
Running in Production
- PostgreSQL keeps durable state for many processes at once, selected by
apostgresql://AZATHOTH_DATABASE
(ADR 0102). - Interrupted runs resume from their journal, under a lease one process
holds at a time (azathoth run resume,
ADR 0105). - A container image for amd64 and arm64, with an SBOM and provenance,
signed by the release workflow
(ADR 0101). - Queue workers take job requests from SQS, settle each message by its
result, keep a running job's message hidden, and drain on SIGTERM
(azathoth worker run,
ADR 0107). - One CloudFormation template deploys a fleet of workers on Fargate, with
least-privilege IAM, secrets from Secrets Manager and scaling on the queue
(ADR 0109).
Models and Evidence
- Amazon Bedrock runs models beside OpenRouter, with its own discovery,
IAM policy and prices
(ADR 0103). - OpenTelemetry tracing of every run, step and model call, as an optional
extra (ADR 0100). - A step can report its confidence, recorded with its outputs
(ADR 0098). - The optimizer evaluates its final generation, so a candidate proposed
last is measured before the winner is chosen
(ADR 0091),
andworkflow promote --experimentdeploys an experiment's winner
directly (ADR 0092).
Security
- Process tools see only the environment they declare, beside a small
base set, so a tool never receives a key it did not ask for
(ADR 0099).
Upgrading From 1.0
Upgrade the package, then run init once against each database:
python -m pip install --upgrade azathoth-ai
azathoth initpython -m pip install --upgrade azathoth-ai
azathoth init1.1 upgrades a 1.0 database in place the first time it opens it, adding the
tables that record a run as it executes. init, which is new in 1.1, then
installs the standard tools, which ship in the package from 1.1 on. A
database holding the composition tools 1.0 shipped as examples gets:
Database: /path/to/azathoth.db (existing)
Installed tool run_workflow@1.0.0 revision 2.
Installed tool loop_workflow@1.1.0 revision 1.
Installed tool map_workflow@1.0.0 revision 1.
Installed tool json_select@1.0.0 revision 1.
Installed tool json_object@1.0.0 revision 1.
Installed tool json_merge@1.0.0 revision 1.
Installed tool validate_json@1.0.0 revision 1.
Installed tool chunk_text@1.0.0 revision 1.
Installed tool join_text@1.0.0 revision 1.
Installed tool render_template@1.0.0 revision 1.
Installed tool split_s3_object@1.0.0 revision 1.
Installed tool read_s3_object@1.0.0 revision 1.
Two changes can stop a workflow that ran on 1.0:
- Process tools no longer inherit Azathoth's environment. The standard tools
grant themselves what they need; a process tool of your own that relied
on an inherited variable, such as a provider key, must grant it (see
Process Environment). - A workflow that requires
loop_workflow1.0.0, the example 1.0 shipped,
must name 1.1.0. The 1.0.0 implementation grants itself nothing, so its
children no longer seeAZATHOTH_DATABASE, and openazathoth.dbin
whatever directory they run in instead of yours.run_workflow1.0.0
needs no change:initgives it its grants as its next revision.