armada runs a command, or a TypeScript or Python function, over many inputs at once on Cloudflare Containers in your own account. I run Kinu's and Dew's CI on it.

• 100 videos re-encoded to 720p in 21 to 22 s on 100 containers, 1.3 to 1.4 times the slowest single encode. One container took 12 minutes for the same 100.
• A 90-row CI suite ran in 7 min 27 s on 13 containers. Its GitHub Actions matrix of 15 jobs took 13 min 2 s.
• armada runs the longest tasks first. Over 325 green runs of Kinu's suite, the tests ended a median 2.0% after their lower bound: the longest row, or all the rows' time split evenly over the containers, whichever is longer.
• One true task came back 2.5 s after it was sent.

curl -fsSL https://raw.githubusercontent.com/AshishKumar4/armada/main/install.sh | sh
armada deploy
armada map --times=3 --json -- echo hello {item}

The script installs Bun if it's missing and puts armada in ~/.armada. armada deploy logs in to Cloudflare in your browser, or uses CLOUDFLARE_API_TOKEN. Containers need the Workers Paid plan.

armada is a Worker, a few Durable Objects and an R2 bucket in your account, so your code and data stay there.

1. armada prepares each environment once. It starts a container from the recipe's base image, runs setup as root and install as the user, and snapshots the result. A changed recipe, or a changed file in environment.key, prepares a new one.
2. Containers start from the snapshot, with a fresh tmpfs on /tmp and /dev/shm.
3. Each container pulls task after task from the job's one queue until it is empty, so no container waits behind another's slow task. Items with a higher weight start first.
4. Results stream back as they land. A small output rides in the result. A large one, and every log, goes to R2.

Each task runs as the user ci, in its own cgroup, and armada stops anything it leaves running. A task the platform loses runs again, up to three attempts, so a task should be safe to run twice.

Each video task makes a 10-second 1080p clip from ffmpeg's test sources, then re-encodes it to 720p H.264.

# A flaky test, 20 times at once, in the commit's own environment
armada map --commit=HEAD --times=20 -- bun test tests/flaky.test.ts

# One task per line of urls.txt, each keeping the file it writes
armada map --items=urls.txt --output -- sh -c 'curl -sL {item} > {out}'

An unknown placeholder is an error. An object item's numeric weight moves it up the queue. map exits 1 if a task exits nonzero and 2 if a task could not run.

Each task's environment names its run: ARMADA_JOB_ID, ARMADA_INDEX, ARMADA_ATTEMPT, and ARMADA_VESSEL for its container. ARMADA_STATUS_TOKEN reads its job's tasks and nothing else:

curl -H "Authorization: Bearer $ARMADA_STATUS_TOKEN" "$ARMADA_STATUS_URL/$ARMADA_INDEX"

The answer gives the job's phase, the task's state and attempt, and its exit code once it ends. It also lists every run of the task that may still be going, by container, slot and attempt. A run stays listed until its process has exited or been killed, or its container is gone, even after the job has moved on. A run whose container armada could not stop stays listed. A task that leaves something behind, such as a test fixture, can record these names, the URL and the token. A later task can then check that the run is gone before it cleans up.

An object item's gang runs it on that many containers at once, for a program that spans hosts, such as a multi-host JAX run. Each rank gets ARMADA_RANK and ARMADA_WORLD, and reaches rank r as host rank<r> (127.0.1.r+1) on any port it listens on. jax.distributed's coordinator at rank0:<port> and gloo's collectives work over it unchanged.

echo '[{"gang": 2}]' | armada map --items=- -- sh -c 'python3 train.py --rank $ARMADA_RANK --coordinator rank0:8476'

The gang starts once every rank has a container. A rank lost to the platform loses the whole gang, which runs again as one task. The task's outcome is its first failing rank's, else rank 0's.

Containers have no inbound address, so ranks connect through the Worker: about 5 ms a round trip and 40 to 90 MB/s a connection. That is enough for tests and coordination, but too slow for bandwidth-bound training. A connection made before the other rank listens closes at once, so connect with retries.

This example reads each video's length and resolution, then re-encodes every video to 720p, each on its own container.

Add armada with bun add github:AshishKumar4/armada. armada.config.ts names the project and its tasks' folder:

// armada.config.ts
import { defineConfig } from 'armada';

export default defineConfig({ project: 'media', tasks: ['armada'] });

// armada/video.ts
import { recipe, sh, task } from 'armada';
import * as v from 'valibot';

export const media = recipe.debian().apt('ffmpeg').size('small');

// Re-encode one video to 720p H.264. sh passes ${url} and ${out} as single words, so nothing needs quoting.
export const transcode = task({
  id: 'transcode',
  recipe: media,
  output: 'bytes',
  run: (url: string, { out }) =>
    sh`ffmpeg -loglevel error -i ${url} -vf scale=-2:720 -c:v libx264 -preset veryfast -c:a aac -f mp4 ${out}`,
});

// Read a video's length and resolution with ffprobe. The output schema checks each answer when it lands.
export const probe = task({
  id: 'probe',
  recipe: media,
  output: v.object({ seconds: v.number(), width: v.number(), height: v.number() }),
  run: async (url: string) => {
    const info = JSON.parse(await sh`ffprobe -v error -select_streams v:0 -show_entries stream=width,height:format=duration -of json ${url}`.text());

    return { seconds: Number(info.format.duration), width: Number(info.streams[0].width), height: Number(info.streams[0].height) };
  },
});

// encode.ts
import { probe, transcode } from './armada/video';

const videos = (await Bun.file('videos.txt').text()).split('\n').filter(Boolean);

const infos = await probe.map(videos); // { seconds, width, height }[], in input order
console.log(`${videos.length} videos, ${Math.round(infos.reduce((sum, info) => sum + info.seconds, 0))} seconds in all`);

const started = Date.now();
const job = transcode.stream(videos);
let slowest = 0;

for await (const result of job) { // each file as soon as its container finishes
  slowest = Math.max(slowest, result.meta.seconds); // how long its encode ran on its container
  if (result.ok) await Bun.write(`720p/${String(result.index)}.mp4`, result.value);
  else console.error(`${result.item}: ${result.kind}`);
}
const { vessels } = await job.summary();
console.log(`encoded in ${((Date.now() - started) / 1000).toFixed(1)} s on ${vessels} containers; the slowest encode took ${slowest.toFixed(1)} s`);

With two 10-second 1080p test videos in videos.txt, bun encode.ts printed:

2 videos, 20 seconds in all
encoded in 9.1 s on 2 containers; the slowest encode took 6.4 s

• .map returns values in input order and throws a MapError if any item fails. .stream yields each result as it lands. .run runs one item on a container, and .local runs it on this machine.
• A failed result's kind is error, timeout, cancelled or lost. Every result's meta has its seconds, exit code, log tail, and peak memory and CPU.
• armada push uploads the project's tasks and drops the ones it no longer exports. A script in the project pushes on its first run, and armada dev pushes on every save.
• A recipe is recipe.debian() or recipe.from(image), then .apt(), .setup(), .install() and .size(). Each container also loads the task's file, so a recipe that reads local files goes in a function, recipe: () => ..., which runs only on the machine that starts the job.
• sh passes each ${} as one word. sh.raw`...` doesn't escape, for a script that is itself shell.
• A task's output checks each answer when it lands, and types it: a schema, 'text' or 'bytes'. Without one, a body's answer is JSON and a command's is null. An input schema checks each item before it's sent.
• Schemas can be valibot, zod or arktype (any Standard Schema). Items and answers are plain JSON, or bytes for an answer. A Date, a Map, any or unknown is a type error.
• An output can be up to 4.995 GiB, R2's limit for one upload, and job.outputStream(i) streams it.
• map(items, { pool, slots, mode, label, env, files, tmpfs }) sets a job's options.
• A job packs its tasks into containers by what they use, so it holds fewer containers. This is mode: 'efficient', the default. armada records each task's CPU, peak memory and disk, by its task and environment and by its item. A container takes another task only while their recorded use fits its vCPUs, memory and disk. A task armada has not measured yet runs alone, and so does a gang's rank.
• Each packed task has its own files and /tmp, and all of them share one memory cap, so a task past its estimate is killed rather than the container. A task that fails beside others, killed for memory, red or past its timeout, runs again alone, before its own retries. Its result says how the packed run ended. If it then passes, its item always runs alone after that.
• mode: 'fast' runs one task a container. Packed tasks share the network, so tasks that bind one fixed port need it.
• slots runs exactly that many tasks at once in each container, each with an even share of the memory, and turns packing off. A gang cannot run in slots.
• A task takes a timeout. speculative lets an idle container rerun a straggler, and hedge runs that many of the heaviest items twice from the start. Either way the first answer is kept.
• retries reruns an item only for the failures you name. With retries: { attempts: 3, backoffSeconds: 5, exitCodes: [75], errors: ['FetchError'] }, an item that exits 75 or throws a FetchError runs up to three times, waiting 5 s, then 10 s. Any other failure is final.
• secrets gives a task values kept out of your code. echo "$KEY" | armada secret set OPENAI_API_KEY stores one, and a task with secrets: ['OPENAI_API_KEY'] reads context.secrets.OPENAI_API_KEY. A command gets it in its environment. Reading a secret the task didn't name is a type error, and no call reads a value back. Each value of 4 bytes or more shows as *** in logs. .local reads secrets from your environment.
• cache: { days: 7 } keeps each green answer for a week, keyed by the task, the item, the recipe and the pushed task files. An item answered before comes back at once with meta.cached, and no container starts. Use it only for a task whose answer its item decides.

The Python SDK runs the same tasks over the same wire as the TypeScript one, so a job's results read the same from either side. Install it with pip install "git+https://github.com/AshishKumar4/armada#subdirectory=python". A [tool.armada] table in pyproject.toml names the project and its tasks' folders, ["tasks"] if it leaves them out. A folder may not be named armada, which is the SDK's own package:

# pyproject.toml
[tool.armada]
project = "media"
tasks = ["tasks"]

# tasks/video.py
from armada import Context, recipe, sh, task

@task(id="transcode", recipe=recipe.debian().apt("ffmpeg").size("small"), output="bytes")
def transcode(url: str, ctx: Context):
    # sh quotes each {} itself; ctx.out is the file the task answers with.
    return sh("ffmpeg -loglevel error -i {} -vf scale=-2:720 -c:v libx264 -preset veryfast -c:a aac -f mp4 {}", url, ctx.out)

python -m armada push bundles the tasks and records them. transcode.map(urls) returns the values in input order, transcode.stream(urls) yields each result as it lands, transcode.run(url) runs one item, and transcode.local(url) runs it on this machine. A body's return value is its answer, and bytes come back as bytes. An output model or function checks each answer when it lands. examples/python-square shows validators, secrets, artifacts, retries and caching.

armada run HEAD

armada run uploads the commit from your machine, so private repos and unpushed commits work. It reads the commit's .armada.json, runs the plan command, and runs one task per matrix entry. armada tests itself this way:

{
  "name": "armada",
  "environment": {
    "setup": "ci/setup.sh",
    "install": "ci/install.sh",
    "key": ["bun.lock", "package.json"]
  },
  "pool": 8,
  "size": "auto",
  "plan": { "command": ["echo", "{\"include\": [{\"name\": \"test\"}, {\"name\": \"typecheck\"}, {\"name\": \"relay\", \"gang\": 8}, {\"name\": \"proofs\"}, {\"name\": \"python\"}, {\"name\": \"lint\"}]}"] },
  "task": { "command": ["bun", "run", "{name}"], "verdict": false, "timeout": 600 }
}

Words after armada run HEAD -- go to the plan command, to run part of the matrix. --secrets=A,B gives one run's tasks those secrets, for a narrowed run whose credentials the whole matrix must not see. Ctrl-C cancels the job.

A run of the whole matrix stores its verdict under the commit. armada verdict <commit> prints it and exits 0 when every row is green, 1 when a row is red and 2 when the commit has none, so a hook or a deploy can reuse the proof. Every run writes its report and prints report: <path>, including a run that cannot grade every row.

armada run --json prints the progress on stderr and ends with one JSON object on stdout: {sha, planJob, job, report, graded: "pass" | "fail" | "not graded", problems, rows}. The report file holds the same graded and rows. A row a problem names exits 2 and carries the problem, so it never reads green.

A task can keep files beside its verdict. It writes them under {artifacts} (ARMADA_ARTIFACTS), and a row names its evidence: "artifacts": ["shots/home.png"]. armada run extracts them next to the report and prints each named file under a red row. A row naming a file its task did not keep leaves the run ungraded. Artifacts are kept as written, so the log's secret mask does not apply to them. armada map --artifacts=<dir> extracts them under <dir>/<index>.

A matrix entry may name the rows its task must report. A task that reports other rows leaves the run ungraded.

A matrix entry may also name an environment from environments. Each named environment is the project's environment with the fields it gives replaced, and it gets its own snapshot. Its entries run in a task job of their own, so a container restores only what its tasks need. The pool is split between the jobs by their entries' past timings, or by their counts. The run grades every row as one, and its report lists each job under jobs. For example, one project tests two Pythons, each from a snapshot that holds only its own packages:

{
  "environment": { "setup": "ci/setup.sh", "install": "ci/install-312.sh" },
  "environments": { "py314": { "install": "ci/install-314.sh" } }
}

Its plan prints {"name": "test-312"} for the project's environment and {"name": "test-314", "environment": "py314"} for the other.

With "size": "auto", each run takes the smallest size that the last five runs' tasks fill to three quarters at most, in peak memory and average busy cores. A new size prepares its own environment once.

armada webhook add armada --repo=owner/name --pull-requests

This gives the project a webhook on the deployment. Each push to the repo's branches, and each pull request from the same repository, starts that commit's whole armada run on the deployment itself. With no --branches, only the default branch builds. The run stores its verdict as a local run does, and posts a GitHub commit status, such as "88 of 90 rows green", that links to the dashboard's CI page.

GitHub's X-Hub-Signature-256 signs each delivery, and a delivery or a commit seen before is skipped. A pull request from a fork never builds, because its code must not get the deployment's secrets. The run uses armada at the deployment's own commit, so deploy from a pushed commit.

armada webhook add creates or updates the GitHub hook through gh when you're signed in. Otherwise it prints the settings to add by hand. armada webhook list and armada webhook remove <project> manage them.

armada dashboard opens the deployment's dashboard in your browser, signed in, and the token never leaves the browser. armada dashboard --serve=<port> serves it from your machine instead, which signs each request itself.

It shows the fleet and its recent jobs, each job's tasks and a timeline of its containers, and each task's item, output, log and artifacts. It also lists the environments and each project's CI verdicts. A search, and a pick of a repository job's project, narrow the jobs.

armada dashboard --all also signs the page in to every other deployment you have a connection to and that answers. Its fleet page then shows each deployment's containers in a row and all their recent jobs in one list, and a pick in the top bar opens one deployment's own pages. Each bearer stays in that one browser.

armada deploy --name=<name> deploys a second armada on the same account and prints the file that --connection=<file> takes, to point any command at it. --vcpus=N caps that deployment's fleet.

A deploy over a running armada keeps its jobs running. It waits for them first with --drain, or when the new version cannot take them over: when it would refuse a client the deployed one serves, or when its Durable Objects call each other differently or store other shapes, since the deployed objects keep running their code for minutes after the deploy. Such a version takes its first job once its deployment-wide objects run its code; until then a new job is refused as during the drain.

The Worker deletes the environment snapshots it no longer needs, because an account's snapshots are limited. It keeps the ones open jobs use, the 3 most recently used others (--keep=N changes the count), and any used in the last hour. It prunes when deployed, after each new environment, and every hour. armada prune prunes now, and --keep=0 keeps only what open jobs use.

A client and a Worker too far apart in version refuse each other's requests, and say which one to update.

armada deploy [--account=<id>] [--name=<name>] [--vcpus=N] [--drain] [--keep=N]
armada map [--env=<recipe.json> | --commit=<rev>] (--times=N | --items=<file|->) [--size=<size>] [--pool=N] [--slots=N] [--mode=efficient|fast] [--timeout=S] [--output] [--artifacts=<dir>] [--speculative] [--hedge=N] [--secrets=<A,B>] [--json] -- <command>
armada run <commit|worktree> [--label=<text>] [--secrets=<A,B>] [--json] [-- <plan args>]
armada verdict <commit|worktree> [--json]
armada push
armada dev
armada status <job-id>
armada dashboard [--serve=<port>] [--all]
armada secret set <NAME> | list | delete <NAME>
armada webhook add <project> --repo=<owner/name> [--branches=a,b] [--pull-requests] [--token-secret=NAME]
armada webhook list
armada webhook remove <project>
armada prune [--keep=N]

armada --help describes every option.