AgentSwarms 1.3.0

AgentSwarms 1.3.0

The data half grows up. 1.2 described an agentic AI platform with a BI surface;
1.3 ships the other half that agents need pointing at: a lakehouse of your
own (DuckDB over Parquet in your bucket, with a transactional catalog), ETL
pipelines that feed it, no-code machine learning that trains on it, and
decision provenance that ties every answer to the snapshot it read. Around
those land web-crawl and Confluence knowledge sources, Jira and Zendesk
datasets, Azure lakes, local embedding models, a production server that uses
every core, a Kubernetes path proven on a real cluster, backups that prove they
restore, and a README half its previous size.

62 commits, 330 files, 27 migrations.

---

Highlights

A lakehouse built in. DuckDB as the engine, a Postgres catalog for
transactions, zstd Parquet in your own object storage. Compute is per-request
and stateless, so replicas behind a load balancer share storage instead of
sharding it. Snapshot time travel, partition pruning, a result cache keyed on
the catalog snapshot, materialized views, spill to disk, hourly compaction, and
mounted data lakes queried in place. Governance is one chokepoint: every
statement is classified and access-checked before the engine sees it, and row
filters and column masks are rewritten into the query's parse tree so a CTE or
alias cannot route around them. It registers as a warehouse with nothing to
enter, so BI, the AI Analyst and agents reach it at once.

ETL pipelines, canvas or code. Object storage, databases, change-data-capture,
HTTP APIs, webhooks and the lakehouse in; object storage, databases, Snowflake,
BigQuery, Databricks and the lakehouse out. Cron with real timezone math, a
retry ladder, overlap guards, incremental watermarks, schema-drift policy,
quality gates, per-node preview, version history, alert policy. Runs execute on
the sandboxed runtime and credentials reach process memory only; every
successful load re-crawls its destination so the new tables appear for BI, the
analyst and agents.

Machine learning without leaving the platform. Pick a table and a goal —
predict a column, forecast a series, find groups, find anomalies, recommend
items — and a sandboxed trainer prepares the data, tries several algorithms
under a time budget, and keeps the best with its metrics, leaderboard,
importance and a passport (snapshot, decision id, artifact digest). Every model
is a registry entry with versions and stages: compare them, read the model card,
retrain on a schedule that promotes only when better, watch drift on every
batch, request GPUs. Predictions write back to the lakehouse, reach agents
through ml_predict, draw on BI dashboards through one shared forecaster, and
answer a scoped public API with per-model keys. And the trainer says when a
score could mislead — a leaked feature, a do-nothing baseline, a period nobody
chose — on the version, in the compare view, in the model card and in the
agent's notes.

Every answer has one id. A chat turn, a swarm run or a dashboard refresh
carries a decision id across the model calls, data reads, cost and approvals it
made, plus the lakehouse snapshot it saw. Export it as a signed Answer Passport,
replay it against the original snapshot and against today, and keep it past a
shorter trace window. Replay tells tampering from non-determinism, and a deleted
account can no longer break the audit chain.

A server that uses the machine, and a Kubernetes path that was run. npm start serves through server.mjs with one worker per CPU instead of Vite's
single-threaded preview: on an 8-core host SSR went from 19 to 55 requests per
second. The fully self-hosted Kubernetes path — Supabase as pods, the app, the
Office renderer, the JS sandbox, the lakehouse catalog, the cron job — came up
on a real cluster and then on one that is not the author's laptop, with GPU
placement and egress for the ML platform.

---

Also in this release

• Knowledge from a website, discovered by sitemap or links and re-checked on
  a schedule; Confluence Cloud and Data Center; Azure Blob and ADLS Gen2
  lakes; Jira and Zendesk as datasets, for 29 connectors in all.
• Embeddings from the provider that is connected, including local Ollama
  and vLLM models at their own vector widths, so an air-gapped install can
  search its own documents without an OpenAI key.
• Session restore, contributed by @theniteshdev: a tab that closed or
  crashed mid-work is offered back on the next visit.
• Backups that prove they restore. npm run backup captures the four
  stateful things a self-hosted install cannot regenerate; `npm run restore -- --drill` restores them into a scratch target first.
• Spend the machine you bought. The six compute knobs are settings under
  Admin, and the sizing guide says what to set them to for ETL, the lakehouse
  and ML training.
• Headless runs read as their owner, agents' warehouse queries are audited
  like the UI's, a dashboard's alerts are evaluated as its owner, and a browser
  cannot switch a destructive confirmation back on.
• Deleting a knowledge base asks first.
• Data Prep reads and writes lakehouse tables, which is the way to wrangle a
  training set.
• The dashboard, the tagline and the README admit the data half; seven
  engineering chapters explain how the thing is built; one worked scenario runs
  the whole platform at three systems that disagree about revenue.

---

Upgrading

git pull
npm install
npx supabase db push                                 # 27 migrations
docker compose --profile all up -d --build           # rebuilds the notebook runtime image too

The setup scripts and the Kubernetes installer apply the migrations for you.
The notebook runtime image gained the machine-learning stack, so rebuild it;
--profile all does. The compose file adds a lakehouse-catalog Postgres
service with its own volume, one of the four things npm run backup captures.
Ten environment variables are new and all optional with working defaults;
PROVENANCE_SIGNING_SECRET is the one worth setting, since it signs Answer
Passports. The generated egress allow-list files are no longer tracked.

Known limits

• No SCIM; users arrive through SSO or invitation, groups are managed in IAM.
• One vector store: pgvector in the application database.
• The lakehouse catalog's high availability is yours to provide; compose
  runs one Postgres container.
• The lakehouse is single-node compute: columnar scans, partition pruning and
  spill to disk keep large tables workable, and the honest ceilings are in
  docs/LAKEHOUSE.md.
• The ML platform has no warm real-time endpoint, feature store or distributed
  training; predictions are batch or small payloads through the API and the
  agent tool. The comparison with Databricks and SageMaker is in docs/ML.md.
• Credentialed connectors are verified to validation, not in CI against live
  tenants.

Thanks

@theniteshdev for session restore (#56).