v0.5.1

A bug fix. One unreachable channel no longer silences upgrade announcements
for every other channel.

One dead channel no longer silences the announcement (#89)

The poster behind upgrade announcements (#77) was an unguarded loop over the
configured channels, so the first chat_postMessage to raise aborted the rest.

Found the hard way while cutting v0.5.0 on a live instance: one configured
channel held a stale IM id that Slack no longer resolves, it happened to sort
first out of the set, and the v0.5.0 announcement reached none of the four
healthy channels. The only trace was a single channel_not_found traceback
about a single channel -- nothing said that four other channels had been
skipped.

Now each channel is posted independently and a failure is logged by channel
name, so the gap is visible instead of silent:

announce: could not post to gregory-dm

The other half is when to report failure upward. announce deliberately does
not record a post that failed, so it retries on the next boot -- correct for a
total failure, wrong for a partial one, which would re-announce to the channels
that already have it on every boot. With KeepAlive plus the #66 watchdog,
boots are frequent. So the poster now raises only when no channel accepted the
message.

Worth stating plainly, because it is why this was silence rather than spam: set
iteration order is stable for a given set of ids. The dead channel sorting first
is the only reason nothing was posted at all. Change the channel list and the
same defect posts to the healthy channels, dies on the dead one, records
nothing, and repeats every boot.

Fan-out semantics stay in the poster rather than moving into announce, which
owns the version comparison and the state file and knows only post(text) --
see CLAUDE.md.

Upgrading

git pull
.venv/bin/pip install -r requirements.txt
.venv/bin/python selfcheck.py
deploy/service.sh restart

• Nothing to change. No new config key, no new dependency.
• Worth checking once: if an announcement ever failed to arrive somewhere,
  a stale channel id may be sitting in your shmobster-config.json. Every
  channels entry should resolve via conversations.info, and the bot has to
  be a member of each private one. A DM entry whose id no longer resolves can be
  re-resolved with conversations.open for that user.

v0.5.0

A codex subscription can now be a rung in the waterfall. It is a full
participant -- tool calls work -- so it can sit anywhere in the chain rather
than only at the bottom.

Codex subscription as a waterfall rung (#35)

Every other rung is an api_key HTTP row that LiteLLM dials from config alone.
A codex subscription is not: it authenticates with the ChatGPT OAuth tokens
the codex CLI keeps in ~/.codex/auth.json, and it speaks the Responses API
rather than chat/completions. shmobster/codex_llm.py bridges that as an
in-process litellm.CustomLLM -- there is no second process to run under
launchd.

One keyless row enables it:

{"name": "codex", "model": "codex/chatgpt/gpt-5.5"}

The chatgpt/ segment is load-bearing, not decoration. LiteLLM dispatches on
the model name before it dispatches on the custom provider, so a row of
codex/gpt-5.5 leaves it holding the bare gpt-5.5, recognising that as an
OpenAI model, and quietly billing a platform API key -- the bridge is never
reached, and if OPENAI_API_KEY happens to be valid there is no error at all.
The segment is stripped back off before the request goes out. Written up
generically as
skillz#181.

Two more things the live endpoint dictates, neither of them documented:

• stream: true is mandatory. A non-streaming request is refused outright
  with 400 Stream must be set to true, so the SSE body is read whole and parsed
  even though nothing downstream streams.
• The model must be one your ChatGPT plan allows. gpt-5.5 works;
  gpt-5.1-codex is refused with "not supported when using Codex with a ChatGPT
  account".

Why read codex's token instead of driving the binary. The alternatives were
codex exec or the codex app-server JSON-RPC that OpenClaw's codex extension
speaks. Both keep the token out of our hands, but both hand us an agent: codex
brings its own tool set, sandbox and approval policy, and shmobster already owns
that loop (handler + YOLT gate + channel policy). Nesting a second agent inside
one waterfall rung buys nothing here, and codex exec additionally returns final
text only -- no tool calls.

Token rotation stays yours. auth.json is re-read on every call, so any
rotation the codex CLI performs is picked up without a restart. shmobster
deliberately does not refresh it: that would mean writing back to auth.json
and racing the CLI over a refresh token that may be single-use, and breaking your
actual codex login is worse than one dead rung. Instead the token's own exp is
read per call:

codex: the ChatGPT token expires in ~19h; run `codex login` to rotate it

logged at most hourly so a busy channel does not turn it into spam. Once expired
the rung returns 401, LiteLLM cools it, and the chain falls through. Either way
the fix is codex login -- or just using the codex CLI for anything, which
rotates the file as a side effect.

Upgrading

git pull
.venv/bin/pip install -r requirements.txt
.venv/bin/python selfcheck.py
deploy/service.sh restart

• Nothing is required. No new dependency, and no config change unless you
  want the codex rung.
• To enable it, add the keyless row above to your waterfall. It needs a
  codex login on that machine (codex login); CODEX_HOME is honoured if you
  set it. Verify with codex login status before restarting.
• If you add the row, do not shorten the model string. codex/gpt-5.5
  without the chatgpt/ segment silently routes to OpenAI on your platform key
  instead of your subscription -- see above.
• Keyless rows are now supported generally. A waterfall entry may omit
  api_key entirely; previously an absent key was still passed through as empty.
  Existing rows are unaffected.

v0.4.0

The waterfall now learns. A vendor that reports no budget is parked instead of
being re-dialled on every turn for as long as its cap lasts.

Out-of-budget vendors are parked (#80)

LiteLLM decides cooldowns by HTTP status and cools only 429/401/408/404.
Budget exhaustion is none of those -- Anthropic answers a usage cap with 400,
OpenRouter answers no-credit with 402 -- so a vendor that is dead for weeks
was dialled first on every single message, failed, and only then did the chain
fall through. Fallback worked; nothing learned. Filed upstream as
BerriAI/litellm#37592.

Now a 4xx whose message names a spend problem drops that vendor from the chain
and rebuilds the Router without it:

waterfall: openrouter is out of budget; parked for 3600s
waterfall: rebuilding, chain is now anthropic -> gemini -> requesty

The subtle part, and the reason the first attempt at this was wrong: parking
hangs off litellm's failure callback, not off an except. When a fallback
covers the failure the Router returns that answer and the primary's exception
never propagates, so an except only ever sees the case where the whole chain is
dead -- which is the one case parking cannot help. Verified against live vendors:
OpenRouter genuinely at 402 with gemini answering returns 'ok', raises nothing,
and OpenRouter is parked anyway.

Details worth knowing:

• Status alone does not park. A 400 is also "your request was malformed", and
  parking a vendor for an hour over one bad prompt would be worse than the
  problem. The message has to name money too.
• A stated recovery date wins over the window; Anthropic returns
  You will regain access on YYYY-MM-DD. A date that fails to parse falls back
  to the window rather than being trusted -- a mis-parse could park a vendor for
  a year.
• The vendor is identified by its exact deployment string, which litellm
  passes as litellm_params.metadata.deployment. Not by the model name in the
  callback: litellm strips the provider prefix there, so two vendors reached
  through different routers both report openai/gpt-4o and the wrong one gets
  parked. An ambiguous match parks nothing, because removing a working rung is
  the worse error.
• Parks survive restarts. The watchdog restarts this process routinely; an
  in-memory park would be re-learned every few minutes.
• A vendor returns on its own, with no timer. Expired entries are pruned
  whenever the chain is read, so the first turn after the window rebuilds with
  that vendor back in place.
• A fully parked chain still tries everything. Refusing to answer is worse
  than one wasted call, and every park is ultimately a guess about someone
  else's billing.

Rate limits cool on the first 429 (#51)

allowed_fails=0, so a rate-limited deployment is cooled immediately rather than
after litellm's default fail threshold. A Slack agent's traffic is bursty and
low-volume: by the time a threshold is reached the burst is over, and every
failure in it was a wasted round-trip.

The rest of #51 stays deferred, and num_retries 1 -> 2 is explicitly declined --
an extra retry would double the latency of exactly the problem #80 removes.

Also

announce and the new parking both need state that outlives a restart, so the
state-file handling moved into a shared state module. No behaviour change to
announcements.

Upgrading

git pull
.venv/bin/pip install -r requirements.txt
.venv/bin/python selfcheck.py
deploy/service.sh restart

• New optional config key: budget_park_sec (default 3600) -- how long to
  skip a vendor that reported no budget when it does not say when it will be
  back. 0 disables parking entirely. Nothing to change if the default suits.
• shmobster-state.json gains a parked_vendors key alongside the announced
  version. It is created and maintained automatically; delete the key to
  un-park everything immediately.
• Upgrading from v0.2.0 or earlier: v0.3.0 requires voitta-yolt v1.0.0+, see
  those notes.

v0.3.0

An operations release. The agent now tells you what it is running, refuses to
repeat a credential back to you, and has a fallback chain that does not fail all
at once.

Credential redaction (#72)

Everything the agent says is scrubbed before it leaves the process. run_shell
returns command output verbatim to a channel, and cat, env and printenv are
read-only -- they clear the YOLT gate and run with no approval. This was not
theoretical: a live config was found holding literal keys, where a single cat
would have posted all five to Slack.

Detection reuses voitta-yolt's secret_redact rather than a second pattern list
that drifts from it, so there is one source of truth. On top of it, the thing
YOLT cannot know: this process's own secrets -- Slack tokens, every waterfall
api_key, every per-channel policy env value -- matched by exact value, so a
credential in an unanticipated format is still caught when it is one of ours.

Scrubbing happens at collection, so the model's own context never holds a
credential to repeat later. The reply, the approval cards (which render a parked
command twice), the exception replies and the log formatter are all covered. The
formatter matters more than it looks: a logging.Filter sees only record.msg,
while the traceback from logger.exception is appended by the formatter from
exc_info -- which is exactly where a leaked credential would sit.

It fails loudly, never open: with the redactor unavailable the agent refuses to
start, and it is installed at import before any statement that can log.

Deliberately excluded: a generic "40 characters of base64" rule. It matches a git
SHA, and a redactor that mangles git log gets switched off.

Upgrade announcements (#77)

An instance announces itself in its channels the first time it boots on a version
it has not announced before, linking the release notes. The trigger is a version
change, not a boot -- the watchdog and launchd restart the process routinely, and
none of that is worth a message.

An instance with no recorded version announces too, without claiming an origin it
never had. Staying quiet there would have skipped the first rollout of this
feature, which is the one it was written for.

announce knows nothing about Slack; it takes a post(text) callable, and a
repo-scoped CLAUDE.md now carries the rule for the ingest modes that come
later.

A waterfall that does not fail all at once (#83)

Debugging an outage found all three configured vendors down simultaneously:
Anthropic capped (400), OpenRouter out of credits (402), NVIDIA on a retired
model id (404). A waterfall whose slots fail together is one outage listed three
times.

The example config now ships four independent failure domains --
anthropic -> gemini -> requesty -> openrouter -- with the lower rungs on free
tiers that are rate-limited separately from each other. Observed directly:
google/gemma-4-31b-it returns 429 on OpenRouter's free tier while the same
model answers through Requesty.

The README documents how to find free, tool-capable models on each router, and
the two rules that matter when adding one:

• It must actually do tool calls. This is a tool-calling loop. A model that
  ignores tool schemas does not error -- it answers in prose and the agent looks
  lobotomized rather than broken. Declared support is not enough.
• Pass a timeout. A free endpoint that hangs blocks the whole turn while the
  Slack ack spins.

Two traps are recorded there as well: NVIDIA's /v1/models answers 200 to any
bearer token, so it cannot tell you whether a key is valid; and model ids retire
without notice -- gemini-2.0-flash and meta/llama-3.1-405b-instruct both went
404 during this round of testing.

Known issue

A vendor that is out of budget is re-probed on every turn (#80). LiteLLM decides
cooldowns by HTTP status and cools only 429/401/408/404, so Anthropic's 400
usage-limit and OpenRouter's 402 insufficient-credits never park the deployment.
Fallback still works, so requests are answered -- the primary is just retried
each time before falling through. Filed upstream as BerriAI/litellm#37592.

Upgrading

git pull
.venv/bin/pip install -r requirements.txt
.venv/bin/python selfcheck.py            # prints the build it just checked
deploy/service.sh restart

• voitta-yolt must be v1.0.0 or newer. Redaction imports secret_redact
  from the same tree as the exec classifier, and the agent will not start
  without it. git pull in your voitta-yolt checkout if it is behind.
• New state file: shmobster-state.json (gitignored, path from
  SHMOBSTER_STATE) holds the last announced version. Nothing to create -- it
  is written on first boot.
• Expect one announcement in each configured channel on the first boot after
  this upgrade.
• No config changes required. The new example waterfall is a suggestion; an
  existing shmobster-config.json keeps working. Adopting it needs
  GEMINI_API_KEY and REQUESTY_API_KEY in the launchd environment, not only
  in ~/.bash_profile.

v0.2.0

Two things land here: the agent can load skills, and the project can say what it
is and release itself.

Skills (#74)

Shmobster reads skills in the skillz
format -- a directory holding SKILL.md, YAML frontmatter (name,
description) over a Markdown body. That catalog already targets claude and
codex; shmobster is now a third host reading the same files unchanged, so a
procedure written once reaches the agent that is actually in the channel with
you instead of being restated per agent.

Sources are explicit directories, skills.paths in the config. Order is
precedence: the first path defining a name wins, so a private catalog listed
first shadows the public one, and a shadowed entry is logged at boot rather than
silently dropped.

Delivery is two-stage, because the system prompt is paid on every turn by every
vendor in the waterfall. The prompt carries a menu -- one line per skill, name
plus the first sentence of its description, capped -- and a load_skill(name)
tool pulls a full body only when the model decides one is relevant. The 44-skill
public catalog measures about 7KB standing; inlining full descriptions would be
~25KB, and a search-only tool would go unused, since a model cannot search for
what it does not know exists. With no paths configured there is no menu and no
tool: the feature costs nothing when unused.

reload_skills re-scans without a restart, behind the same trust gate as
set_policy. Reading files is not a mutation, but it changes which instructions
the agent will follow, which is what the gate is for.

A skill is instructions, not permission. Anything a skill tells the agent to run
still passes YOLT and the channel policy, so no skill can widen what a channel
can do. Known edge: if a turn starts with an empty menu, reload_skills fills
the index immediately but the load_skill tool is not offered until the next
turn.

Versioning and releases (#76)

An instance can now state what it is. shmobster.__version__ is the anchor and
build() reports <version>+<short-sha> -- logged at boot, printed by
selfcheck, and in the system prompt, so asking an instance which build it is
gets an answer rather than a guess. Instances run as services on separate
machines that pull on their own schedule, and between tags the sha is the only
thing that tells two of them apart.

Releases cut themselves. Feature PRs leave the version alone; a release is its
own commit bumping __version__, and .github/workflows/release.yml turns a
version with no matching tag into the tag plus this release. Notes come from
docs/release-notes/vX.Y.Z.md when the file exists and from merged PR titles
otherwise, so prose never blocks a release.

The posture is adapted from skillz, minus what only serves a plugin catalog.
There the version is the install cache key, so a PR shipping code without a
bump silently freezes every install -- hence their per-PR bump gate. Nothing
here is keyed on the version, so that gate would only churn numbers.

This repo also has CI for the first time: selfcheck.py, a parse check on the
example configs every new operator copies, and a structural sensitive-term gate
ported from skillz (token and key shapes, account ids, private IPs, internal
domains). The name-wordlist half of that gate stays off CI deliberately -- it
reads a private out-of-repo file, and an Actions secret would not reach fork
PRs, so the job would report green without having run the check.

Fixed

python selfcheck.py had been failing since v0.1.0 for anyone without the live
keys exported -- the case README step 5 calls an offline sanity check. The
example config's own _comment spelled a literal dollar-brace VAR, so the
${VAR} interpolator read the documentation as a reference and killed startup
on an unset variable named VAR. The comment no longer spells one, and
selfcheck stubs the names the example refers to with placeholders.

Upgrading

Skills are opt-in and nothing else changes behavior, but the dependency and the
config key are new:

git pull
.venv/bin/pip install -r requirements.txt   # adds pyyaml
.venv/bin/python selfcheck.py               # prints the build it just checked
deploy/service.sh restart                   # pick up the new code under launchd

• New dependency: pyyaml, for skill frontmatter. An instance that skips
  the pip install will fail to import.
• New config key: skills.paths, a list of directories of <name>/SKILL.md.
  Omit it to run exactly as before.
• No policy-file changes; shmobster-policies.json is untouched.

v0.1.0 - first tagged release

First tagged release. shmobster has been running in production under launchd for
a while; this puts a version on it.

A standalone Slack agent built bottom-up. Two requirements force it to exist:
multi-vendor API waterfall (rate limits) and multi-user Slack authz
(collaborators). Everything else is borrowed or transplanted, and credited.

What is in it

The loop - Slack Socket Mode to a tool-calling loop and back. Thread-as-session
context, configurable tool-step cap with a near-limit warning, and a final
tools-less answer when the cap is hit so the loop cannot dead-end. Mentions are
acked with a reaction so long tasks do not look silent, and duplicate mention
events are deduped.

Authz = f(user, channel) - per-channel policy covering working directory,
cloud profile and repo scope, held in a separate gitignored file with per-channel
credentials. Trusted users can reconfigure live via a set_policy tool behind a
trust gate. Policy re-resolves mid-turn, ~ expands, and a best-effort exclude
list marks paths off-limits under the working directory.

Mutating commands need approval - an approval flow with Slack-native
approve/deny buttons. A model-initiated policy change that gets refused posts a
refusal that does not blame the user.

Operational - a liveness watchdog that restarts when Socket Mode goes deaf,
service.sh reloading the plist rather than kickstarting it, identity derived
from configured label rather than a hardcoded name, and history labelled by real
speaker (self / sibling agent / human). Reads image and text attachments on a
mention.

Secrets - config takes ${VAR} interpolation so credentials are referenced by
name, never written as literals, and values are never logged (#73).

Credential hygiene, learned the hard way

That last item came out of a credential-leak audit run across our agent machines.
The transferable findings are written up as skills in
voitta-ai/skillz - agent-session-credential-audit
and agent-credential-leak-surfaces (latest catalog release: v1.9.0; an extension
covering the items below is open as PR #175). The short version, because it
changes how an agent should be built:

• Agent sessions are permanent plaintext records. A secret that appears in a
  session for one second is on disk indefinitely, including the full stdout of
  every command. Hence ${VAR} references and never logging values.
• A liveness probe lies in at least five distinct ways. Same status for valid
  and invalid keys; a catalogue route that answers 200 to unauthenticated
  callers; a live-but-spend-capped key answering 400; non-2xx for both states;
  and an endpoint that rejects an unrelated field before evaluating auth, so it
  answers identically for every key. Always probe against a known-bad control, and
  discriminate on the response body rather than the status line.
• Fingerprint the exact byte span. Referring to secrets by a hash prefix is
  right, but a pattern with a bounded quantifier records the hash of a truncated
  span, and the entry can then never match the value it was meant to track - a
  scrub reports clean while the secret is still there.
• Verify a cleanup by fingerprint, never by pattern count. Documentation,
  examples and test fixtures share the pattern permanently, so counting matches
  makes a successful run look failed.

Open question for anyone building on this: should shmobster observe skills?

It does not today - there is no skill-loading mechanism in the codebase. The
skillz catalog currently targets claude and codex as hosts, and a third Slack-
native host is a plausible next step: same authored-once instructions, delivered
to the agent that is actually in the channel with you.

Open questions if we do it: which subset applies when the agent is multi-user
(skills are authored for a single operator), how per-channel authz interacts with
a skill that assumes a working directory, and whether skills load at startup or
resolve per turn like policy does. Opinions welcome on the issue tracker.

Credits

The workspace spine is transplanted from OpenClaw
(MIT), credited in #47. MIT licensed.