EverOS 1.2.3

Background maintenance that fails loudly instead of quietly. A soak run on
1.2.2 found a table that had stopped reclaiming disk for 100 minutes while
/health stayed green — nothing had failed, a call had simply never returned,
and every signal was built from failure counters. Auditing for that shape turned
up six more places it could happen: reads with no deadline (which stop the whole
md to LanceDB projection, not just one table), background loops that die
permanently on one exception with no log at all, an alert counter reset by the
remediation it triggers. All of them are now bounded, and a stall that does
happen names the table it happened to. Alongside that, agent-skill extraction is
rescued from a retry-then-dead-letter loop, keyword search no longer returns 500
during an index rebuild, and the maintenance cadences moved into settings.

Fixed

• Agent skill extraction is no longer stuck in a retry-then-dead-letter
  loop. Target case data now travels on SkillClusterUpdated and existing
  skills for the cluster are read from markdown (strong-consistency), so the
  strategy never races cascade indexing. Prior to this fix, running a fresh
  agent trajectory produced zero SKILL.md files — .skills/ did not exist.
  The related stale-index clobber is fully closed only for clusters at or
  below MAX_SKILLS_IN_PROMPT (10). Above it, markdown still supplies the
  candidate set but LanceDB orders it, and the skill a lagging index omits is
  by definition the one written most recently — the one most likely to need
  update — so it can be ranked out of the prompt and re-added instead. The
  window is narrow (it needs a cluster over 10 skills and an index that has
  not caught up) and the consequence is the pre-existing full-replace, not a
  new failure mode.

• POST /api/v2/ome/trigger no longer masks strategy state. The status
  field now distinguishes not_dispatched (all dispatch gates rejected the
  strategy — usually a missing "force": true) from ok (dispatched and
  settled). The new runs field surfaces dead-lettered strategy runs that
  were previously invisible to the caller. If your client matches
  status exhaustively (Python Literal, TypeScript union), add a
  not_dispatched branch.

• Agentic search on agent memory now uses the skill-shaped rerank
  passage. The cross-encoder previously saw only the raw description
  field instead of the name + description + skill instruction triple that
  the HYBRID lane uses. A skill with empty description (a legal everalgo
  output — see everalgo/agent_memory/skill_ops.py:294) no longer causes
  HTTP 500 during the LLM sufficiency check.

• OME strategy retries now back off between attempts. A retry-class error
  (e.g. waiting on eventually-consistent state) previously exhausted its
  max_retries budget in milliseconds; the loop now sleeps
  min(base * 2**(attempt-1), cap) plus up to jitter seconds
  (defaults: 1s base / 10s cap / 0.5s jitter — code-only defaults,
  not currently exposed via everos.toml or ome.toml). engine_sem is
  now held per attempt rather than across the whole retry chain, so the
  backoff sleep does not occupy a concurrency slot. The cap bounds
  concurrent strategy work — LLM calls, embeddings, storage IO — and a
  coroutine waiting to retry consumes none of it; holding the slot would
  have turned a partial outage into a total stall, since enough
  simultaneously-failing runs park every one of the
  max_concurrent_runs slots in asyncio.sleep and starve strategies that
  would have succeeded. Backpressure on failing work is intended;
  backpressure on everything else is not.

• Path-traversal hardening for LLM-generated agent-skill names (CWE-22).
  AgentSkillFrontmatter.name comes straight from LLM output
  (extract_agent_skill) and was concatenated unsanitized into the
  skills/skill_<name>/ directory segment on both the write and read
  paths; given a sufficiently long ../ prefix, the write target could
  escape the memory root. This is the same class of defect previously
  fixed for knowledge-upload titles/categories (see knowledge_writer.py
  in an earlier 1.2.x). The sanitizer is now a single shared helper
  (everos.core.persistence.markdown.sanitize_dirname) used by both
  KnowledgeWriter and the new SkillPathMixin.skill_dir_name() /
  sanitize_skill_name(), instead of two independently maintained copies.
  extract_agent_skill now sanitizes the LLM-emitted name before
  constructing AgentSkillFrontmatter, so AgentSkillFrontmatter.name
  and the LanceDB agent_skill primary key now hold the sanitized name
  (spaces become _, characters outside [\w\-.] are dropped, capped at
  50 chars), not the raw LLM output — a user-visible change for anything
  that reads a skill's name field expecting the verbatim LLM string.
  AgentSkillFrontmatter.name also gained a validator rejecting a name
  containing a path separator, or being exactly .., so a hand-edited
  SKILL.md that bypasses the writer's sanitization is caught on read
  rather than silently relocated (the substring form, e.g. a name that
  merely contains .., is deliberately allowed — sanitized output can
  legitimately contain runs of literal dots). sanitize_dirname itself
  falls back (not just on an empty result, but also on . or ..) so a
  short input that is itself a sanitizer fixpoint — e.g. "../" sanitizes
  to ".." verbatim without this fallback — cannot resolve to the same
  directory or its parent; this closes both the agent-skill case and an
  equivalent one-level escape on the knowledge-upload path, which has no
  skill_-style prefix protecting its sanitized segment. No data
  migration is needed for agent skills: extraction has never
  successfully produced a SKILL.md before this release (see the
  cascade-lag fix above), so there is no legacy skill corpus whose
  directory names would change. Knowledge documents do have a
  pre-existing corpus, and two inputs resolve to a different directory
  than before: a decomposed (NFD) topic or category now keeps its
  combining marks ("Résumé" no longer degrades to "Resume") because
  the shared helper NFC-normalizes first, and a topic or category of
  exactly . or .. now falls back instead of resolving onto the
  parent directory. Precomposed input — including CJK — is unaffected;
  the character class is unchanged from the previous private copy.
  Sanitizing is lossy: skills whose raw names
  differ only in characters the sanitizer drops or replaces (e.g.
  "fix django" vs. "fix_django") now share one SKILL.md, and so do
  names differing only in a combining mark regardless of script (e.g.
  Devanagari "किताब" vs. "कताब" — a combining mark alone is not \w
  and is stripped either way; same for Thai tone marks, Hebrew niqqud,
  Arabic harakat). The later write wins — the earlier skill's
  source_case_ids, maturity_score, and body are silently lost, not
  merged. Case is not folded, so "Fix Django" and "fix django" stay
  two distinct sanitized names — two LanceDB rows, but one directory on a
  case-insensitive filesystem (macOS APFS and Windows NTFS defaults),
  where the index then advertises a name whose content was overwritten.
  This is accepted for now rather than mitigated: detecting a collision
  and raising would reintroduce the dead-letter DoS the sanitizer was
  built to avoid, and a disambiguating suffix — the workable option —
  needs a collision probe plus a case-folding rule, so it is deferred to
  a deliberate pass rather than added here.

• A renamed skill no longer leaves an orphan directory that pollutes the
  next extraction. everalgo treats a name change as a first-class update
  (skill_ops._apply_update preserves prior.id while swapping the name),
  so the emitted skill was written to a new skill_<new_name>/ while the
  old directory survived carrying the same cluster_id. Because existing
  skills are now read from markdown rather than LanceDB, that orphan did
  not merely sit on disk — it came back in the next run's
  existing_relevant_skills as a duplicate of a skill the LLM had already
  renamed, feeding exactly the add-instead-of-update full-replace
  clobber this release set out to close, once more per rename. The old
  directory is now reaped after the new one is written, keyed on the
  skill's id (the only thing that survives a rename; a fresh add mints
  a uuid and can never match). A prior name that another skill in the same
  batch just claimed is never deleted.

• extract_agent_skill retire ops are documented as unimplemented rather
  than silently mispersisted. AgentSkillExtractor.aextract returns a
  flat list with no op discriminator, so a retirement arrives as an
  ordinary skill with confidence < retire_confidence and was written back
  like any other — staying in markdown, in the next prompt, and in search.
  The behaviour is unchanged; the module docstring no longer claims retire
  is handled. Honouring it is a design decision (delete the directory, or
  add a retired flag that the enumeration, cascade, and search all
  filter on) deferred to its own change.

• reference_name and script_filename are sanitized. Both are
  appended after the skill_<name> segment, so skill_dir_name never
  covered them; they now go through the same sanitize_dirname primitive
  on both the reader and the writer. No caller in src/ reaches them
  today, so nothing was exploitable — this closes the gap before
  progressive disclosure wires them up.

• A single unparseable SKILL.md no longer disables skill extraction
  for its whole cluster. AgentSkillReader.list_by_cluster propagated
  any frontmatter ValidationError, which aborted the enumeration that
  feeds extract_agent_skill its existing skills — so one hand-edited
  file (or, after a future schema revision adds a required field, every
  existing file at once) dead-lettered that cluster's extraction on every
  run. Offending files are now logged and skipped. read_main still
  raises, since a caller naming one specific skill needs to hear about
  corruption rather than receive the None that already means "not
  created yet".
  merged. This is accepted, not mitigated, on two grounds: a
  disambiguating suffix would break the name ≡ directory-suffix
  identity the reader/writer relies on, and detecting a collision and
  raising would reintroduce the dead-letter DoS the sanitizer was built
  to avoid.
  "fix django" vs. "fix_django") now share one SKILL.md, and the
  later write wins — the earlier skill's source_case_ids,
  maturity_score, and body are silently lost, not merged. This is
  accepted, not mitigated: the LLM's add/update decision is keyed on the
  name it sees, so a collision usually reads as an intended update
  anyway.
  under the new sanitizer.
  KnowledgeWriter and the new SkillPathMixin.skill_dir_name(), instead
  of two independently maintained copies. AgentSkillFrontmatter.name
  also gained a validator rejecting path separators / .. so a
  hand-edited SKILL.md is caught on read rather than silently
  relocated. No data migration: agent-skill extraction has never
  successfully produced a SKILL.md before this release (see the
  cascade-lag fix above), so there is no legacy skill corpus whose
  directory names would change under the new sanitizer.

• Reads now carry a deadline (count / get_by_id / find_where /
  find_where_paginated / search). The write-side deadline work skipped them
  on the reasoning that a read takes no lock and so blocks no writer — true, but
  the cascade drain loop reads on every batch and advances strictly one batch at
  a time, so a read that never returns stops the whole md → LanceDB
  projection: claimed rows stay processing forever, nothing new is indexed,
  and /health still reports healthy because a hang raises nothing. Budget 60s
  (~1000x the measured 62ms flat scan over 117k rows); expiry raises the
  retryable VectorStoreBusyError.

• Background loops are supervised. The drain / heartbeat / rebuild loops were
  plain create_task coroutines: one uncaught exception ended that loop
  permanently, and because the worker holds a strong reference to the task the
  interpreter never printed "Task exception was never retrieved" either — the
  loop's job simply stopped happening with zero output. Each now runs under a
  supervisor that logs and restarts with escalating backoff (5s / 15s / 45s),
  then asks the process to exit via SIGTERM so a restarting supervisor
  (systemd, Docker, k8s) can recover it. The restart budget counts consecutive
  quick crashes, not crashes over the process lifetime — a body that ran 60s+
  before raising starts a fresh incident, so independent transients days apart
  cannot pool into a process exit (same windowed counting as systemd's
  StartLimitIntervalSec). A done-callback covers the case the supervisor
  itself ends unexpectedly.

• The optimize-failure alert is reachable again. The fallback rebuild reset
  the same counter the health verdict reads, so a table failing 100% of the time
  cycled 1..5 → 0 → 1.. and the threshold value existed only during the
  sub-second rebuild — roughly 1% observable against a 30s scrape, so
  cascade.healthy stayed green while the table never reclaimed a version. The
  rate limiter now lives in its own counter (failures_since_fallback); only a
  successful optimize clears the alert streak. Same shape as the cross-kind
  max() masking bug: a remediation path refreshing the signal meant to report
  it.

• A rebuild no longer leaves the column without an FTS index. rebuild_indexes
  dropped every index and recreated it, on the assumption that LanceDB falls
  back to a brute-force scan meanwhile. That holds for vector search and not
  for FTS: with no inverted index a BM25 query raises Cannot perform full text search unless an INVERTED index has been created, and since the recall legs
  are gathered without return_exceptions, one failing leg 500s the whole
  search request. Now uses create_index(replace=True), which swaps atomically
  — measured 0 failures across 49 queries spanning 3 replaces, versus 55
  failures for the same test against drop-then-create — and collapses the live
  index fragment set exactly as before (7 index files back to 4).

• The empty-index-dir sweep is bounded by lance's own threshold. lance's
  cleanup.rs unlinks a superseded index's files but never its directory — it
  contains no rmdir at all, which is structural rather than an oversight: it
  targets object stores, where paths are flat keys and an empty directory does
  not exist. Only a local filesystem materialises them, and a soak run reached
  13061 dirs, 98% empty. everos sweeps them, now with three independent
  guarantees instead of a self-chosen age: rmdir cannot delete a non-empty
  directory (the kernel refuses it, so no file can be lost and there is no
  check-then-act window), live index UUIDs are excluded via list_indices(),
  and anything else must outlive UNVERIFIED_THRESHOLD_DAYS = 7 — lance's own
  bound for deciding an unreferenced index UUID is dead rather than mid-build.
  The previous 300s was our invention, which is what made it indefensible.
  Two consequences worth knowing: the age gate reads the dir's mtime, which
  POSIX bumps when lance's cleanup empties it, so the effective reclaim horizon
  is up to ~14 days (file wait + age gate) and the ceiling-load steady state is
  ~1.8M dirs / ~7GB; and a sweep that blows its 60s deadline is swallowed
  inside prune() — the cleanup commit already succeeded, so escaping would
  bill a prune "failure" (feeding the fallback-rebuild threshold) and stall the
  prune-staleness clock for a cleanup stall that did not happen.

• The optimize runner's wait on an in-flight rebuild is bounded too. The two
  maintenance jobs park on each other — whichever arrives second waits — so an
  unbounded wait on this side is the same hazard as the one already fixed on
  the rebuild side, just seen from the other end: the kind's task slot stays
  occupied, _schedule_optimize keeps short-circuiting on it, and that table
  quietly stops being pruned. It was left open on the argument that
  rebuild_indexes carries its own 300s deadline, which covers its critical
  section but not the task's dispatch and teardown, so the transitive bound was
  never real. Now bounded at 180s, logging
  cascade_lancedb_optimize_skipped_rebuild_unfinished and skipping the beat
  rather than compacting under a live rebuild — the two commit on the same
  manifest, which is what the wait exists to prevent.

• A rebuild that loses a commit race is retried instead of waiting out the
  full 12h cadence. Lance labels the conflict Retryable and it is: a
  concurrent writer in another process won the manifest, nothing is wrong with
  the table. Retries are scheduled on the kind (10min / 30min / 3h) rather than
  slept through, so the other kinds in the sweep are not parked behind the
  backoff. A soak run at a 600s cadence hit 3 conflicts in 119 attempts, all
  while a concurrent CLI storm was running.

• The index-rebuild sweep can no longer park forever waiting on the
  optimize runner. That wait had no deadline, and the runner's loop condition is
  "keep going while there is unindexed data" — which under sustained writes is
  never, since the drain loop re-raises the flag every second against a 10s
  cooldown. Now bounded at 180s, logging
  cascade_lancedb_rebuild_skipped_optimize_unfinished and skipping the sweep
  rather than dropping indices under a live optimize. This makes the stall
  visible, not absent: under sustained ingest every sweep still times out, so
  active index-UUID / FTS part_N growth stays unbounded there. The functional
  fix requires the optimize runner to yield when a rebuild is pending, which
  changes the optimize/rebuild mutual-exclusion contract and needs its own
  validation — the rebuild cadence is 12h, longer than any soak run so far, so
  the periodic sweep has never been exercised under load.

• The memory-root lock wait is bounded and visible. Acquisition polls with
  LOCK_NB instead of blocking inside a worker thread: a blocking flock could
  not be bounded or cancelled — cancelling the awaiting coroutine left the
  thread to acquire the lock later with nobody to release it. The wait itself is
  by design (the second process is supposed to wait, then find the migration
  already done), but it now logs memory_root_lock_waiting and gives up after
  timeout_seconds (default 30min) instead of leaving a server startup looking
  like a hang whose last message is lifespan_provider_startup name=lancedb.
  The default sits an order of magnitude above the worst legitimate hold (a
  large migration is minutes) on purpose: the wait is already visible from the
  first poll, and against the one case the bound exists for — a holder that is
  alive but wedged — giving up at 5 minutes buys nothing over 30, while a bound
  near the legitimate hold turns a slow migration into startup crashes for
  every waiting process.

Added

• OfflineEngine.trigger_manual now returns
  tuple[BaseEvent, list[tuple[StrategyMeta, str]]] instead of None,
  enabling the dispatched/runs fields below.
• TriggerResponse gains dispatched: int and runs: list[RunSummary].
• OMEConfig gains retry_backoff_base_seconds, retry_backoff_cap_seconds,
  and retry_jitter_seconds for the retry-loop sleep.
• AgentSkillReader.list_by_cluster() enumerates the cluster's SKILL.md
  files from markdown (strong-consistency existence check).
• [cascade] settings section — the four maintenance cadences
  (optimize_heartbeat_seconds, optimize_prune_interval_seconds,
  optimize_prune_retention_seconds, optimize_rebuild_interval_seconds) are
  now configurable. They were already constructor arguments on CascadeWorker,
  but CascadeConfig did not carry them and no production path passed one, so
  the defaults were unreachable — which is why the 12h rebuild sweep could not
  be exercised by any soak run shorter than half a day. The deadlines that
  bound a hung call are deliberately not exposed: they are hang-catchers
  sized from measured durations, where too low manufactures failures on a
  healthy table and too high leaves a wedged one invisible for longer. Note
  optimize_prune_retention_seconds has a second effect worth reading before
  tuning — it also decides how long index files keep a manifest naming them,
  and below LanceDB's 7-day unverified window they then wait out the full 7
  days.

Changed

• extract_foresight now ships disabled (enabled=False). Not because
  it is broken — the crash below is fixed — but because it is one LLM call
  per sender per memcell whose output nothing in EverOS reads today: no
  search route surfaces foresights and no prompt slot consumes them. Until
  something does, running it by default spends tokens on write-only data.
  Re-enable per install in ome.toml (hot-reloaded, no restart):

  [strategies.extract_foresight]
  enabled = true

  Editing default_ome.toml alone would not have reached existing installs
  — everos init does not overwrite an existing ~/.everos/ome.toml — so
  the code default is what changed.

• extract_foresight no longer crashes on a memcell containing tool
  calls. The sender scan read m.role off every item, but only
  ChatMessage carries it (ToolCallRequest has sender_id without it,
  ToolCallResult has neither), so the first tool call raised
  AttributeError — before any sender was resolved. The strategy was
  correct on plain user chat and dead-lettered every time on agent
  trajectories. everalgo explicitly contracts for the mixed case
  (user_memory/_render.chat_messages: the caller need not pre-filter),
  and every other user-memory extractor gets that for free by delegating;
  this was the one place the filter was hand-rolled. The scan now tests
  isinstance(m, ChatMessage), so a pure agent trajectory yields no senders
  and returns without an LLM call. Matters even with the strategy off by
  default: it is what makes the opt-in above actually usable.

• SkillClusterUpdated carries the case's 1024-dim embedding, growing the
  OME run_record table. The event payload is persisted verbatim in
  run_record.event_payload (and in the APScheduler jobstore while a job is
  queued), so a skill_cluster_updated record goes from roughly 0.8 KB to
  14 KB. At the default max_records_per_strategy = 1000 ring buffer that is
  ~14 MB for this one strategy instead of ~0.8 MB. Operators sizing
  ~/.everos/.index/sqlite/ome.db should expect this. The vector is only
  read when a cluster holds more skills than MAX_SKILLS_IN_PROMPT, so it
  usually rides along unused; trimming it from the persisted copy is not a
  local change, because crash recovery replays event_payload to rebuild the
  event and a trimmed payload would silently take the recovered run down a
  different branch than the original. Tracked as a follow-up.

• cascade_lancedb_optimize_conflict now records pruned — which
  maintenance beat lost the commit race. Lance labels both beats' commit the
  same way (This Rewrite transaction was preempted by concurrent transaction …), so the message alone cannot separate a free loss from a costly one: a
  lost light beat retries ~10s later, while a lost heavy beat means
  that table skipped a whole prune cadence and its superseded files stay on
  disk. Attributing index-dir growth previously meant back-inferring which
  beats were heavy from the 300s cadence. Log level (debug) and the
  benign-conflict semantics are unchanged — this adds one field.

Upgrade

pip install --upgrade everos   # or: uv sync

Two behaviour changes to know about before upgrading.

extract_foresight now ships disabled — a deployment relying on foresight
entries must set enabled = true for it in ome.toml.

SkillClusterUpdated now carries the case's 1024-dim embedding, so a
skill_cluster_updated row in the OME run_record table grows from ~0.8 KB to
~14 KB — about 14 MB for that strategy's default 1000-record ring buffer,
against ~0.8 MB before. Anyone sizing ~/.everos/.index/sqlite/ome.db should
account for it.

Nothing else needs action: the new [cascade] section is optional and a config
written by an earlier version falls back to the same defaults (verified on a
clean install).

One deployment note. When a supervised background loop crashes repeatedly and
exhausts its restart budget, the worker now sends itself SIGTERM rather than
serving on with a dead projection pipeline. That assumes something restarts the
process — systemd Restart=always, Docker restart: unless-stopped, a k8s
Deployment. Without one the process simply stops, which is still preferable to a
server answering searches from a silently frozen index, but it is worth knowing
before the first time it happens.

Full changelog: v1.2.2...v1.2.3