Skip to content

v1.4.2

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 27 Sep 01:41
· 6 commits to main since this release
v1.4.2
7ac54c6

Release Notes 1.4.2

Released: 2026-09-27

This release makes embedded SurrealDB safe to run, adds Amazon Bedrock as a first-class model and embedding provider, and teaches Sibyl to re-embed its own vectors when the embedding model changes. It also collapses machine onboarding into a single sibyl setup line and deletes a large amount of pre-SurrealDB and pre-1.0 residue: 546 files touched, with more lines removed than added.

Highlights

Amazon Bedrock for Claude and Cohere embeddings

A new bedrock provider serves Claude models through InvokeModel or the bedrock-mantle Messages API, and Cohere cohere.embed-v4:0 for embeddings. Auth works two ways: a bearer token (SIBYL_BEDROCK_API_KEY or AWS_BEARER_TOKEN_BEDROCK) or SigV4 through the standard boto3 credential chain, so IRSA, Pod Identity, SSO profiles, and instance roles all work without storing a key. bedrock is now accepted for llm_provider, embedding_provider, and graph_embedding_provider, and never gets persisted to the settings database because it reads only from the environment and credential chain.

Vectors re-embed themselves when the model changes

Every vector now carries an embedding_metadata stamp recording its provider, model, and dimensions. A new embedding_sweep lifecycle job walks the graph plane (entity.name_embedding, relates_to.fact_embedding) and the document chunk plane, re-embedding anything whose stamp no longer matches the configured model. Passes are budgeted (SIBYL_EMBEDDING_SWEEP_BUDGET_SECONDS, default 45s), resumable through a persisted cursor, and lease-guarded per organization so restarts and rolling deploys do not double-work. sibyld db reembed --org-id <uuid> --plane graph|documents|all forces a sweep, with --dry-run to see counts first.

Embedded SurrealDB is now safe to run

Five fixes close the gaps that made sibyld serve --embedded lose data. Previously each DedicatedSurrealClient opened its own AsyncSurreal engine on the same SurrealKV directory, so auth, content, and graph clients appended to one log with private indexes and read each other's bytes back as corruption (Invalid revision 109 for type Value on GET /auth/me after signup). Separately, the graph client read the core library's import-time default of memory://, so every memory written through the local daemon vanished on restart.

One line connects a machine

sibyl setup <url> probes the server, creates or reuses a context, signs in through the device flow, installs the skill pack, and registers the Claude Code SessionStart hook. Two new public endpoints back it: GET /setup/connect returns per-OS install-and-setup lines, and GET /setup/agent.md returns a markdown document a coding agent can follow unassisted. The web Connect panel shows the same line with a macOS/Linux/Windows switcher.

Bounded dream-cycle spend

SIBYL_CONSOLIDATION_RUN_MAX_TOKENS (default 10_000_000) caps what one reflection run may reserve across all its proposals, critiques, corrections, and individual passes. Budgets now reserve an estimate before dispatch and settle against actual usage afterward, refunding overestimates and charging overages, so a long run no longer strands reserved tokens.

Amazon Bedrock

  • New sibyl-core[bedrock] extra pulls anthropic[bedrock], boto3, and botocore.
  • SIBYL_BEDROCK_REGION (falling back to AWS_REGION, then AWS_DEFAULT_REGION) is required; startup refuses with a message naming both variables.
  • SIBYL_BEDROCK_API selects invoke (default) or mantle. SIBYL_BEDROCK_INFERENCE_SCOPE picks the cross-region inference profile prefix: us (default), eu, apac, jp, au, ca, us-gov, global, or regional.
  • Setting both a Bedrock API key and SIBYL_BEDROCK_PROFILE is refused rather than silently resolved.
  • Cohere Embed v4 dimensions are validated at boot against COHERE_EMBED_V4_DIMENSIONS (256, 512, 1024, 1536), so a bad value fails fast instead of at first embed.
  • BedrockEmbeddingError carries status_code and error_type, mapping ThrottlingException, ServiceUnavailableException, ModelNotReadyException, and InternalServerException into the retry path.
  • Model IDs are accepted bare (claude-sonnet-4-5), geo-prefixed (us.claude-sonnet-4-5), or as application inference profile and foundation-model ARNs.

Embeddings and recall

  • Graph schema v31 and content schema v47 add an embedding_states table holding per-plane legacy verdicts, pass receipts, walk cursors, and leases, unique on (organization_id, plane).
  • Legacy vectors written before stamping existed get one durable verdict per plane, decided from evidence photographed at upgrade time so it cannot be forged afterward. SIBYL_EMBEDDING_LEGACY_VECTORS accepts auto (default), adopt, or reembed.
  • The raw vector recall lane now filters stored captures by embedding model inside the HNSW bracket. Before this, a provider switch left old-model vectors scoring against new-model queries, producing cosine noise that still ranked in top-k. Excluded rows remain reachable through the raw_fulltext lane.
  • Bedrock model names are normalized to their trailing path segment when matching, so cohere.embed-v4:0, us.cohere.embed-v4:0, and the ARN form all compare equal.
  • Content schema v48 adds raw_embedding_refusals, separating refused captures (provider rejected the input, permanent) from deferred ones (model failed, retried on a 1h/6h/1d backoff) so a poison row no longer stalls the repair walk.
  • Raw embedding repair fences its writes on the stored stamp: a re-embed fails rather than overwriting a stamp another pass changed first, which matters during rolling deploys.
  • idx_raw_captures_org_uuid on (organization_id, uuid) speeds repeated paging through large capture tables.
  • Lane readiness skips the vector lane when under 5% of captures sit in the query model, reporting vector_lane_model_switched or vector_lane_model_sparse and letting fulltext carry the query instead of paying a full-table scan.
  • sibyl debug status reports per-plane verdict, progress counts (checked, re-embedded, adopted, pending, skipped, refused, deferred), and readiness.

Embedded SurrealDB

  • One engine per directory. File-backed embedded URLs now lease a single process-wide engine per data directory. The engine session never selects a namespace; each client prefixes its own USE statement, which scopes only that execution. A new test runs eight namespaces concurrently on one real engine (including a batch with a mid-statement sleep and a transaction with a pause between writes) and fails on the pre-shared-engine code with the corrupted-read InternalError.
  • Graph persistence. reload_settings_from_env now also refreshes the core library config through a new reload_core_config_from_env, so sibyld serve --embedded points the graph client at the SurrealKV store instead of the import-time memory:// default. A test drives a real sibyld serve --embedded process through signup, /auth/me, /orgs, and an entity write, restarts it, and checks the entity is still there.
  • Lost concurrent writes. The embedded engine words write-write conflicts differently than the server: the server says Transaction conflict: ... This transaction can be retried, while surrealdb-core 2.3 says Failed to commit transaction due to a read or write conflict. This transaction can be retried. is_retryable_transaction_conflict() now matches both markers, each still paired with the retry suffix, so conflicting writes retry instead of surfacing as InternalError. The caveat: that engine still misses some write-write conflicts entirely, even inside BEGIN/COMMIT, so two connections can both commit a read-modify-write from the same stale read. The actual safeguard is the hard clamp of every embedded client to a single connection, which is what keeps the compare-and-set fences in dream checkpoints, source states, revisions, schema leases, and write witnesses correct. A strict xfail test in test_embedded_shared_engine.py pins the engine bug, so an SDK bump that fixes it will XPASS and fail the suite, flagging the moment to revisit the clamp.
  • Missing scheme. Enforcing that clamp surfaced a gap: surrealkv+versioned:// shares an engine but was absent from the embedded scheme lists, so it got a pool of four connections. It is now listed in all four places (dedicated client, schema bootstrap, raw recall, and the API's effective pool size).
  • Cancellation safety. close() and warm_pool() both drained the connection pool outside their try, so a cancellation mid-drain permanently dropped the slots already taken, and every later close waited forever. Both now drain inside the try and finish under _finish_despite_cancellation, re-raising the cancellation afterward. Shutdown and the end of a CLI command are exactly where that cancellation happens.

Security and operations

  • Per-client login buckets behind a proxy. SIBYL_FORWARDED_ALLOW_IPS names trusted proxy IPs and CIDR ranges, defaulting to loopback only (127.0.0.1, ::1). Without it, every user behind one ingress shared a single 5-logins-per-minute bucket, session and audit records resolved to the proxy address, and the break-glass IP allowlist was bypassed by any traffic from the proxy. Every launcher (sibyld, sibyl serve --reload, sibyl up, the moon dev stack) passes the same list to uvicorn. Entries are validated strictly at startup because uvicorn silently ignores malformed ones: 10.20.3.4/16 is rejected with a suggestion to use 10.20.0.0/16.
  • No URL secrets in errors or logs. surreal_url_scheme split on the first ://, so a URL with no scheme but a later :// took everything before it as the scheme. For Admin:Hunter2@host:8000/rpc?next=http://x the refusal message echoed the password, lowercased, into the error and any log carrying it. url_schemes.py now owns every split (split_surreal_url) and accepts only an RFC 3986 scheme token, and adds redact_surreal_url, surreal_http_base_url, and surreal_url_credentials. The SurrealConnectTimeout message, surreal_connect_failed logs, the sibyld service check, the admin health payload (visible through sibyl debug status --json), and the CLI migrate POST target all carry the redacted form now.
  • Settings errors no longer echo secrets. A failed settings validation used to print part of the settings input, provider keys included. Both settings models now set hide_input_in_errors, and URL errors name only the scheme and the fix.
  • Project scope no longer widens silently. Recall, briefs, and context packs resolve one project in order: --project, then the directory link, then the context default, then the legacy defaults.project. Reading everything requires explicit --all on the CLI or all_projects=True on the MCP compile_context_pack tool; otherwise the command refuses and names sibyl project list. Packs now report their scope in JSON (scope: "project" | "all_projects") and in the markdown header. The web app opens on the most recently active project and waits for scopeReady before issuing any query, so no page fires an unscoped request.

Onboarding

  • sibyl setup [url] takes --yes/-y for agents and scripts, --context/-c to name the context, --no-hooks to skip the Claude Code hook, and --insecure/-k to skip TLS verification. Plain HTTP is refused for anything but localhost, 127.0.0.1, and ::1 unless --insecure is passed, because sign-in sends a password.
  • --insecure is honored only for that one setup and is never inherited from a sibling context pointing at the same server, closing a trust-downgrade path.
  • only_target_credentials() isolates SIBYL_AUTH_TOKEN during setup so a foreign automation token cannot leak to another server, and warns when it unsets one.
  • Server URLs are shell-quoted in every generated command (shlex.quote for POSIX, single quotes with doubled quotes for PowerShell) and validated as RFC 3986 http(s) with no userinfo, query, fragment, whitespace, or control characters.
  • Onboarding is deployment-aware: GET /setup/status reports providers_configured and configured_providers, treating keyless providers as ready. Bedrock counts as configured when a region plus per-plane credentials exist, and local embeddings when sentence_transformers is installed. The model-keys step then appears only for instance admins with an unconfigured provider; members skip it since they cannot change server settings anyway.
  • Managed hooks are matched by exact normalized command, so a user's own wrapper, extra flags, or bash -c variant is never deleted by mistake.

Release engineering

  • Core tests shard across pytest-xdist plus a new sibyl_core.pytest_shard plugin. --shard K/N assigns tests by zlib.crc32 of the node ID, giving stable disjoint shards; CI drives it through a CORE_SHARD matrix.
  • CI SurrealDB moved from memory to rocksdb:///data/sibyl.db, matching the compose service. The memory engine is not exclusive under contention, which made shared-engine tests flaky.
  • The Release workflow proves its own gates. tools/release/ci_evidence.py requires a green CI run on the candidate commit (accepting path-skipped jobs, refusing cancelled runs), and tools/release/nightly_evidence.py requires a Nightly Regression run on that same commit with no job skipped, dispatching one at most once. Both are stdlib-only.
  • A new required expected_sha input pins the release to the exact commit a dry run proved, so an advancing main cannot slip an unapproved commit into a cut.
  • sibyl update now upgrades the runtimes Sibyl actually created. It finds compose files at the paths sibyl docker and sibyl local own rather than a hardcoded ~/.sibyl/docker-compose.yml, reads the current version from the running API container's tag (falling back to the compose pin only when nothing is running), rejects image IDs and digest-shaped tags as unknown instead of calling them up to date, and never starts a stopped runtime or downgrades SurrealDB.
  • The embedded daemon round-trip test passes a nonce as SIBYL_GIT_COMMIT and requires it back in /health, after a verification run was caught talking to another lane's daemon on a recycled port.

Breaking changes

  • Removed sibyld migrate subcommands: rehearse, cutover, auth-flow, and auth-flow-compare. These were pre-SurrealDB cutover tooling. Auth-flow replay now lives in the e2e suite.
  • Removed Python extras: sibyl-core[runtime] (use the individual embeddings, graph, graphrag, llm extras), sibyl-core[crawler] (crawl4ai and mistune are now sibyld dependencies), and sibyl-core[graphrag-leiden].
  • Removed sibyl_core.utils exports: retry, RetryConfig, calculate_delay, GRAPH_RETRY, SEARCH_RETRY. Any external import of these breaks.
  • Removed modules: sibyl_core.models.tools and sibyl_core.models.responses, plus sibyl_core.tools.admin.rebuild_indices() and its RebuildResult type, and the SimilarTask alias in sibyl_core.tasks.estimation (use SimilarTaskInfo).
  • Leiden community detection removed. detect_communities_leiden() is gone and detect_communities() no longer takes an algorithm argument; Louvain is the only path. The uncalled cohere_rerank() reranking path was removed as well.
  • MCP context packs read one project. compile_context_pack() refuses without a project unless called with all_projects=True. Agent configurations relying on the previous implicit all-projects read must pass the flag.
  • Local Kubernetes dev stack removed: the Tiltfile, the whole infra/local/ tree (Kong, cert-manager, TiKV, Valkey manifests), and docs/deployment/tilt-minikube.md.
  • Removed env vars (none were read by the runtime): SIBYL_RETRIEVAL_MODE, SIBYL_NATIVE_FUSION_BACKEND (renamed to SIBYL_FUSION_BACKEND), SIBYL_RUN_WORKER, the four SIBYL_SANDBOX_* variables, and SIBYL_KNOWLEDGE_REPO_PATH / SIBYL_WISDOM_PATH / SIBYL_TEMPLATES_PATH / SIBYL_CONFIGS_PATH along with their settings fields.
  • Removed web client surface: 13 React Query hooks (including useTaskManage, useEpicManage, useBackup, useTestProviderKey, useCrawlProgress) and 20 API client methods (task and epic lifecycle calls, projectsApi.get(), orgsApi.get(), the backup getters, settingsApi.testProviderKey(), rawCapturesApi.updateReviewState()). The Next.js /ws rewrite is gone; use /api/ws directly.
  • Removed docs: the FalkorDB migration guide and the SurrealDB migration release notes, plus the FalkorDB migration playbook in the skill packs and the broken apps/api/examples/ scripts.
  • SurrealDB URL validation. rocksdb://, tikv://, surrealdb:// and a bare host:port are now rejected at startup in every environment; they are surreal start storage arguments, not client URLs. mem:// is banned in production like memory://, and surrealkv+versioned:// and file:// now need SIBYL_ALLOW_EMBEDDED_SINGLE_WRITER in production, as surrealkv:// already did. Uppercase schemes such as WS:// and SURREALKV:// now work. Server URLs (ws://, wss://, http://, https://) are unchanged.
  • Removed moon tasks: retrieval-mode-history and the dev-surreal alias (use moon run dev). The publish-dogfood-images.yml workflow and the retrieval_mode / longmemeval_native_fusion_backend dispatch inputs in eval.yml are gone.

Upgrade notes

  1. If you run behind a reverse proxy, set SIBYL_FORWARDED_ALLOW_IPS to your proxy's IPs or CIDR ranges before upgrading. The default is loopback only, which means every client shares one login bucket and audit records name the proxy. CIDRs must have no host bits set.
  2. Expect a re-embedding pass on first boot. Graph schema v31 and content schema v47/v48 migrate automatically and photograph pre-upgrade stamps. The embedding_sweep job then classifies each plane once. Set SIBYL_EMBEDDING_LEGACY_VECTORS=adopt to trust existing vectors as-is, or reembed to force a rebuild. Tune SIBYL_EMBEDDING_SWEEP_BUDGET_SECONDS, SIBYL_EMBEDDING_SWEEP_BATCH_SIZE, and SIBYL_EMBEDDING_SWEEP_CONCURRENCY if the default 45-second pass is too aggressive for your provider quota. Watch sibyl debug status for per-plane progress. If you upgrade and switch embedding providers in the same deploy, set SIBYL_EMBEDDING_LEGACY_VECTORS=reembed for that deploy (or upgrade first and switch in a second deploy), and restart every API and worker process together.
  3. Do not raise the pool size for embedded deployments. Embedded URLs are clamped to one connection on purpose; that clamp is what keeps compare-and-set fences correct against surrealdb-core 2.3. Configured pool sizes are ignored for memory://, mem://, surrealkv://, surrealkv+versioned://, and file://.
  4. Switching to Bedrock: sibyld already includes the bedrock extra (install sibyl-core[bedrock] only when using the library directly). Set SIBYL_BEDROCK_REGION, and provide either a bearer token or an AWS credential chain, not both a key and SIBYL_BEDROCK_PROFILE. Verify Cohere Embed v4 dimensions match one of 256, 512, 1024, or 1536 before boot. A model change triggers the sweep in note 2.
  5. Audit scripts and agent configs for project scope. Anything that relied on recall or context packs implicitly reading every project now needs --all or all_projects=True.
  6. Replace removed migrate subcommands and Python imports. Check CI scripts for sibyld migrate rehearse/cutover/auth-flow, replace sibyl-core[runtime] and sibyl-core[crawler] extras with the specific ones you need, and move off sibyl_core.utils.retry and the deleted models.tools / models.responses modules.
  7. Embedded daemon users: through 1.4.1, sibyld serve --embedded kept its graph in memory and lost it on every restart, so there is no earlier graph data to migrate. From 1.4.2 the graph persists in the data directory.
  8. Local Kubernetes dev users: the Tilt stack is gone. Use the docker-compose path or the Helm chart; docs/deployment/ was rewritten for 1.4.2.

Install

Local server

curl -fsSL https://raw.githubusercontent.com/hyperb1iss/sibyl/main/install.sh | sh -s -- --version 1.4.2

Remote CLI

curl -fsSL https://raw.githubusercontent.com/hyperb1iss/sibyl/main/install.sh | sh -s -- --remote --version 1.4.2
sibyl init --remote https://sibyl.example.com
sibyl auth login

Homebrew

brew install hyperb1iss/tap/sibyl
sibyl up

Arch Linux (AUR)

paru -S sibyl
sibyl up

Headless server

curl -fsSL https://raw.githubusercontent.com/hyperb1iss/sibyl/main/install.sh | sh -s -- --version 1.4.2 --no-open

Kubernetes (Helm)

helm repo add sibyl https://raw.githubusercontent.com/hyperb1iss/sibyl/gh-pages
helm repo update sibyl
helm upgrade --install sibyl sibyl/sibyl --version 1.4.2

Artifacts

This release includes Python wheels and sdists, the generated
Homebrew formula, the generated AUR PKGBUILD, Helm charts, Docker
SBOMs, aggregate dual-registry cosign receipts, and a SHA256 checksum
manifest.

Assets 19
Loading