Skip to content

Layer v0.8.0

Latest

Choose a tag to compare

@hevbot hevbot released this 03 Nov 17:02

Highlights

  • Index SharePoint and OneDrive. A clouddrive Warehouse connects to Microsoft 365 through an Entra app registration with the Graph application permission Sites.Read.All and admin consent, authenticating by certificate (the default) or client secret. The operator verifies access before any pipeline runs and reports the token's granted roles. A clouddrive Pipeline source then syncs selected sites, libraries, folders or individual items through Graph delta, and keeps deselected rows until you remove them. PDFs, Office documents and scanned images extract once per content version; photos embed with CLIP and deduplicate by content. Originals are retained independently of indexing. Cloud drives / Microsoft Graph, Cloud drive sources and Warehouse originals. (#693, #695, #697, #698, #852, #853, #866, #876, #1023)

  • Python SDK path parameters. The client now percent-encodes path parameters. Pass raw namespace names and other path arguments; remove caller-side URL encoding to avoid double encoding.

  • Stage and restore record revisions. Pgvector namespaces can opt into immutable record history with automatic activation or explicit publishing. Search and normal reads use only the active revision; inactive revisions stay outside candidate retrieval. Activation supports expected-current-version conflict checks and a separate event history. Revisions remain until namespace deletion; named record deletion preserves history for restoration. This first slice supports string IDs and full upserts. Record versions. (#1025)

  • PDF pages embed in full. layer-extract write replaces the extractor's CPU sidecar embedding stage. It writes staged page rows with a native text embedding schema, preserving page provenance and removing the 1,500-character cap. On a targeted 30-question long-page regression set, top-five page hits improved from 16/30 to 26/30 with voyage/voyage-4; this is not a production-wide quality estimate. (layer-pro#711)

  • Functions rerun on changed inputs. Same-version source changes requeue completed rows, including pipeline metadata writes. Claims return input_revision; completion items echo it so delayed outputs cannot overwrite a newer source revision. The generated Python worker forwards revisions automatically. Manual workers must forward claim receipts after inputs change. (layer-pro#721)

  • MCP discovers and searches your namespaces. Minted keys authenticate on pass-through gateways; list_namespaces exposes readable namespaces, and generic tools support larger namespace sets. Text search selects BM25 for non-fuzzy fields and hybrid ranking for fuzzy fields; declared CLIP profiles support image search. The Desktop extension connects these tools to Claude Desktop. MCP reference and Desktop setup. (layer-pro#741, #754, #745, #743, #749; docs #814)

  • Collapse returns one result per document. Add collapse: {by: "document"} to a ranked query and the gateway reads top_k × expansion rows, groups them, and returns the best top_k groups with their matching rows. "document" uses _hevlayer_parent_id or a schema attribute marked document: true, and returns 400 when a namespace has neither. missing takes own, together, drop, or {"mode", "values"}, so a ["job", "folder"] list can fall back past a "No job" placeholder. It works on turbopuffer and Postgres, grouping in the gateway on both. The tunable expansion defaults to 10. A larger candidate window can expose more distinct groups at increased store work, response size and latency; reads are capped at 10,000 rows. Paging with cursor is not supported yet. Collapse reference. (#827, #824, #833)

  • Field stats describe filterable attributes. Pro metadata accepts field_stats=true for missing counts, distinct values and numeric ranges, plus field_stats_top=N for common values. Index.spec.fieldStats defaults to automatic coverage and supports lists, exclusions, a value cap and an interval. Reads never start scans; stats and snapshot facets share one reconcile projection. Initial coverage remains explicit rather than reporting an exact zero. Namespace metadata. (#825)

Deferred

High-cardinality estimates above the exact-value cap and measured latency, accuracy and scan-byte acceptance remain for later increments. This increment reports a conservative lower bound above the cap.

Clients

  • The Python, TypeScript and Go clients are published at 0.8.0.
Merged since v0.7.3: auto-generated ledger

Release branch: v0.8
Docs preview:

Added

Changed

Fixed

Docs