SLayer 0.5.0: Memory, Dealing with Schema Drift, and Type Cleanup

SLayer 0.5.0 Release Notes

A feature release: five PRs since 0.4.3 add an agent-memory layer, a schema-drift validation and idempotent re-ingestion subsystem, three new ranking transforms, and type-aware CAST emission. Two breaking changes ride along, both covered by an automatic v4 -> v5 storage migration: DataType enum values are now uppercase sqlglot-aligned names (TEXT, INT, DOUBLE, BOOLEAN, DATE, TIMESTAMP), and the unused aggregation pseudo-types (count, sum, avg, min, max, last, count_distinct) have been dropped from DataType.

Schema-drift detection and idempotent re-ingestion (DEV-1356)

A new read-only diagnostic, engine.validate_models(data_source=...), diffs persisted models against live database schemas and returns the minimal list of deletes needed for SQL generation to remain valid (EditModelDelete for individual columns / measures / joins, WholeModelDelete when an entire backing table is gone). The same surface is exposed as the MCP validate_models tool, the REST POST /validate-models endpoint, and the CLI slayer validate-models [--data-source X] [--json]; --force-clean actually applies the deletes for batch repair workflows. The new top-level slayer/engine/schema_drift.py module owns the cascade walking and a two-pass invariant so dropping a column also prunes anything that referenced it. Query-time DBAPI errors that are caused by drift are now wrapped as the new SchemaDriftError (in slayer/core/errors.py) carrying a structured payload instead of opaque driver text. ingest_datasource_idempotent() in slayer/engine/ingestion.py walks the live datasource and only adds to existing sql_table-mode models -- it never overwrites user customizations and silently skips sql-mode and query-backed models. See docs/concepts/schema-drift.md and docs/examples/schema-drift.md.

Agent memory layer: learnings and saved queries (DEV-1357)

A new slayer.memories package (models.py, service.py, resolver.py) lets agents save free-form notes ("learnings") and example queries indexed by the schema entities they reference, then recall them before drafting new queries. Three MCP tools (save_memory, forget_memory, recall_memories), three REST endpoints (POST /memories, DELETE /memories/{memory_id}, POST /memories/recall), three CLI subcommands (slayer memory save|forget|recall), and matching SlayerClient async methods all front the same MemoryService orchestrator. Entity references are canonicalized through slayer.memories.resolver; unresolvable references raise the new EntityResolutionError (HTTP 422), and missing memory IDs raise the new MemoryNotFoundError (HTTP 404). inspect_model now renders a Learnings section listing memories that reference the inspected model. Storage adds a memories table (SQLite) plus an entity-overlap index, and a memories.yaml + counters.yaml pair (YAML backend). See docs/concepts/memories.md and specs/DEV-1357-memories.md.

Type-aware CAST emission and DataType realignment (DEV-1361, BREAKING)

DataType enum values were lowercase strings ("string", "number", "integer", "boolean", "date", "time") plus seven unused aggregation pseudo-types ("count", "sum", ...); they are now uppercase sqlglot-aligned names (TEXT, INT, DOUBLE, BOOLEAN, DATE, TIMESTAMP), and the pseudo-types are gone. Generated SQL now emits explicit CAST(... AS <type>) on column SQL and measure expressions (idempotent against pre-cast input, so CAST(... AS INT) is not double-wrapped) so result tuples match downstream comparators that care about driver-returned types. Auto-ingestion now narrows generic numeric SQL columns to INT vs DOUBLE based on live precision/scale via slayer/storage/type_refinement.py instead of always emitting DOUBLE. The SlayerModel schema bumps to v5: the v4 -> v5 converter in slayer/storage/v5_migration.py is a pure dict transform that rewrites legacy lowercase types and strips the dropped pseudo-types so the field falls through to its default after Pydantic loads; it recurses into source_queries[].source_model inline dicts so multi-stage models migrate cleanly. Migration is automatic on load via the converter chain in slayer/storage/migrations.py. The new slayer storage migrate-types [--dry-run] [--data-source X] CLI exposes the optional post-migration datasource introspection refinement step explicitly for batch workflows. Lenient before-validators on Column.type and ModelMeasure.type still accept legacy lowercase input from external callers without erroring.

Ranking transform family (DEV-1353)

Three new first-class transforms -- percent_rank, dense_rank, ntile -- join the existing rank. All four accept an optional partition_by= kwarg to rank within subsets of rows (a column name, a dotted join path, or a list of either) and require no time dimension; ntile additionally takes a positive-integer n= kwarg. The SQL generator emits native PERCENT_RANK(), DENSE_RANK(), NTILE(n) window functions; positional args beyond the wrapped measure are rejected with an actionable error pointing at the keyword form. See the rank-family entries in slayer/help/topics/04_transforms.md and docs/concepts/formulas.md.

Demo: upstream jafgen is now a core dependency

The Jaffle Shop demo dataset switched from a vendored, extras-only git-URL install to the upstream jafgen package pinned to ^0.4.14 as a core dependency, so pip install motley-slayer alone enables slayer datasources create demo (and the --demo flags on slayer serve / slayer mcp) without any manual git-based install. slayer/demo/jaffle_shop.py was rewritten to drive the upstream CLI rather than ship its own vendored DDL.

Schema versions

SlayerModel is now version 5 (was 4 in 0.4.3). SlayerQuery remains version 3 and DatasourceConfig remains version 1. All older versions are upgraded automatically on load via the converter chain in slayer/storage/migrations.py; saves always emit the current version.