Curate lance-context records into trainable datasets (SFT/preference/rollout export)

[dcfocus]

Summary

Downstream half of the post-training data pipeline (upstream raw → records ingestion is #97). Add a workflow to curate stored ContextRecords — semantic dedup, decontamination, quality/lifecycle filtering, consolidation into conversations/trajectories, optional enrichment — into a versioned curated cut, then export that cut as SFT / preference / RL-rollout training data (JSONL + manifest).

raw logs ──(#97) ingest/normalize──▶ ContextRecords ──(this issue) curate + export──▶ trainable dataset
                                     (source="raw", faithful)        (SFT / preference / rollout
                                                                      JSONL + manifest)

lance-context acts as a curation/staging layer between raw logs and a trainer — not the system of record for raw data, and not a full curation platform.

Motivation

Post-training workflows need stable, auditable, reproducible datasets assembled from stored records at different levels and scopes (turn, session/run, user/project/tenant, source/kind), pinned to a version for reproducible training cuts.

Today users can approximate this with Context.list(filters=...) or direct Lance/PyArrow reads, but there are no curation primitives, no supported dataset schema, no export manifest, no streaming exporter, and no namespace-wide aggregation.

Why here (the bet)

Curation maps onto features lance-context already has, which is the test for whether this belongs here vs. in a dedicated tool:

• Embeddings (native vector search) → semantic dedup / near-duplicate collapse, and decontamination (drop training rows whose embedding is within ε of any eval/holdout row).
• Versioning (Lance manifests) → reproducible dataset versions with time-travel; curate-in-place + snapshot beats re-deriving from flat files.
• Lifecycle columns (supersede/retire/tombstone) → mark dropped/replaced rows non-destructively, auditably, reversibly instead of deleting.
• Relationships → stitch multi-turn trajectories and link preference/candidate sets during consolidation.

Target method coverage

The post-training stack has shifted since the original SFT → DPO → PPO recipe; the export shapes below should cover the methods teams actually use today:

• SFT for format/cold-start, including rejection-sampling / Best-of-N / ReST-EM distillation (SFT onto candidates that pass a reward/verifier threshold) → the SFT shape.
• Preference optimization — DPO and its reference-free / single-stage descendants (SimPO, ORPO), plus KTO-style unpaired binary feedback and LLM-as-judge N-way trajectory rankings (e.g. RULER) → the preference shape, which must support more than pairwise chosen/rejected.
• RL with verifiable rewards (RLVR) trained with GRPO/DAPO (and PPO, RLOO, REINFORCE++) → the rollout shape, generalized to groups of N responses per prompt each carrying a verifiable reward, and to multi-turn agentic tool trajectories with per-step / outcome rewards.

Proposed work

A. Curation: records → curated, versioned cut

• Semantic dedup / near-duplicate collapse via embedding distance threshold.
• Decontamination against an eval/holdout set by embedding distance.
• Quality/metadata filters; lifecycle-aware exclusion by default (drop tombstoned/expired/retired/superseded/contradicted).
• Consolidation: group by session_id / run_id / prompt; stitch ordered turns into conversations; rebuild multi-turn trajectories from state_metadata (step, active_plan_id, tokens_used) and relationships; link same-prompt samples (group_id) for GRPO/RLVR groups.
• Mark cut/replaced rows non-destructively via existing lifecycle fields (auditable, reversible).
• Pluggable enrichment hooks (user-supplied) — callbacks for reward/verifier scoring, preference labeling, PII redaction, and long-trace summarization/compaction. lance-context calls the hook and persists the result; it bundles no models.
• Snapshot/tag the curated cut → reproducible, version-pinned, then handed to export.

B. Export schemas + provenance + manifest

• SFTExample: ordered messages, optional system context, metadata, provenance, quality fields. Doubles as the rejection-sampling / Best-of-N target (export only candidates above a reward/verifier threshold).
• PreferenceExample: prompt/messages plus one of — paired chosen/rejected (DPO/SimPO/ORPO), a single response with an unpaired binary label (KTO-style), or an N-way ranked list (LLM-judge / RULER-style) — with feedback/reward metadata and provenance.
• RolloutExample for RL (PPO/GRPO/RLVR): prompt, one or more responses, optional group_id linking the N samples for one prompt, per-response reward plus reward_source (verifier / tests / judge), multi-turn tool trajectory, state metadata, terminal flags.
• Shared provenance: context URI, version, record ids, external ids, tenant, source, bot_id, session_id, run_id, created_at range.
• Export manifest: context URI + pinned version, curation parameters (dedup/decontam thresholds, filters/selectors), schema version, created_at range, source record ids, counts.

C. Aggregation APIs

• Group records by session_id, run_id, external-id prefix, tenant/source/bot/session, or time window; build ordered windows by (created_at, id).
• Preserve lifecycle behavior by default; allow version pinning so an export can be reproduced later.

D. Export APIs

ctx.export_training(
    task="sft",  # "sft" | "preference" | "rollout"
    format="jsonl",
    group_by="session_id",
    filters={"tenant": "acme", "source": "memory"},
    version=ctx.version(),
    output_uri="s3://bucket/training/acme-sft.jsonl",
)

Potential Rust/server equivalents: ContextStore::export_training(...); REST POST /api/v1/contexts/{name}/exports/training; Python helpers for JSONL and Parquet/Arrow output. Export runs in a streaming/chunked mode (mirrors streaming ingestion in #97).

E. Namespace-level aggregation

• For ContextNamespace, support partial selectors (e.g. all source partitions for one tenant); fan out across matching partitions and write one merged export; record partition URIs and versions in the manifest. Defer strong cross-partition consistency unless a snapshot/tag mechanism is available.

F. Examples + docs

• Raw chat logs → deduped/decontaminated SFT cut.
• Rejection-sampling / Best-of-N SFT export filtered by a reward/verifier threshold.
• Preference export covering all three shapes: paired chosen/rejected (DPO), unpaired binary (KTO), N-way judge rankings.
• RL rollout export for GRPO/RLVR: groups of N responses per prompt with verifiable rewards.
• Multi-turn agentic rollout export from tool-use trajectories using state_metadata and relationships.
• Direct Lance/PyArrow inspection path for advanced users.

Non-goals

• Not a replacement for dedicated curation tooling (NeMo Curator / datatrove); not a labeling UI.
• Does not compute rewards / run verifiers / call judge or summarizer models itself — those are user-supplied hooks.
• Does not ingest raw logs — that is Ingest raw interaction logs into lance-context records (batch + streaming) #97.

Acceptance criteria

• A user can produce a deduped/decontaminated, lifecycle-correct, versioned curated cut from stored records.
• A user can export SFT JSONL grouped by session, plus a rejection-sampling variant filtered by a reward/verifier threshold.
• A user can export preference JSONL in paired (chosen/rejected), unpaired binary (KTO-style), and N-way ranked forms from records carrying the corresponding feedback metadata.
• A user can export RL rollout rows with prompt, response(s), reward and reward_source, optional group_id for GRPO/RLVR groups, and multi-turn tool-trajectory metadata.
• Exports include a manifest with context version, curation parameters, filters/selectors, schema version, created_at range, and source record ids.
• Export can run in a streaming/chunked mode without materializing the full dataset in memory.
• Namespace exports can aggregate across matching partitions via partial selectors.
• Tests cover dedup/decontamination, lifecycle filtering, version pinning, grouping order, and reproducibility.

Notes

This builds on the existing ContextRecord schema, lifecycle fields, relationships, embeddings, tenant/source columns, and ContextNamespace resolver rather than introducing separate training tables. The ContextRecord schema + export manifest is the shared contract with #97 (ingestion). First implementation can target JSONL plus a manifest, then add Parquet/Arrow once the schema stabilizes.

The task names use preference and rollout (rather than dpo/ppo) so a single shape covers the broader method family — DPO/SimPO/ORPO/KTO/judge-ranked for preference, and PPO/GRPO/RLVR for rollout — without a schema change per algorithm.

Implementation audit (2026-06-23)

• Validated against the current repo: the issue is legitimate; there is no export_training API, supported training dataset schema, export manifest, or streaming/chunked training exporter today.
• Existing substrate is real: ContextRecord has embeddings, relationships, tenant/source/session/run fields, state_metadata, lifecycle columns, external_id, and Lance version/checkout support.
• Clarification for namespace export: partial-selector fan-out is new work. ContextNamespace currently resolves complete selectors and lists partitions; cross-partition reads are still a phase-2 design item. Strong cross-partition reproducibility should record each partition URI and pinned version unless/until namespace snapshot/tag support lands.
• Clarification for curation lifecycle: use append-only lifecycle/metadata markers for dropped, deduped, or decontaminated rows. Do not use delete/tombstone for training-cut exclusion because that would conflate curation with record removal.