Skip to content

Add SGD context extraction - #24

Merged
aryamantodkar merged 2 commits into
mainfrom
codex/add-sgd-context
Aug 24, 2026
Merged

Add SGD context extraction#24
aryamantodkar merged 2 commits into
mainfrom
codex/add-sgd-context

Conversation

@aryamantodkar

Copy link
Copy Markdown
Contributor

Summary

  • add a validated SGD train context iterator for USER turns
  • retain exact character offsets, slot values, service provenance, and context-only turns
  • validate dialogue structure against train/schema.json
  • reject unknown services, categorical span annotations, malformed offsets, duplicate dialogue IDs, and non-train splits
  • preserve valid overlapping SGD annotations for the later ontology-mapping stage
  • add focused tests for extraction and failure modes

Why

SGD supplies conversational context, not ITN labels. Enrichment needs a deterministic extraction seam before any service/slot pair can be approved for a SpanKind or combined with trusted donor values.

Invariant protected

This module reads only SGD train data and yields USER turns in deterministic file/dialogue/turn order. Every emitted span must reference a schema-declared non-categorical slot, exactly slice its utterance, and retain its source provenance. It deliberately does not create BIO labels or infer ITN classes.

Without this invariant, malformed offsets, categorical values, dev/test leakage, or guessed slot semantics could enter enrichment and recreate the semantic-label problems found in Google PHONE data.

Full-train verification

  • dialogues: 16,142
  • USER turns: 164,982
  • non-categorical slot spans: 66,603
  • context-only USER turns: 114,951
  • two valid nested annotation cases were preserved

Validation

  • focused tests: 11 passed
  • full Python suite: passed
  • Ruff lint: passed
  • targeted Ruff format check: passed
  • package build: passed
  • Rust tests: 21 passed

The repository-wide Ruff format check still reports two pre-existing formatting differences in tests/test_golden.py and tests/test_types.py. Neither file is changed by this PR.

Scope

This PR does not add the audit CLI, ontology policy, donor pools, example generation, BIO compilation, dataset combination, splitting, or training.

@aryamantodkar
aryamantodkar merged commit 96b385a into main Aug 24, 2026
@aryamantodkar
aryamantodkar deleted the codex/add-sgd-context branch August 24, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant