1.5.0

Renders dataset entries as training text for a specific model.

The package could stream DatasetEntry values but not the thing a trainer
actually consumes: a stream of text with each example terminated. Turning one
into the other means deciding where an example ends and what marks the
boundary — and that decision belongs to the model being trained, not to the
stored data.

final texts = dataset.texts(
  DatasetTextFormatter(const DatasetSpecialTokens(eos: '<|endoftext|>')),
);
// "What is 2 + 3?\n5<|endoftext|>"

Added

• DatasetSpecialTokens — terminator, BOS, input/output prefixes, thinking
  delimiters and chat role prefixes, as plain strings. The package still never
  tokenizes; these are written into the rendered text and tokenized by whoever
  is training, so any tokenizer's markers work. Presets .plain(), .chatMl().
• DatasetTextFormatter — renders text, chat/agent/tool-call and reasoning
  entries. DatasetTextFormatter.chatTurns(...) optionally promotes a generated
  question/answer entry to a user turn and an assistant turn, through the same
  path a stored conversation takes — so a corpus mixing generated Q&A with real
  multi-turn data is not distinguishable to the model by format.
• Dataset.texts() and CurriculumDataset.textsPhase().
• DatasetTextFormatter.isPair() — the promotion rule, exposed as a
  subclass hook for corpora that mark unanswered entries their own way.

Design

Read time, not generation time. Stored entries keep neutral input /
output, so one dataset renders for a ChatML model and for a plain base model
without being regenerated. A store whose entries already carried <|im_end|>
could only ever train one family of model.

Three behaviours chosen so a naive render cannot quietly corrupt training data:

• an output equal to its input renders once, not twice — the plain-text
  generators emit the sentence as both fields because there is nothing to answer
• a thinking trace with no delimiters configured is dropped rather than
  concatenated into the answer, which would train the model to think out loud
  where it should reply
• promotion to chat turns applies only to entries that are genuinely a pair, so
  The kitten. does not become a user turn with an assistant echoing it back

Duplicate markers are suppressed at both ends, so ChatML turns stop producing
…hello<|im_end|>\n<|im_end|> — a sequence that occurs nowhere in real ChatML.
A terminator that genuinely differs from the turn suffix (<|endoftext|>
separating packed documents) is still appended.

Motivation

A consumer hit the gap directly. Packing entries into one token stream without a
terminator produces The kitten.The kitten.kid reads — examples run together,
the model never learns where one ends, and generation has no stop condition, so
a sampler runs until it exhausts its token budget. With a terminator the same
model stops by itself after two tokens.

---

341 tests, dart analyze clean.

Full changelog: v1.4.0...v1.5.0

1.4.0

Highlights

Curriculum mixed-phase streaming now honors review mixing policy, exercise abstractions are consolidated, and test coverage reaches ~94% (305 tests). This release also removes publish_to: none to prepare for pub.dev.

Added

• CurriculumPhaseDataset — CurriculumDataset.mixed() now interleaves review entries via Dataset.stream / batches
• ExerciseMetadataKeys, ExerciseCatalog, CatalogExerciseGenerator shared exercise abstractions
• DatasetStoreCapabilities extension (configureBulkInsertIfSupported, listDistinctMetadataValues)
• CurriculumLifecycle.stageStats() for per-stage entry counts
• doc/ARCHITECTURE.md, CONTRIBUTING.md, test/fixtures/ for CI/tests
• Query, lifecycle, mixed-phase, exception, and codec validation tests

Changed

• CurriculumBuilder.failIfVersionExists defaults to true (aligned with DatasetPipeline)
• Public exports deduplicated in lib/llm_dataset.dart (exercise modules via curriculum_api.dart only)
• Metadata keys consolidated under ExerciseMetadataKeys / TextExerciseMetadata
• Tests load curriculum and phrase templates from test/fixtures/

Fixed

• CurriculumDataset.mixed() no longer ignores manifest review mixing policy

Full changelog: https://github.com/OmnyGrid/llm_dataset/blob/v1.4.0/CHANGELOG.md