Skip to content

v0.1.0

Choose a tag to compare

@sileod sileod released this 25 Sep 08:32
· 433 commits to main since this release

Breaking changes

  • Python ≥ 3.10 is required.
  • Readable labels. Many label sets now use readable names (e.g. liar, ShellRisk), boolean labels become "False"/"True", and some label orders changed. Code relying on label indices or old names may need updating.
  • Parked tasks. Evaluation benchmarks, duplicates and unloadable sources moved to tasksource.parked and are no longer in list_tasks(); they still load via parked.<name>.load().
  • Loaders. Many tasks now load from maintained mirrors or Hub parquet exports; a few task ids changed.

New

  • question on annotations, load_task(..., prompted=True), and dataset.question.
  • Recasts: recast="instruct" and recast="jev" (typed decisions), with per-example seeded sampling.
  • Provenance: hub_datasets() and task_provenance(id) name the loading repo and the original dataset behind tasksource copies.
  • metadata/weights.py sampling weights; ordinal=True for ordered label sets.
  • Many new and fixed tasks, including safety/prompt-injection sets, IntentGrasp, prm800k_dpo (split by MATH train/test) and multilingual sentiment.
  • tasks.md / mtasks.md: a one-line-per-task catalog.

What's Changed

  • Add modern classification, multilingual, relation and reward datasets by @sileod in #11
  • Native Jev sources: procedural generators and graded families by @sileod in #12
  • Add synthetic Jev dataset-construction package with live pilot sample by @sileod in #13
  • Language names for language ID tasks; fix unloadable tasks; support pandas 3 and huggingface_hub 1.x by @sileod in #14
  • Review fixes: strict Jev targets, seeded loading and recasts, Python ≥3.10, CI before publishing by @sileod in #15
  • Jev recast fixes from a sample audit: unshuffled free-text options, broken entities, HelpSteer2 turns by @sileod in #16
  • Jev graded: annotator distributions from UNLI, DynaSent, HateXplain, Measuring Hate Speech and LeWiDi by @sileod in #18

New Contributors

Full Changelog: v0.0.47...v0.1.0