Repository navigation
v0.1.0
Breaking changes
- Python ≥ 3.10 is required.
- Readable labels. Many label sets now use readable names (e.g. liar, ShellRisk), boolean labels become
"False"/"True", and some label orders changed. Code relying on label indices or old names may need updating. - Parked tasks. Evaluation benchmarks, duplicates and unloadable sources moved to
tasksource.parkedand are no longer inlist_tasks(); they still load viaparked.<name>.load(). - Loaders. Many tasks now load from maintained mirrors or Hub parquet exports; a few task ids changed.
New
questionon annotations,load_task(..., prompted=True), anddataset.question.- Recasts:
recast="instruct"andrecast="jev"(typed decisions), with per-example seeded sampling. - Provenance:
hub_datasets()andtask_provenance(id)name the loading repo and the original dataset behind tasksource copies. metadata/weights.pysampling weights;ordinal=Truefor ordered label sets.- Many new and fixed tasks, including safety/prompt-injection sets, IntentGrasp, prm800k_dpo (split by MATH train/test) and multilingual sentiment.
tasks.md/mtasks.md: a one-line-per-task catalog.
What's Changed
- Add modern classification, multilingual, relation and reward datasets by @sileod in #11
- Native Jev sources: procedural generators and graded families by @sileod in #12
- Add synthetic Jev dataset-construction package with live pilot sample by @sileod in #13
- Language names for language ID tasks; fix unloadable tasks; support pandas 3 and huggingface_hub 1.x by @sileod in #14
- Review fixes: strict Jev targets, seeded loading and recasts, Python ≥3.10, CI before publishing by @sileod in #15
- Jev recast fixes from a sample audit: unshuffled free-text options, broken entities, HelpSteer2 turns by @sileod in #16
- Jev graded: annotator distributions from UNLI, DynaSent, HateXplain, Measuring Hate Speech and LeWiDi by @sileod in #18
New Contributors
Full Changelog: v0.0.47...v0.1.0