Skip to content

v0.5.1

Release v0.5.1: Add inspect_ai parity features for dataset operations
@nshkrdotcom nshkrdotcom tagged this 24 Dec 07:25 
This release introduces four major enhancements to achieve feature parity
with the inspect_ai Python library's dataset handling capabilities while
maintaining the idiomatic Elixir API design.

New Modules:

CrucibleDatasets.MemoryDataset provides lightweight in-memory dataset
construction without requiring file I/O. The from_list/2 and from_samples/2
functions accept lists of maps with automatic ID generation, custom naming,
and metadata preservation. This enables rapid prototyping and testing
workflows where datasets are constructed programmatically.

CrucibleDatasets.FieldMapping implements declarative field mapping for
flexible schema handling when loading datasets from external sources.
Supports mapping of input, expected, id, choices, and metadata fields
with optional transform functions for value processing. Handles both
atom and string keys transparently for maximum compatibility with JSON
and CSV sources.

CrucibleDatasets.Loader.Generic provides a unified loader for JSONL, JSON,
and CSV file formats with automatic format detection by file extension.
Integrates with FieldMapping for schema flexibility and supports options
for limiting results, shuffling with reproducible seeds, and automatic
ID generation for records lacking explicit identifiers.

Dataset Extensions:

The existing CrucibleDatasets.Dataset module gains four new operations:

  - filter/2 applies a predicate function to select matching items
  - sort/2 and sort/3 order items by atom key or function, ascending
    or descending
  - slice/2 and slice/3 extract subsets by range or start/count
  - shuffle_choices/2 randomizes multiple-choice options while
    preserving correct answer mapping through index tracking

All operations return new Dataset structs with updated metadata reflecting
the transformed item count, maintaining immutability guarantees.

Testing:

Added 142 comprehensive tests across four new test modules covering all
new functionality including edge cases for empty datasets, missing fields,
format detection, transform application, and reproducible shuffling with
seeded random number generation.

Documentation:

Updated README with version 0.5.1 examples demonstrating MemoryDataset
construction, Generic loader usage with FieldMapping, and Dataset
operation chaining. Added design specification document detailing the
implementation rationale and inspect_ai feature mapping analysis.

Cleanup:

Removed IMPLEMENTATION_SUMMARY.md as the information is now consolidated
in the CHANGELOG and inline module documentation.
Assets 2
Loading