Skip to content

0.5.0: Composite experiments

Compare
Choose a tag to compare
@Jacob-Stevens-Haas Jacob-Stevens-Haas released this 24 Apr 22:41
· 8 commits to main since this release
0.5.0
64142fd

Composite experiments

This change adds the ability to stitch together steps and lookup dicts from different modules into a multi-step experiment. The intent is to support research workflows that draw from different data creation pipeline, different evaluation criteria, and different intermediate steps, all of which may have scientifically important variants to evaluate.

This applies some additional requirements to the type signature of steps (steps which pass data must return a dict with a "data" key, which is then passed as the first positional argument to the next step). It also allows better static typing of experiment steps.

Old behavior is still accessible, but requires the -m flag on the command line.

Additional features:

  • Default trials folder is now repo root
  • Remove special "group" CLI argument. Treat group as any eval argument
  • Add --version to CLI and __version__ to module
  • Add logfile "experiment.log" and source.py to metadata. Given requirements.txt and source.py, it should be possible to reproduce the result of any trial.

Bugfixes:
No longer create kernels that persist after trial