Release v2.2.3

Release v2.2.3

What's changed

Bug fixes:

• Fixed list handling in local LLM response parsing — single-prompt batches were not unwrapped correctly
• Fixed chat template handling for local LLMs (#72)
• Fixed ExperimentConfig not being applied before resolving x_column in tasks (#68)
• Fixed keyword handling in RandomSearchSelector.select_exemplars (#70)
• Fixed various typos

Further changes:

• Added CONTRIBUTING.md with workflow, code quality, and testing guidelines
• Added accelerate as a dependency of the [transformers] extra, enabling local inference without manual installation
• Overhauled README: added quickstart code example, PyPI badge, installation guide, scientific publications section, and citation block
• New tests now account for 96% test coverage

Full Changelog: here

Release v2.2.2

Release v2.2.2

What's changed

Added features:

• further stabilize prompt handling

Full Changelog: here

Release v2.2.1

Release v2.2.1

What's changed

Added features:

• add a multi-objective task
• more seamless and robust interfaces between the components

Further changes:

• refactor the tutorial
• improve robustness in handling strings & prompt objects
• fees in block tracking and idx subsampling in CAPO

Full Changelog: here

Release v2.2.0

Release v2.2.0

What's changed

Added features:

• Extended interface of APILLM allowing to pass kwargs to the API
• Improve asynchronous parallelization of LLM calls shortening inference times
• Introduced a Prompt class to encapsulate instructions and few-shot examples

Further changes:

• Improved error handling
• Improved task-description infusion mechanism for meta-prompts

Full Changelog: here

Release v2.1.0

Release v2.1.0

What's changed

Added features:

• We added Reward and LLM-as-a-Judge to our task family

  • Reward allows you to write a custom function that scores the prediction, without requiring groundtruth
  • LLM-as-a-Judge allows you to deligate the task of scoring a prediction to a Judge-LLM, optionally accepting groundtruth

• Changes to CAPO, to make it applicable to the new tasks:

  • CAPO now accepts input parameter "check_fs_accuracy" (default True) - in case of reward tasks the accuracy cannot be evaluated, so we will take the prediction of the downstream_llm as target of fs.
  • CAPO also accepts "create_fs_reasoning" (default is True): if set to false, just use input-output pairs from df_few_shots

• introduces tag-extraction function, to centralize repeated code for extractions like "<final_answer>5</final_answer>"

Further changes:

• We now utilize mypy for automated type checking
• core functionalities of classification task has been moved to base task to prevent code duplication for other tasks
• test coverage is now boosted to >90%

Full Changelog: here

Release v2.0.1

Release v2.0.1

What's changed

• updated python requirement to >=3.10 (as 3.9 will lose support after October 2025)
• fixed numpy version constraints (thanks to @asalaria-cisco)
• make dependencies groups extras optional

Full Changelog: here

Release v2.0.0

Release v2.0.0

What's changed

Added features

• We welcome CAPO to the family of our optimizers! CAPO is an optimizer, capable of utilizing few-shot examples to improve prompt performance. Additionally it implements multiple AutoML-approaches. Check out the paper by Zehle et al. (2025) for more details (yep it's us :))
• Eval-Cache is now part of the ClassificationTask! This saves a lot of LLM-calls as we do not rerun already evaluated data points
• Similar to the Eval-Cache, we added a Sequence-Cache, allowing to extract reasoning chains for few-shot examples
• introduced evaluation strategies to the ClassificationTask, allowing for random subsampling, sequential blocking of the dataset or just retrieving scores of datapoints that were already evaluated on prompts

Further changes

• rearanged imports and module memberships
• Classificators are now called Classifiers
• Fixed multiple docstrings and namings of variables.
• Simplified testing and extended the testcases to the new implementations
• Classification task can now also output a per-datapoint score
• Introduced statistical tests (specifically paired-t-test), for CAPO

Full Changelog: here

Release v1.4.0

Release v1.4.0

What's changed

Added features

• Reworked APILLM to allow for calls to any API that follows the OpenAI API format
• Added graceful failing in optimization runs, allowing to obtain results after an error
• Reworked configs to ExperimentConfig, allowing to parse any attributes

Further Changes:

• Reworked getting started notebook
• Added tests for the entire package, covering roughly 80% of the codebase
• Reworked dependency and import structure to allow the usage of a subset of the package

Full Changelog: here

Release v1.3.2

Release v1.3.2

What's changed

Added features

• Allow for configuration and evaluation of system prompts in all LLM-Classes
• CSV Callback is now FileOutputCallback and able to write Parquet files
• Fixed LLM-Call templates in VLLM
• refined OPRO-implementation to be closer to the paper

Full Changelog: here

Release v1.3.1

Release v1.3.1

What's changed

Added features

• new features for the VLLM Wrapper (accept seeding to ensure reproducibility)
• fixes in the "MarkerBasedClassificator"
• fixes in prompt creation and task description handling
• generalize the Classificator
• add verbosity and callback handling in EvoPromptGA
• add timestamp to the callback
• removed datasets from repo
• changed task creation (now by default with a dataset)

Full Changelog: here