Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
eb20e60
Feature/workflows (#8)
timo282 Oct 3, 2024
1f0119f
Fix/workflows (#11)
timo282 Oct 4, 2024
ba39fd6
Opro implementation (#7)
finitearth Oct 4, 2024
37591b7
Patch/pre commit config (#10)
finitearth Oct 4, 2024
5e2c0a9
Feature/prompt generation (#12)
finitearth Oct 4, 2024
689701a
Create LICENSE (#14)
mo374z Oct 5, 2024
f69bbb2
Refactor/remove deepinfra (#16)
mo374z Oct 9, 2024
1768297
Usability patches (#15)
finitearth Oct 12, 2024
65f02a5
Feature/examplar selection (#17)
finitearth Oct 13, 2024
e23dd74
Chore/docs release notes (#18)
timo282 Oct 13, 2024
25639c9
Merge branch dev accepting all incoming changes
finitearth Oct 13, 2024
2ac359b
revert Chore/docs release notes (#18)"
finitearth Oct 13, 2024
3e650cc
revert last commit
finitearth Oct 13, 2024
3f4c757
updated release notes and read me
finitearth Oct 13, 2024
ac2af13
Feature/read from df (#21)
finitearth Nov 4, 2024
7987f99
Update pyproject.toml
finitearth Nov 19, 2024
7217011
Update release-notes.md
finitearth Nov 19, 2024
2727b6e
merge
finitearth Nov 19, 2024
87fd54c
merge
finitearth Nov 19, 2024
aa6b7d7
Merge branch 'dev' of https://github.com/finitearth/promptolution int…
finitearth Nov 19, 2024
6e69b58
merges
finitearth Nov 19, 2024
e935fa2
resolve merge mistakes
finitearth Nov 19, 2024
68c67f3
delete duplicated lines
finitearth Nov 19, 2024
bee6d8d
Update release-notes.md (#24)
mo374z Nov 19, 2024
3ca7893
Fix/dependencies (#28)
finitearth Feb 21, 2025
d6203d6
Merge branch 'main' into dev
timo282 Feb 21, 2025
683d926
Add vllm as feature and a llm_test_run_script
mo374z Mar 3, 2025
69837fa
small fixes in vllm class
mo374z Mar 3, 2025
7563712
differentiate between vllm and api inference
mo374z Mar 3, 2025
af6f9f8
set up experiment over multiple tasks and prompts
mo374z Mar 3, 2025
bc9997a
change csv saving
mo374z Mar 3, 2025
7958b86
add base llm super class
mo374z Mar 4, 2025
e82db35
add changes from PR review
mo374z Mar 4, 2025
0045de7
change some VLLM params
mo374z Mar 5, 2025
0b3c7cb
fix tensor parallel size to 1
mo374z Mar 5, 2025
a73c378
experiment with batch size
mo374z Mar 5, 2025
1f68410
experiment with larger batch sizes
mo374z Mar 5, 2025
f5fe188
add continuous batch llm
mo374z Mar 5, 2025
1330a9e
remove arg
mo374z Mar 5, 2025
c6dbb7b
remove continuous batch inference try
mo374z Mar 5, 2025
42ab6c9
add batching to vllm
mo374z Mar 5, 2025
0be3d06
add batching in script
mo374z Mar 5, 2025
16514f4
Merge branch 'dev' of https://github.com/finitearth/promptolution int…
timo282 Mar 5, 2025
c5ac101
Add release notes and increase version number
timo282 Mar 5, 2025
0eb701b
remove llm_test_run.py script
mo374z Mar 5, 2025
f4f9722
Merge branch 'feature/vllm' of https://github.com/finitearth/promptol…
mo374z Mar 5, 2025
fae0113
change system prompt
mo374z Mar 5, 2025
b6440c7
Merge pull request #30 from finitearth/feature/vllm
mo374z Mar 5, 2025
b61c9ee
Fix/vllm (#33)
mo374z Mar 9, 2025
2f6f499
Merge branch 'main' into dev
mo374z Mar 9, 2025
b873ee0
implement changes from review
mo374z Mar 9, 2025
739e568
Merge branch 'dev' of https://github.com/finitearth/promptolution int…
mo374z Mar 9, 2025
ec5b709
add typing to token count callback
mo374z Mar 9, 2025
d2fecf7
Feature/deterministic (#35)
mo374z Mar 12, 2025
1f6a4f7
Fix/prompt creation (#36)
finitearth Mar 12, 2025
7858b49
Merge branch 'main' of https://github.com/finitearth/promptolution in…
mo374z Mar 12, 2025
89fa8b2
Merge branch 'dev' of https://github.com/finitearth/promptolution int…
mo374z Mar 12, 2025
7f5a26d
add arguements to test script
mo374z Mar 12, 2025
39b0779
added some feature notes
mo374z Mar 12, 2025
03fec71
Fix/template (#39)
finitearth Mar 20, 2025
02b6193
Fix/opro (#38)
finitearth Mar 20, 2025
3e86324
Merge branch 'main' of https://github.com/finitearth/promptolution in…
mo374z Mar 20, 2025
5d33d97
Refactor/generic api llm (#41)
finitearth Apr 17, 2025
2afa79e
Refactor/interfaces (#42)
finitearth Apr 20, 2025
cfbb22b
Merge branch 'main' into dev
finitearth Apr 20, 2025
3020c57
Remove obsolete Docstring
finitearth Apr 20, 2025
8bd84e3
remove redundant init
finitearth Apr 20, 2025
cfd9a67
remove redundant init
finitearth Apr 20, 2025
f11a7e1
resolve merge chaos
finitearth Apr 20, 2025
4d28bbf
formatting
finitearth Apr 20, 2025
babf2ae
remove obsolete reproduce experiments part in readme
finitearth Apr 21, 2025
c983be0
Update README.md
finitearth Apr 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .coverage
Binary file not shown.
3 changes: 2 additions & 1 deletion .github/actions/python-poetry/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,6 @@ runs:
path: .venv
key: ${{ runner.os }}-poetry-${{ inputs.poetry-version }}-${{ hashFiles('**/poetry.lock') }}
- name: Install dependencies
run: poetry install --no-interaction --no-root --only ${{ inputs.groups }}
run:
poetry install --no-interaction --no-root --only ${{ inputs.groups }}
shell: bash
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,11 @@ jobs:
- name: Set up Python and Poetry
uses: ./.github/actions/python-poetry
with:
groups: dev
groups: main,dev,test
- name: Run pre-commit
uses: pre-commit/action@v3.0.1
# - name: Run tests
# run: poetry run python -m pytest
- name: Run tests
run: poetry run python -m pytest

build:
runs-on: ubuntu-latest
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ on:
push:
branches:
- main
workflow_call:

permissions:
contents: write
Expand All @@ -22,6 +23,10 @@ jobs:
with:
groups: "docs"

- name: Generate notebook examples
run: |
poetry run jupyter nbconvert --to markdown --allow-errors --output-dir docs/examples notebooks/*.ipynb

- name: Deploy docs
run: |
poetry run mkdocs gh-deploy --force
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,5 @@ dist/
outputs/
results/
poetry.lock
CLAUDE.md
**/CLAUDE.local.md
30 changes: 16 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,33 @@
![promptolution](https://github.com/user-attachments/assets/84c050bd-61a1-4f2e-bc4e-874d9b4a69af)

[![CI](https://github.com/finitearth/promptolution/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/finitearth/promptolution/actions/workflows/ci.yml)
[![Docs](https://github.com/finitearth/promptolution/actions/workflows/docs.yml/badge.svg?branch=main)](https://github.com/finitearth/promptolution/actions/workflows/docs.yml)

# Promptolution
Promptolution is a library that provides a modular and extensible framework for implementing prompt tuning experiments. It offers a user-friendly interface to assemble the core components for various prompt optimization tasks.

In addition, this repository contains our experiments for the paper "Towards Cost-Effective Prompt Tuning: Evaluating the Effects of Model Size, Model Family and Task Descriptions in EvoPrompt".

This project was developed by [Timo Heiß](https://www.linkedin.com/in/timo-heiss/), [Moritz Schlager](https://www.linkedin.com/in/moritz-schlager/) and [Tom Zehle](https://www.linkedin.com/in/tom-zehle/).

## Getting Started
Take a look at our getting started notebook: [getting_started.ipynb](https://github.com/finitearth/promptolution/blob/main/notebooks/getting_started.ipynb)

## Installation

Use pip to install our library:

```
pip install promptolution
pip install promptolution[api]
```

If you want to run your prompt optimization locally, either via transformers or vLLM, consider running:

```
pip install promptolution[vllm,transformers]
```


Alternatively, clone the repository, run

```
Expand Down Expand Up @@ -58,17 +72,5 @@ Create API Keys for the models you want to use:
- Detailed logging and callback system for optimization analysis


## Getting Started
Take a look at our getting started notebook: [getting_started.py](https://github.com/finitearth/promptolution/blob/main/notebooks/getting_started.ipynb)
## Reproduce our Experiments

We provide scripts and configs for all our experiments. Run experiments based on config via:

```
poetry run python scripts/experiment_runs.py --experiment "configs/<my_experiment>.ini"
```
where `<my_experiment>.ini` is a config based on our templates.



This project was developed for seminar "AutoML in the age of large pre-trained models" at LMU Munich.
This project was developed for the "AutoML in the age of large pre-trained models" seminar at LMU Munich.
2 changes: 1 addition & 1 deletion docs/api/config.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Config

::: promptolution.config.Config
::: promptolution.config.ExperimentConfig
options:
show_root_heading: true
show_root_full_path: false
Expand Down
Loading
Loading