Skip to content

WIP fix: cache for the baleen test and add notebook cache to everywhere. #606

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

WIP fix: cache for the baleen test and add notebook cache to everywhere. #606

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion dsp/modules/cache_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ def wrapper(*args, **kwargs):
cachedir = os.environ.get('DSP_CACHEDIR') or os.path.join(Path.home(), 'cachedir_joblib')
CacheMemory = Memory(location=cachedir, verbose=0)

cachedir2 = os.environ.get('DSP_NOTEBOOK_CACHEDIR')
project_home = Path(__file__).resolve().parent.parent.parent
cachedir2 = os.environ.get('DSP_NOTEBOOK_CACHEDIR') or os.path.join(project_home, 'cache')
NotebookCacheMemory = dotdict()
NotebookCacheMemory.cache = noop_decorator

Expand Down
10 changes: 6 additions & 4 deletions tests/examples/test_baleen.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
import pytest
import os
from dsp.modules.cache_utils import *
from dsp.utils import deduplicate
import dspy.evaluate
import dspy
Expand Down Expand Up @@ -87,8 +89,8 @@ def validate_context_and_answer_and_hops(example, pred, trace=None):
if max([len(h) for h in hops]) > 100:
return False
if any(
dspy.evaluate.answer_exact_match_str(hops[idx], hops[:idx], frac=0.8)
for idx in range(2, len(hops))
dspy.evaluate.answer_exact_match_str(hops[idx], hops[:idx], frac=0.8)
for idx in range(2, len(hops))
):
return False

Expand All @@ -106,7 +108,7 @@ def gold_passages_retrieved(example, pred, trace=None):

# @pytest.mark.slow_test
# TODO: Find a way to make this test run without the slow hotpotqa dataset
def _test_compiled_baleen():
def test_compiled_baleen():
Copy link
Collaborator

@thomasahle thomasahle Mar 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it going to break things to have a unittest using OpenAI like this?

Copy link
Collaborator Author

@ammirsm ammirsm Mar 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the other change that I added in another file this should run based on the cache of the notebook.

trainset, devset = load_hotpotqa()
lm = dspy.OpenAI(model="gpt-3.5-turbo")
rm = dspy.ColBERTv2(url="http://20.102.90.50:2017/wiki17_abstracts")
Expand All @@ -133,4 +135,4 @@ def _test_compiled_baleen():
compiled_baleen, metric=gold_passages_retrieved
)
# assert compiled_baleen_retrieval_score / 100 == 27 / 50
assert uncompiled_baleen_retrieval_score < compiled_baleen_retrieval_score
assert uncompiled_baleen_retrieval_score < compiled_baleen_retrieval_score