-
Notifications
You must be signed in to change notification settings - Fork 49
Open
Description
The SamplingStrategy interface has a major issue: everything about a sampling strategy is (unnecessarily and sneakily) stateful because the generate and validate fields carry around a MelleaSession. We should refactor this code before accumulating too much code that depends on the current interface. The strategy itself should be stateless, and sessions should be passed into the sample method explicitly (as the first parameter).
import mellea
from mellea.stdlib.sampling import RejectionSamplingStrategy
m1 = mellea.start_session(model_id="model_foo")
def generate(*args, **kwargs):
m1.backend._generate_from_raw(...)
strategy = RejectionSamplingStrategy(..., generate=generate)
m2 = mellea.start_session(model_id="model_bar")
m2.instruct("blah", strategy=strategy)Having the generate method close over the context is ruinous for a lot of reasons. Here are the two biggest issues:
- This code will not result in any calls to
m2._backend.generate_*; onlym1._backend.generate_from_contextgets called. This is very surprising, because we calledm2.instruct. - If you make async calls, then any code doing sampling will almost always have race conditions.
Metadata
Metadata
Assignees
Labels
No labels