Skip to content

Commit a6ee755

Browse files
Enhance logging for valset usage in GEPA (#8770)
* Enhance logging for valset usage in GEPA Clarified warning about using trainset as valset and added guidance on valset size. * Enhance warning for missing validation set * Refine logger messages in GEPA implementation Updated warning and info messages for clarity and detail.
1 parent 6ae6736 commit a6ee755

File tree

1 file changed

+3
-1
lines changed

1 file changed

+3
-1
lines changed

dspy/teleprompt/gepa/gepa.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -428,8 +428,10 @@ def compile(
428428

429429
logger.info(f"Running GEPA for approx {self.max_metric_calls} metric calls of the program. This amounts to {self.max_metric_calls / len(trainset) if valset is None else self.max_metric_calls / (len(trainset) + len(valset)):.2f} full evals on the {'train' if valset is None else 'train+val'} set.")
430430

431+
if valset is None:
432+
logger.warning("No valset provided; Using trainset as valset. This is useful as an inference-time scaling strategy where you want GEPA to find the best solutions for the provided tasks in the trainse, as it make GEPA overfit prompts to the provided trainset. In order to ensure generalization and perform well on unseen tasks, please provide separate trainset and valset. Provide the smallest valset that is just large enough to match the downstream task distribution, while keeping trainset as large as possible.")
431433
valset = valset or trainset
432-
logger.info(f"Using {len(valset)} examples for tracking Pareto scores. You can consider using a smaller sample of the valset to allow GEPA to explore more diverse solutions within the same budget.")
434+
logger.info(f"Using {len(valset)} examples for tracking Pareto scores. You can consider using a smaller sample of the valset to allow GEPA to explore more diverse solutions within the same budget. GEPA requires you to provide the smallest valset that is just large enough to match your downstream task distribution, while providing as large trainset as possible.")
433435

434436
rng = random.Random(self.seed)
435437

0 commit comments

Comments
 (0)