Skip to content

Fix InSampleUniformGenerator selecting arms from CANDIDATE trials#5109

Closed
ItsMrLin wants to merge 1 commit into
facebook:mainfrom
ItsMrLin:export-D98627073
Closed

Fix InSampleUniformGenerator selecting arms from CANDIDATE trials#5109
ItsMrLin wants to merge 1 commit into
facebook:mainfrom
ItsMrLin:export-D98627073

Conversation

@ItsMrLin
Copy link
Copy Markdown
Contributor

Summary:
When LILO labeling generates pairwise comparisons, InSampleUniformGenerator selects 2 arms uniformly at random from generated_points built by RandomAdapter. Previously, this pool included arms from CANDIDATE trials (which have no observed data) via two sources:

  1. arms_by_signature_for_deduplication (all non-FAILED arms)
  2. pending_observations (CANDIDATE/STAGED/RUNNING arms appended for dedup)

When a CANDIDATE trial arm was selected, LILOPairwiseMetric could not find source metric data for it, causing fetch_data to raise an ExceptionGroup.

Fix: In RandomAdapter._gen(), when the generator is InSampleUniformGenerator:

  • Filter arms_to_deduplicate to only arms from expecting_data trials (COMPLETED, EARLY_STOPPED, RUNNING)
  • Skip appending pending_observations (which re-adds CANDIDATE arms)

This is correct for all current and foreseeable use cases of InSampleUniformGenerator. The generator is used in two scenarios: LILO labeling (active) and potentially bandit candidate generation (not currently using it). In both cases, we need arms with observed data -- selecting an arm from a CANDIDATE or STAGED trial that has never been evaluated is never meaningful. Excluding pending observations is similarly safe: pending points exist to prevent regular generators (Sobol, BO) from re-suggesting in-flight arms, but for in-sample selection the entire point is to pick from already-observed arms.

The change is gated behind an isinstance check -- no effect on other generators (Sobol, Uniform, etc.).

Reviewed By: saitcakmak

Differential Revision: D98627073

Summary:
When LILO labeling generates pairwise comparisons, `InSampleUniformGenerator` selects 2 arms uniformly at random from `generated_points` built by `RandomAdapter`. Previously, this pool included arms from CANDIDATE trials (which have no observed data) via two sources:

1. `arms_by_signature_for_deduplication` (all non-FAILED arms)
2. `pending_observations` (CANDIDATE/STAGED/RUNNING arms appended for dedup)

When a CANDIDATE trial arm was selected, `LILOPairwiseMetric` could not find source metric data for it, causing `fetch_data` to raise an `ExceptionGroup`.

**Fix:** In `RandomAdapter._gen()`, when the generator is `InSampleUniformGenerator`:
- Filter `arms_to_deduplicate` to only arms from `expecting_data` trials (COMPLETED, EARLY_STOPPED, RUNNING)
- Skip appending `pending_observations` (which re-adds CANDIDATE arms)

This is correct for all current and foreseeable use cases of `InSampleUniformGenerator`. The generator is used in two scenarios: LILO labeling (active) and potentially bandit candidate generation (not currently using it). In both cases, we need arms with observed data -- selecting an arm from a CANDIDATE or STAGED trial that has never been evaluated is never meaningful. Excluding pending observations is similarly safe: pending points exist to prevent regular generators (Sobol, BO) from re-suggesting in-flight arms, but for in-sample selection the entire point is to pick from already-observed arms.

The change is gated behind an `isinstance` check -- no effect on other generators (Sobol, Uniform, etc.).

Reviewed By: saitcakmak

Differential Revision: D98627073
@meta-cla meta-cla Bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Mar 30, 2026
@meta-codesync
Copy link
Copy Markdown

meta-codesync Bot commented Mar 30, 2026

@ItsMrLin has exported this pull request. If you are a Meta employee, you can view the originating Diff in D98627073.

@codecov-commenter
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.41%. Comparing base (3bcf8c4) to head (cb2b1fb).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5109   +/-   ##
=======================================
  Coverage   96.41%   96.41%           
=======================================
  Files         613      613           
  Lines       68106    68131   +25     
=======================================
+ Hits        65663    65688   +25     
  Misses       2443     2443           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@meta-codesync
Copy link
Copy Markdown

meta-codesync Bot commented Mar 30, 2026

This pull request has been merged in 218bf55.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported Merged meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants