fix(solve): wait and collect results when --timeout 0#735
Merged
Conversation
`ref solve --timeout 0` queued executions to the local process pool and then exited without ever joining, so completed outputs were left in the scratch directory and never copied to the results directory or ingested into the database. The timeout value was overloaded as the "don't wait" sentinel (`timeout if wait else 0`), so a `0` timeout meant fire-and-forget rather than the intuitive "wait with no time limit". Decouple waiting from the timeout: `solve_required_executions` now takes an explicit `wait` and joins whenever it is True, regardless of the timeout value. A non-positive timeout (`<= 0`) means "wait with no time limit". Warn when `--no-wait` is used with an executor that only persists results during `join` (local and HPC), since skipping the join would silently orphan completed work in scratch. This is captured by a new `collects_results_on_join` marker on each executor.
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
ref solve --timeout 0ran every diagnostic successfully into the scratch directory but copied nothing to the results directory and ingested nothing into the database.Root cause
The default
LocalExecutoronly submits work inrun(); all result handling (copying scratch → results, DB ingestion,mark_successful) happens insidejoin(). The solver only calledjoin()whentimeout > 0:The CLI overloaded
timeoutas the "don't wait" sentinel (timeout=timeout if wait else 0), so--no-waitand--timeout 0were the same thing internally. Passing--timeout 0— which intuitively reads as "no time limit" — therefore skippedjoin()entirely, orphaning the completed work in scratch and leaving everyExecutionrow stuck atsuccessful=None.Fix
solve_required_executionsnow takes an explicitwaitand joins wheneverwaitis True, regardless of the timeout value.timeout(<= 0) now means wait with no time limit inLocalExecutor.joinandCeleryExecutor.join(HPC already did this via walltime).waitandtimeoutthrough separately instead of collapsing--no-waitontotimeout=0.--no-waitis used with an executor that only persists results duringjoin(local, HPC), since skipping the join silently orphans completed work. This is driven by a newcollects_results_on_joinmarker on each executor (Truefor local/HPC;Falsefor synchronous, which collects inrun(), and Celery, which collects via thehandle_resultworker callback).Recovery for affected users
Existing scratch outputs from a previous
--timeout 0run can be recovered without re-computing viaref executions reingest --include-failed.Checklist
Please confirm that this pull request has done the following:
changelog/