fix: report an unparseable action as FAIL, not as a completed task - #16
Merged
Conversation
`ActionType.DONE` is a successful terminal outcome: a runner that sees it
ends the episode as complete. Every parse and conversion failure in this
package produced exactly that value, so "the model said it finished" and
"we could not read the model's output at all" were the same Action.
- `openadapt_types/parsing.py`: the `_done()` helper behind all 19 failure
paths now returns `Action(type=ActionType.FAIL)` carrying the reason in
`reasoning` and in `raw[PARSE_ERROR_KEY]`. This also repairs
`parse_action`, which used `type != DONE` as its own success test and so
re-parsed a legitimate `{"type": "done"}` as DSL before returning it.
- `openadapt_types/_compat.py`: `from_benchmark_action`, `from_ml_action`
and `from_omnimcp_action_decision` defaulted a missing type to `"done"`
and mapped every unrecognized type to `DONE` with no log at all. An
unconvertible record became a successful terminal step in converted
training data. They now return FAIL with `raw[UNCONVERTIBLE_ACTION_KEY]`
and the source dict.
- `openadapt_types/computer_state.py`: `get_children` returned `[]` both
for a node with no children and for a node_id that is not in the state.
An unknown node now raises KeyError.
- `openadapt_types/__init__.py`: `__version__` was the literal "0.1.0"
against a package on 0.6.3. It is now read from installed distribution
metadata.
Existing tests that asserted the DONE-on-failure behaviour are updated to
assert FAIL; they enshrined the defect. `tests/test_no_false_success.py`
adds the direct regressions, and every production change was reverted
individually to confirm the matching test fails against pre-fix code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
ActionType.DONEis a successful terminal outcome — a runner that sees it ends the episode as complete. Every parse failure and every conversion failure in this package produced exactly that value. So "the model said it finished the task" and "we could not read the model's output at all" were the same Action.This is the canonical shape the sweep was hunting: a failure rendered as a successful result.
ActionType.FAILalready existed in the enum for exactly this case and was never used.1.
openadapt_types/parsing.py— all 19 failure pathsThe
_done()helper backing every failure return logged a warning and then returnedAction(type=ActionType.DONE). The module docstring documented this as intended behaviour, which made it worse: a logged failure returning a success-shaped value is the same bug with better documentation.Now
_failed()returnsAction(type=ActionType.FAIL)carrying the reason in bothreasoningandraw[PARSE_ERROR_KEY].This also repairs a second-order bug in
parse_action, which usedresult.type != ActionType.DONEas its own success test:A model that genuinely emitted
{"type": "done"}was treated as a parse failure and sent down the DSL path. WithFAILas the failure marker, the check is now correct and unambiguous.2.
openadapt_types/_compat.py— the converters were worsePresent in
from_benchmark_actionandfrom_ml_action;from_omnimcp_action_decisiondid the same viatype_map.get(action_type_str, ActionType.DONE). A recorded step with a missing or future action type became a successful task completion in converted training/eval data, silently. All three now return FAIL withraw[UNCONVERTIBLE_ACTION_KEY]and the source dict. A real omnimcp"finish"still converts toDONE.3.
ComputerState.get_children—[]meant two thingsget_children("does_not_exist")returned[], identical to a node that genuinely has no children. A caller with a stale or mistyped node id observed "this element has no children" instead of "this element is not here". An unknown node now raisesKeyError.get_nodeis unchanged — it documentsNonefor not-found and its callers handle it.4.
__version__was a stale literal__version__ = "0.1.0"against a package on 0.6.3. Now read from installed distribution metadata; an unmeasurable version reportsunknownrather than a confident wrong number.Test changes
16 existing tests asserted the DONE-on-failure behaviour — they enshrined the defect and are updated to assert
FAIL. Because they now expectFAIL, they fail against pre-fix code, which is itself the mutation proof.tests/test_no_false_success.pyadds the direct regressions.Mutation checks
Each production change was reverted individually and the matching test confirmed to fail:
_failed()->return Action(type=ActionType.DONE)TestParseFailureIsNotCompletionassert <ActionType.DONE: 'done'> is <ActionType.FAIL: 'fail'>_compat.pyDONE fallbacksTestCompatConversionFailureIsNotCompletionassert <ActionType.DONE: 'done'> is <ActionType.FAIL: 'fail'>get_children->return []test_children_of_an_unknown_node_raisesFailed: DID NOT RAISE <class 'KeyError'>__version__ = "0.1.0"test_reported_version_matches_installed_distributionAssertionError: assert '0.1.0' == '0.6.3'One test in the new file,
test_parse_action_does_not_retry_a_successful_done_as_dsl, passes against pre-fix code — the pre-fix retry produced the same value by a wasteful route. It is a contract test, not a regression proof, and is labelled as such here rather than being presented as evidence.Verification
uv sync --locked --extra dev && uv run pytest-> 206 passed.uv.lockuntouched.Candidates examined and dismissed
control_overlay_tracking.pytracking_for_observation/event_for_media_frame/tracking_for_media_framereturningNone(5 sites) — deliberate and correct. These implement the AGENTS.md rule to omit a target rectangle rather than reconstruct one when the exact binding is absent, and they usecompare_digestfor the identity check.Nonehere means "no verified binding", which is precisely the representable distinction this sweep asks for.HumanDecisionReceiptV1.verify_hmacreturningFalsefor an unsigned receipt — documented ("An unsigned receipt never verifies") and the safe direction; an unsigned receipt genuinely does not verify._safe_floatreturningNone— an internal helper whose every call site checks the result and routes to a failure return.ComputerState.get_nodereturningNone— documented, and its callers all branch on it.scripts/verify_release_lock.pyreturn 0— reached only afterverify_release_lock()has raised or passed; the failure path callsparser.exit(1, ...)._make_targetreturningNonewhen every field is absent — a genuine "no target specified", not a failed computation.Fake-gate inventory
.github/workflows/contains no|| true,continue-on-error,exit 0,set +e, or swallowed step.test.ymlrunsuv sync --locked && uv run pytestand fails properly;--lockedis a real gate. One observation, not fixed here: there is no lint job in CI for this repo, only pytest.