fix: deliver mid-action cancels through SessionCancelHandle#312
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| # openjd-expr = { path = "../../openjd-rs/crates/openjd-expr" } | ||
| # openjd-model = { path = "../../openjd-rs/crates/openjd-model" } | ||
| # openjd-sessions = { path = "../../openjd-rs/crates/openjd-sessions" } | ||
| [patch.crates-io] |
There was a problem hiding this comment.
This uncomments the local-dev [patch.crates-io] overrides, but the comment block directly above (lines 51-53) states these must stay commented out for committed work and CI: "the published wheel is built against the crates.io versions, which is what the Cargo.lock file pins."
With this active, any build outside a machine that has a sibling ~/openjd-rs checkout at ../../openjd-rs/crates/... will fail to resolve the path dependencies — this breaks CI and the published-wheel build. Consistent with that, Cargo.lock now drops the source/checksum for openjd-expr, confirming it is being resolved from the local patch rather than crates.io.
This looks like an accidental commit of a local iteration state. It should be reverted (re-comment the block, and remove the corresponding [patch.crates-io] addition in the workspace Cargo.toml) before merge.
cancel_action previously failed with 'session is busy' whenever an action was running, because Session::cancel_action needs &mut Session and a background action thread owns the Session for exactly that window - the primary window in which callers want to cancel. openjd-sessions 0.4.0 introduced Session::cancel_handle(), a thread-safe handle that delivers a cancel to the in-flight action without touching the Session, following the action's declared cancelation method on both the direct and cross-user helper paths. Obtain the handle at construction and route the taken-session case through it. The handle cannot perform the Running -> Canceling state transition (it does not own the Session), so the snapshot is updated to Canceling directly, keeping the state visible to Python-side pollers until the terminal state lands. Signed-off-by: Sean Tang <171081544+seant-aws@users.noreply.github.com>
ea09394 to
493ff96
Compare
seant-aws
changed the title
| // reflect it in the snapshot so Python-side pollers observe the | ||
| // cancel-in-progress phase until the terminal state lands. | ||
| let mut snap = lock_recover(&self.snapshot); | ||
| snap.state = SessionState::Canceling; |
There was a problem hiding this comment.
The snap.state = Canceling write here can be clobbered by (or can clobber) the terminal state that run_action writes concurrently, since the two use independent lock acquisitions with no ordering guarantee.
Consider this interleaving:
cancel_actionlockssession, seesNone(taken), drops the guard.cancel_handle.cancel(...)returnstrue.- The background action returns;
run_actionrestores the session and then writes the terminal state (Ready+ terminalaction_status) to the snapshot. cancel_actionnow locks the snapshot and overwritesstate = Canceling.
The snapshot is then stuck at Canceling even though the action has finished and the session is restored. A Python poller watching state would never observe the terminal transition and could hang.
To make the Canceling write safe it should be conditional (e.g. only set Canceling if the current snapshot state is still Running) or otherwise coordinated with run_action's restore/update so a terminal state cannot be overwritten.
3 participants
What was the problem?
Session.cancel_action()failed with "session is busy" whenever an action was actually running: the RustSession::cancel_actionneeds&mut Session, but a background action thread owns theSessionfor exactly that window — the primary window in which callers want to cancel. Mid-action cancels (service-initiated cancels andtime_limitexpiries in Deadline Cloud's worker agent) never reached the job process.What is the fix?
Obtain a
SessionCancelHandleat session construction and route the taken-session case throughhandle.cancel(time_limit, mark_action_failed). The handle delivers the cancel to the in-flight action without touching theSession, following the action's declared cancelation method on both the direct and cross-user helper paths (openjd-sessions 0.4.0 also carries themin(time_limit, terminate_delay)grace capping from openjd-rs#258).One semantic note: the handle cannot perform the
Running → Cancelingsession-state transition (it does not own theSession), so the binding updates its state snapshot toCancelingdirectly — Python-side pollers keep observing the cancel-in-progress phase until the terminal state lands, preserving the contract that openjd-sessions-for-python#331 (merged) relies on.How was this change tested?
cargo check/clippy --all-targets/fmtclean;cargo test(build/link gate) passes against the published crates.ioopenjd-sessions 0.4.0— no patch overrides.test/openjd/expr/test_operation_limit.pyreproduce identically on pristinemainlinein the same environment (pre-existing, unrelated).