Remoting: long-poll wait endpoint (with session ownership enforcement)

[michaellwest]

Problem

Wait-RemoteScriptSession and Wait-RemoteSitecoreJob poll by re-submitting a PowerShell scriptblock to the remoting endpoint every N seconds. Each poll pays the full HTTP + JWT validation + policy scan + runspace setup overhead and burns one request against the API Key's throttle budget.

Two real-world consequences:

1. Rate-limit pressure. Polling a long job at 1-second intervals against a key with RequestLimit=60/minute exhausts the budget in 60 seconds. The wait silently fails or surfaces as a 429 after the fix in feature/stream-fix.
2. ConstrainedLanguage incompatibility. Wait-RemoteSitecoreJob's polling script uses [Sitecore.Jobs.JobManager]::GetJob(...) and [Sitecore.Handle]::Parse(...) - .NET static method invocations that ConstrainedLanguage blocks regardless of AllowedCommands. Operators with restrictive policies cannot use the cmdlet.

A third concern surfaced while reviewing this: the existing per-poll script path also doesn't enforce session ownership. ScriptSessionManager.GetSession(sessionId, ...) returns whatever session matches the id; any authenticated caller who guesses or observes a session id can attach to a session another identity created. Pre-existing gap, but the new long-poll endpoint would inherit it. Fixing it alongside.

Proposed design

New wire route: GET /-/script/wait/

Added to the existing RemoteScriptCall.ashx.cs dispatcher. No new handler file, no new config, no new assembly - ships with the normal SPE deployment pipeline.

GET /-/script/wait/?sessionId=X&jobId=Y&jobType=scriptsession|sitecore&timeoutSeconds=30

Auth: same JWT + API Key flow as every other remoting request (AuthenticateRequest). Throttle: counts as one request against the key's budget regardless of hold time.

Response (200 regardless of isDone):

{ \"isDone\": true, \"status\": \"Idle|Busy|Done|Failed|NotFound\", \"name\": \"...\", \"elapsedSeconds\": 17 }

401 / 403 / 429 on auth / policy / throttle failures - same as existing endpoints.

Implementation notes

• Handler becomes async (HttpTaskAsyncHandler). Existing sync routes still work - ProcessRequestAsync delegates to the existing ProcessRequest for anything that isn't the new route.
• Internal 200 ms Task.Delay poll loop inside ProcessWaitAsync, checking IJobManager.GetJob(handle) or ScriptSessionManager.GetSession(id, ...) for state transitions.
• Not event-based. Subscribing to JobManager / ScriptSessionManager internals is tight coupling across the Spe.Abstractions / Spe.Sitecore92 boundary; 200 ms polling inside one async handler is already sub-second to the caller without paying per-call HTTP overhead.
• Max hold time: 60 s. timeoutSeconds clamped to 1..60. Client loops on timeout.
• Uniform response for unknown ids: {\"status\":\"NotFound\",\"isDone\":true} with 200. Prevents session-id enumeration via 404 vs 200 probing.

Client changes

• New internal helper modules/SPE/Invoke-RemoteWait.ps1: builds the GET URL, issues the request, handles the 404 fallback signal.
• Wait-RemoteScriptSession: uses Invoke-RemoteWait for polling; keeps the existing Invoke-RemoteScript { Receive-ScriptSession } for receive-after-done.
• Wait-RemoteSitecoreJob: uses Invoke-RemoteWait only. No more .NET static calls in a scriptblock - closes the CLM gap.
• Back-compat fallback: on 404 from the wait endpoint, fall back to the legacy per-poll scriptblock path for that session. One verbose log line per session so operators see the downgrade.

Session ownership enforcement (expanded scope)

• Capture creator identity at session creation time. Add CreatedByIdentity on ScriptSession or equivalent. For API Key auth, store the API Key name. For config-based shared-secret auth, store the username.
• Verify on every subsequent session access (existing script-execute endpoint + new wait endpoint). Mismatch returns 403 with X-SPE-Restriction: session-not-owned.
• Backward compatibility: sessions are in-memory, app-domain-bound. App recycle (automatic on SPE deploy) wipes them. No migration needed - pre-existing sessions are gone after the deploy that introduces this change. Clean break.

Acceptance criteria

• GET /-/script/wait/ route responds under the existing auth / throttle / policy pipeline.
• Wait-RemoteScriptSession completes a short-running -AsJob within expected time (no 429 on tight budgets).
• Wait-RemoteSitecoreJob works under ConstrainedLanguage + narrow AllowedCommands policy (no scriptblock shipped).
• Accessing a session created by identity A from identity B returns 403 on both the execute endpoint and the wait endpoint.
• Old client against new server: existing per-poll flow still works (no regression).
• New client against old server: 404 triggers automatic fallback with a single verbose log line per session.
• Integration test: Wait-RemoteSitecoreJob against a Constrained policy (currently fails, should pass).
• Unit test: mocked HttpMessageHandler verifies long-poll + 404 fallback + 429 retry paths.

Out of scope

• Event-driven notification from JobManager (version-coupling).
• WebSocket / SSE push (requires IIS WebSocket module - violates the constraint of no additional server-side install).
• Combining wait + receive into one response (keeping the receive pipeline separate preserves its auditability).
• New throttle bucket class for long-polls (existing bucket is fine).

Related

• Follow-up to feature/stream-fix which added 429/503 retry, policy stream-baseline, and server-side stream capture.
• Supersedes the tracking note under "Deferred UX / compat items" in .claude_worklog.md that flagged the Wait-RemoteSitecoreJob CLM incompatibility.

[michaellwest]

Follow-up notes during implementation

1. 200 ms poll interval was chosen heuristically, not measured

The Task.Delay(200) cadence inside ProcessWaitAsync was picked by judgment, not profiling. Reasoning:

• Upper bound on "job flipped to Done" -> "client notified" latency: ~200 ms. Low enough for interactive feel.
• At 200 ms, each active waiter issues <=5 in-memory state checks/sec against IJobManager.GetJob(handle) / ScriptSessionManager.GetSessionIfExists(...). Both are cheap dictionary/cache reads.
• Worst-case probes per 30 s hold: ~150, all on a single async task with no thread pinning.

What was not done: actual profiling of JobManager lock contention under many concurrent waiters, and comparison against other cadences (100 / 500 / 1000 ms) under load.

If the interval needs revisiting:

• Anywhere in the 100-500 ms range is defensible without data.
• A future iteration could make it configurable via a setting (e.g. Spe.Remoting.WaitPollIntervalMs) or go adaptive (100 ms for the first 2 s, 500 ms after) to cover both short and long jobs.

2. Session-ownership info leak (minor, narrow)

The TryEnforceSessionOwnership path uses a "claim on first use" model:

scenario	response
sessionId never claimed	200 (the current identity becomes the owner)
sessionId owned by current caller	200 (no header)
sessionId owned by a different identity	403 + X-SPE-Restriction: session-not-owned

Asymmetry: an attacker who already knows a specific sessionId can tell it is active and claimed by someone else (403) vs unused (200). Narrow (requires brute-forcing a ~128-bit GUID), but a real side channel.

Mitigation options for a follow-up:

1. Drop X-SPE-Restriction: session-not-owned on mismatch. Removes the header-level signal; 403 status and body message still carry the semantic. Operator diagnostics slightly degraded.
2. Pretend unknown sessions also fail: return 403 for every non-owner access. Uniform response, but breaks the "client generates GUID and server claims on first call" flow unless a dedicated create endpoint is added.
3. Accept the narrow leak as the GUID brute-force barrier is already sufficient.

Shipping current branch with behaviour #3 + this note. Happy to pivot to #1 if operators care.