Skip to content

fix(genai): isolate EPContext compile from og.Model load (#1087)#1103

Merged
DingmaomaoBJTU merged 4 commits into
mainfrom
dingmaomaobjtu-isolate-genai-model-load
Jul 14, 2026
Merged

fix(genai): isolate EPContext compile from og.Model load (#1087)#1103
DingmaomaoBJTU merged 4 commits into
mainfrom
dingmaomaobjtu-isolate-genai-model-load

Conversation

@DingmaomaoBJTU

Copy link
Copy Markdown
Collaborator

Problem

winml perf -m <bundle> --compile --runtime winml-genai --device npu can native-crash (0xC0000005 / 0xC0000374) at og.Model() load, before the first token — even when every stage is already compiled and cached.

Root cause: GenaiSession.load() runs the crash-prone EPContext compile orchestration (_prepare_compiled_bundle, which spawns and reaps per-stage QNN compile workers that native-fault during interpreter/driver teardown) and then loads og.Model in the same process. Orchestrating those crash-prone subprocesses leaves the orchestrating process's native accelerator state fragile, so the subsequent model load faults.

The proven workaround was to compile in one process (--compile) and then load _compiled/ in a fresh process (no --compile).

Fix

Run the compile orchestration in a throwaway spawned subprocess: it writes _compiled/ to disk and exits, keeping the parent process — the one that loads and runs og.Model — pristine. This mirrors the two-step workaround (the load process never runs the poison-prone orchestration) while keeping the in-process streaming/timed generation API intact for every GenaiSession consumer, with no cross-process result serialization.

The parent only spawns/reaps one clean orchestrator child (which exits 0); it never directly reaps the crashing per-stage QNN workers, so it stays clean for the model load. The child writing _compiled/ to disk is the durable hand-off — the queue only carries the resolved load-dir path as a fast-path signal.

  • Add module-level _prepare_compiled_bundle_worker (spawn target): reconstructs a compile-only GenaiSession, runs _prepare_compiled_bundle, posts ("ok", <load_dir>) / ("error", <detail>).
  • Add GenaiSession._prepare_compiled_bundle_isolated: spawns the worker, drains its single result, and on any child failure falls back to the on-disk _compiled/ bundle if present — never an in-process compile (which would reintroduce the poisoning), else the original bundle dir.
  • load() routes the --compile path through the isolated variant; the override-only path (JSON rewrite + file mirror, no native accelerator work) stays in-process.
  • Docs: winml perf --compile now runs compile + generation end-to-end in one command; the manual two-step is kept as an optional way to reuse a prebuilt _compiled/ bundle.

Verification

  • tests/unit/session/test_genai_session.py — 142 pass (12 new: worker posts ok/error, isolated wrapper returns path + drains-on-exit + fallbacks, load() routing).
  • tests/unit/commands/test_perf_genai.py — 54 pass (the only compile=True consumer, unchanged by design).
  • ruff check clean on both changed Python files.

Note: the actual native-crash fix is validated by design (it mirrors the proven cross-process workaround) plus unit tests with mocked subprocess machinery — I don't have NPU hardware to reproduce end-to-end.

Fixes #1087.

@DingmaomaoBJTU DingmaomaoBJTU requested a review from a team as a code owner July 13, 2026 07:53
Comment thread src/winml/modelkit/session/genai_session.py Fixed
`winml perf --compile --runtime winml-genai --device npu` could
native-crash (0xC0000005 / 0xC0000374) at og.Model() load. GenaiSession.load()
ran the crash-prone EPContext compile orchestration (spawning + reaping
per-stage QNN compile workers that native-fault on teardown) and then loaded
og.Model in the SAME process. Orchestrating those crash-prone subprocesses
leaves the process's native accelerator state fragile, so the subsequent model
load faults before the first token -- even when every stage is already compiled.

Run the compile orchestration in a throwaway spawned subprocess: it writes
_compiled/ to disk and exits, keeping the parent process (which loads and runs
og.Model) pristine. This mirrors the proven two-step workaround (compile in one
process, load the compiled bundle in a fresh one) while keeping the in-process
generation API intact for every GenaiSession consumer.

- Add module-level _prepare_compiled_bundle_worker (spawn target).
- Add GenaiSession._prepare_compiled_bundle_isolated: spawns the worker, drains
  its single (status, load_dir) result, and falls back to the on-disk _compiled/
  bundle (never an in-process compile) if the child reports nothing.
- load() routes the compile path through the isolated variant; the override-only
  path (no compilation, no native work) stays in-process.
- docs: winml perf --compile now runs end-to-end in one command.
@DingmaomaoBJTU DingmaomaoBJTU force-pushed the dingmaomaobjtu-isolate-genai-model-load branch from d432da4 to 19b0366 Compare July 13, 2026 08:10
Comment thread src/winml/modelkit/session/genai_session.py Fixed
Comment thread docs/samples/qwen3-genai-bundle.md Outdated
github-actions Bot added 2 commits July 14, 2026 14:48
CodeQL py/empty-except flagged the queue_mod.Empty handler in the isolated
compile drain as having no explanatory comment. The pass is intentional: when
the worker exits without posting a result, status stays None and the failure
path reports the error without an in-process fallback compile (#1087).
…ndle

Address review feedback: remove the '--compile compiles and generates in one
command' admonition from the Qwen3 genai-bundle sample.
Comment thread src/winml/modelkit/session/genai_session.py Outdated
Comment thread src/winml/modelkit/session/genai_session.py Outdated
…ed_bundle (#1087)

Address review feedback on the isolated-compile path.

Stale-bundle guard: add a per-run completion marker (.winml_bundle_complete)
written as the final step of _prepare_derived_bundle. The isolated-compile
parent removes any stale marker before spawning the child and, on the
post-failure fallback, reuses an existing _compiled/ only when this run's
marker proves it was written for the current effective config + stage inputs.
Previously any leftover _compiled/genai_config.json from an earlier run
satisfied the fallback, so a child that failed before rewriting the derived
bundle could silently load stale EPContext artifacts instead of surfacing the
failure.

Naming: rename _prepare_compiled_bundle -> _prepare_derived_bundle (and its
_isolated/_worker variants). The method also runs the override-only path
(compile=False), where it rewrites routing without compiling anything, so
"compiled" was misleading.
@DingmaomaoBJTU DingmaomaoBJTU merged commit 3aff9f4 into main Jul 14, 2026
8 checks passed
@DingmaomaoBJTU DingmaomaoBJTU deleted the dingmaomaobjtu-isolate-genai-model-load branch July 14, 2026 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

genai --compile: model load can native-crash when compile and load share one process; isolate og.Model load into its own subprocess

3 participants