Skip to content

Lazy runner design: defer engine load#91

Merged
carinapeng merged 3 commits into
apple:mainfrom
carinapeng:carina/fm-design-lazy-runner-defer-engine-load
Jul 9, 2026
Merged

Lazy runner design: defer engine load#91
carinapeng merged 3 commits into
apple:mainfrom
carinapeng:carina/fm-design-lazy-runner-defer-engine-load

Conversation

@carinapeng

Copy link
Copy Markdown
Contributor

Purpose

Introduces lazy inference-engine lifecycle management for the FoundationModels adoption layer. Instead of CoreAILanguageModel holding a live engine from construction, engine load/unload is now owned by a new ModelResources type: lazy by default, loaded on first respond(...) (or explicitly via load()), borrowed per-generation, and releasable via unload(). This lets callers construct a model cheaply, query its on-disk footprint, and control when the (expensive) engine actually loads.

Changes

  • ModelResources (new) — owns the engine's load/unload lifecycle. Concurrent callers share a single in-flight load; withEngine { } borrows the engine for the duration of a generation and defers teardown if an unload() races an active borrow. Shared per-configuration via a weak registry.
  • CoreAILanguageModel — now builds directly from LanguageBundle (removes the CoreAIRunner.makeLanguageModel() hop). Adds:
    • LoadMode { .lazy, .eager } (default .lazy; .eager loads tokenizer + engine concurrently)
    • load() / unload() / estimatedSizeOnDiskBytes
    • guided-generation capability gating that works before the engine is resident (isGuidedGenerationSupported)
    • reasoning models default to a higher max-token budget
  • prewarm — now kicks off the engine load in the background (non-blocking) instead of a synchronous semaphore wait.
  • ResourceManaging — moved from CoreAIDiffusionPipeline/Components/ to CoreAIShared/Runtime/; diffusion pipelines updated to import it from the shared module.
  • FileSize (new)URL.recursiveFileSizeInBytes() helper backing estimatedSizeOnDiskBytes.
  • Tests — ModelResourcesTests (load/unload/borrow races via an injectable loader), FileSizeTests, and PublicInterfaceTests additions.

Comment thread swift/Sources/CoreAILanguageModels/LanguageModel/CoreAILanguageModel.swift Outdated
Comment thread swift/Sources/CoreAILanguageModels/LanguageModel/CoreAIRunner.swift
Comment thread swift/Sources/CoreAILanguageModels/LanguageModel/CoreAILanguageModel.swift Outdated
@carinapeng

Copy link
Copy Markdown
Contributor Author

CoreAILanguageModel.swift syncing with recent PRs

@carinapeng carinapeng requested a review from kevchengcodes July 8, 2026 23:22
@carinapeng carinapeng merged commit eb3998e into apple:main Jul 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants