Cherry-pick [SDKv1] Self-heal catalog cache on BYOM cache-miss (#760)

[bmehta001]

The SDKs cache the catalog in memory with a 6-hour TTL. When a user manually drops a model directory (BYOM) into the cache while the SDK is running, the SDK''s in-memory map is stale: Core returns the BYOM id from get_cached_models and get_model_list IPCs, but the SDK filters it out for getModel/getModelVariant because its modelIdToModelVariant map is still stale.

Self-heal on cache-miss

Force-refresh path in the four user-facing entry points:

• get_cached_models / get_loaded_models: when any id in the fresh list returned by Core does not resolve against the local map, force a catalog refresh and re-resolve only the unresolved ids.
• get_model(alias) / get_model_variant(id): on miss, force a refresh and retry the lookup once.

A UpdateModels(ct) warm-up call was added to the C# get_model / get_model_variant paths for consistency with the other SDKs (not strictly required for BYOM).

Catalog refresh is now incremental

Previously every catalog refresh was clear-and-rebuild, which orphaned every Model / ModelVariant wrapper a caller was holding. After this PR, refresh preserves wrapper identity for ids that survive:

• C# / JS / Python: Catalog.UpdateModels reuses existing ModelVariant wrappers and calls RefreshInfo(info) to surface fresh metadata in place. Model wrappers are reused per alias and RefreshVariants swaps the backing variant list by reference (no torn reads). Evicted ids are dropped from the maps.
• Rust: apply_model_list reuses the same Arc<Model> across refreshes when the (id, cached) fingerprint is unchanged, so the inner AtomicUsize selected-variant index survives.

Explicit SelectVariant() choices are preserved trivially because the wrapper identity is preserved.

Limitations

If the previously-selected variant is removed by a refresh, SelectedVariant keeps pointing at the dropped wrapper and callers must explicitly re-select - pre-existing issue that would take ~30 lines per SDK to fix, but unlikely to happen.

Tests

New CatalogSelfHealTests and CatalogIncrementalRefreshTests (+ JS / Python equivalents) cover: self-heal on cache miss, no refresh on empty/whitespace input, identity + info propagation across refreshes, selection survival, and add/remove transitions.

Validated against a real BYOM directory dropped into the cache at runtime.

=================

More detailed description:
_cachedModels (disk) and _cachedInfo (Azure metadata) are always populated at the start

FLCore Changes:

• Disk gets scanned in
• RefreshLocalCachedModelsAsync (thread-safe function bc lock) in AzureModelCatalog.cs — refreshes the in-memory _cachedModels dictionary from disk.
• "We expect users will not manually delete an Azure-cataloged model, so we do not make a new trip to Azure Catalog or invalidate _cachedInfo here."
• New BYOM ids get inserted with Model = null (we assume they do not match a model in Azure Catalog), so no Azure call.
• GetCurrentCachedModelsAndPaths (existing) to extract modelId.

In FLCore, RefreshLocalCachedModelsAsync is called by GetCachedModelsAsync, GetModelsAsync, and RemoveCachedModelAsync. It will be called by GetModelInfoAsync only if we have not created a BYOM model stub for the catalog, but it will be cached after that.

| Core method | When RefreshLocalCachedModelsAsync is called |

|------------------------------------------|------------------------------------------------------------|
| GetCachedModelsAsync | Always |
| GetModelsAsync | Always |
| RemoveCachedModelAsync | Always |
| GetModelInfoAsync | At most once each time we see a new BYOM model |

GetModelInfoAsync ONLY calls RefreshLocalCachedModelsAsync if we looked up a BYOM model by calling (get_model_path, load_model, unload_model IPC handlers called by SDKs or OpenAI API and by GetChatClientAsync on every chat completion), and we have not created a BYOM model stub for the catalog. Once we have created a BYOM model stub, it is cached for the rest of the session, so we only validate Directory.Exists for the BYOM model (1 syscall is much quicker than scanning whole directory) after that to ensure it has not been deleted. (If it has been deleted, we return null for AzureFoundryLocalModel)

HTTP Method changes:
| HTTP endpoint | Core method backing it |

|------------------------------------------------|--------------------------------------------------------------------------------------------------------------| | GET /v1/models | GetCachedModelsAsync() (always rescans disk after this PR) + then will only check if directory for BYOM model exists in GetModelInfoAsync |
| GET /openai/models | GetCachedModelsAsync() (always rescans disk after this PR) |
| POST /v1/chat/completions (and friends) |
GetModelInfoAsync(id_or_alias) — validates Directory.Exists, falls through to refresh + synth if needed |
| GET /openai/load/{model} | same GetModelInfoAsync lookup | | GET /openai/unload/{model} | same GetModelInfoAsync lookup |

SDK changes:

| SDK method | Scan disk trigger |

|-----------------------------------------|------------------------------------------------------------| | catalog.GetModelAsync(alias) | Alias not in local map and if we exceed TTL of 6 hours* |
| catalog.GetModelVariantAsync(id) | Id not in local map and if we exceed TTL of 6 hours* |
| catalog.GetCachedModelsAsync() | Always **|
| catalog.GetLoadedModelsAsync() | If load-manager's list contains an
id missing from local map (unlikely, but may be possible if multiple
processes using SDK) |
| model.IsCachedAsync() | Always, bc this is fresher than
model.Info.Cached |

* we may end up scanning twice if we have a. once if we have hit the TTL and would have anyway and b. because of this PR, after scanning, if we still can't find the model (likely bc of a typo), we will scan again due to force refresh after cache miss

** - we will end up scanning twice - once to get the model name (string) for GetCachedModels, another time because we call UpdateModels(ct, force: true) to get the ModelInfo

Flow for GetModelAsync

Program.cs
  └─ catalog.GetModelAsync(byomAlias)
       │  sdk/cs/src/Catalog.cs:73 → GetModelImplAsync at line 157
       │   ├─ UpdateModels(ct, force: false)                # within 6h TTL → no-op
       │   ├─ _modelAliasToModel.TryGetValue(alias, …)→null # cache miss
       │   ├─ UpdateModels(ct, force: true)                 # SELF-HEAL
       │   │    └─ _coreInterop.ExecuteCommandAsync("get_model_list")
       │   │         │  IPC → foundry-local-service
       │   │         ▼
       │   │       Core AzureModelCatalog.GetModelsAsync(ct)
       │   │         ├─ base.GetModelsAsync(ct)                       # uses 4h Azure cache; no extra catalog calls
       │   │         ├─ RefreshLocalCachedModelsAsync(ct)             # ALWAYS re-scans disk
       │   │         │    └─ GetCurrentCachedModelsAndPaths()
       │   │         │         └─ Directory.GetDirectories(_cacheDirectory, *, AllDirectories)
       │   │         │              + for each dir: read inference_model.json → modelId
       │   │         ├─ drop stale Local stubs whose dir was deleted
       │   │         ├─ append cache-only stubs for new BYOMs
       │   │         └─ return combined list
       │   ▼
       └─ _modelAliasToModel.TryGetValue(alias, …) → Model

Flow for GetModelVariantAsync is similar; we just access a different map

Program.cs
  └─ catalog.GetModelVariantAsync(byomId)
       │  sdk/cs/src/Catalog.cs:81 → GetModelVariantImplAsync at line 187
       │   ├─ UpdateModels(ct, force: false)                # within 6h TTL → no-op
       │   ├─ _modelIdToModelVariant.TryGetValue(id, …) → null # cache miss
       │   ├─ UpdateModels(ct, force: true)                 # SELF-HEAL (same IPC + disk rescan as A1)
       │   ▼
       └─ _modelIdToModelVariant.TryGetValue(id, …) → Model 

Flow for GetCachedModelsAsync must scan the disk twice - once to return strings for model names from FLCore's GetCachedModelsAsync and once to fetch latest GetModelsAsync (which has Model information). GetLoadedModelsAsync is similar.

Program.cs
  └─ catalog.GetCachedModelsAsync(byomId)
       │  sdk/cs/src/Catalog.cs:73 → GetModelImplAsync at line 157
       │   ├─ UpdateModels(ct, force: false)                # within 6h TTL → no-op
       │   ├─ _modelAliasToModel.TryGetValue(alias, …)→null # cache miss
       │   ├─ UpdateModels(ct, force: true)                 # SELF-HEAL
       │   │    └─ _coreInterop.ExecuteCommandAsync("get_model_list")
       │   │         │  IPC → foundry-local-service
       │   │         ▼
       │   │       Core AzureModelCatalog.GetModelsAsync(ct)
       │   │         ├─ base.GetModelsAsync(ct)                       # uses 4h Azure cache; no extra catalog calls
       │   │         ├─ RefreshLocalCachedModelsAsync(ct)             # ALWAYS re-scans disk
       │   │         │    └─ GetCurrentCachedModelsAndPaths()
       │   │         │         └─ Directory.GetDirectories(_cacheDirectory, *, AllDirectories)
       │   │         │              + for each dir: read inference_model.json → modelId
       │   │         ├─ drop stale Local stubs whose dir was deleted
       │   │         ├─ append cache-only stubs for new BYOMs
       │   │         └─ return combined list
       │   ▼
       └─ _modelAliasToModel.TryGetValue(alias, …) → Model

Get Loaded models

Program.cs
  └─ model.LoadAsync(ct)                                    # sdk/cs/src/Detail/Model.cs:153
       └─ SelectedVariant.LoadAsync(ct)                     # sdk/cs/src/Detail/ModelVariant.cs:95
            └─ _modelLoadManager.LoadModelAsync(modelId, customPath=null, ct)  # ModelManager.cs:65
                ├─ _modelCatalog.GetModelInfoAsync(modelId)
                │    │  AzureModelCatalog.GetModelInfoAsync override
                │    ├─ base.GetModelInfoAsync(modelId)             # miss for new BYOM
                │    ├─ RefreshLocalCachedModelsAsync(ct)           # SELF-HEAL: rescan disk
                │    │    └─ GetCurrentCachedModelsAndPaths()       # finds the new BYOM dir
                │    └─ TrySynthesizeByomStubAsync(modelId)         # writes stub into _cachedInfo
                │         └─ returns stub                           # cached for next time
                ├─ modelInfo != null → proceed to load
                └─ _modelCatalog.GetModelPathAsync(modelId)         # re-uses cached stub
                        → InferenceModel constructed from dir
                        returns Status.Success  ✓ FIX_ACTIVE

---

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	Jun 4, 2026 7:56pm