[Open Data] Add instance size flexibility dataset from Azure Catalogs API

[RolandKrummenacher]

Summary

Adds a new instance size flexibility (ISF) open data dataset, sourced from the authenticated Azure Reservations Catalogs API, to replace the two deprecated static CSVs hosted on ccmstorageprod.blob.core.windows.net (AutofitComboMeterData.csv and isfratioblob.csv / aka.ms/isf). Microsoft is retiring those files — no updates after 9 May 2026, removed 30 Aug 2026.

Addresses #2090. This implements the open-data-generation approach discussed on the issue: the authentication complexity lives only in the generator, so downstream tools consume a public CSV and need no Azure credentials. Consumer repoints are tracked separately in #2200.

What's included (foundation only)

• src/scripts/Update-InstanceSizeFlexibility.ps1 — extracts ISF groups/ratios from Microsoft.Capacity/catalogs. Handles both REST (ReservationsAutofit*) and PS-cmdlet (InstanceSizeFlexibility*) property names, pagination with in-place 429/5xx retry, placeholder-SKU filtering, and a cache-merge that preserves SKUs the API drops. Defaults to the four ISF-bearing reserved resource types (VirtualMachines, BlockBlob, RedisCache, DedicatedHost) unioned across a broad region set.
• src/open-data/InstanceSizeFlexibility.csv — populated with 2,436 SKUs / 557 groups of live data (3-column schema: InstanceSizeFlexibilityGroup,ArmSkuName,Ratio).
• .github/workflows/opendata-instance-size-flexibility.yml — weekly refresh; opens a PR on change.
• README + docs download tile.

Coverage vs the deprecated files

Dataset	Groups	SKUs
New InstanceSizeFlexibility.csv	557	2,436
isfratioblob.csv (Optimization Engine)	434	2,524
AutofitComboMeterData.csv (Power BI, frozen since 2023)	132	721

Raw Microsoft ratios are kept as-is for drop-in parity with the deprecated files (normalization is available via -Normalize).

Schema (3 columns, by design)

The file uses the canonical Catalogs API ISF schema — InstanceSizeFlexibilityGroup, ArmSkuName, Ratio — which is a 1:1 match for the Optimization Engine's isfratioblob.csv. The Power BI AutofitComboMeterData.csv had additional columns (ResourceLocation, meter IDs, composite key, normalized SKU), but those are Cost Management connector artifacts not exposed by the Catalogs API, so the Power BI repoint (#2200) will join on ArmSkuName instead.

Authentication

The Catalogs API is authenticated (Microsoft.Capacity/catalogs/read). The workflow reuses the existing OIDC service principal from the aoe-prod environment (the same identity the AOE deployment workflows use), following the repo's azure/login@hf_447_release pattern — no new secrets to provision. The principal's subscription needs the Microsoft.Capacity resource provider registered (done). A one-time workflow_dispatch after merge confirms the principal can read the catalog.

Test plan

• Generator validated end-to-end against the live Catalogs API (response shape, property names, armSkuName fallback all confirmed)
• Discovery confirmed only 4 reserved resource types expose ISF ratios
• Ratios confirmed region-stable (merge reported Modified: 0 across all regions)
• Offline tests: multi-type/region union, dedup, cache-merge/preserve, 429-retry, placeholder filtering; script parses clean
• Reviewer spot-checks a few SKUs against the Azure Pricing Calculator
• workflow_dispatch after merge to confirm CI auth path

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR introduces a new open-data dataset for instance size flexibility (ISF) ratios by generating InstanceSizeFlexibility.csv from the authenticated Azure Reservations Catalogs API, replacing reliance on deprecated static CSV blobs.

Changes:

• Added a PowerShell generator script to fetch, normalize (optional), and cache-merge ISF ratios from Microsoft.Capacity/catalogs.
• Added the generated InstanceSizeFlexibility.csv open-data artifact and documented it in open-data docs/README.
• Added a scheduled GitHub Actions workflow to refresh the dataset weekly and open a PR when it changes.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/scripts/Update-InstanceSizeFlexibility.ps1	New generator that pages the Catalogs API and emits the ISF CSV (with cache-merge + optional normalization).
src/open-data/InstanceSizeFlexibility.csv	New published open-data dataset containing ISF group/SKU/ratio rows.
.github/workflows/opendata-instance-size-flexibility.yml	Weekly automation to refresh the dataset and open an update PR.
src/open-data/README.md	Documents the new ISF dataset, its source, and manual update instructions.
docs/open-data.md	Adds a download tile linking to the CSV in the latest release assets.

[RolandKrummenacher]

Addressed the Copilot review (commit pushed):

1. 429/5xx retry semantics (✅ fixed) — moved transient HTTP status handling inside the inner retry loop so a throttled/5xx page is retried in place without advancing $page or resetting the retry cap. Verified with a mock that returns 429 then 200: the page is retried once and the cap holds.
2. arm_sku_name_placeholder in the CSV (✅ fixed) — the live Catalogs API returned a placeholder name for one unreleased DedicatedHost SKU. The generator now skips any *placeholder* SKU name, and I removed the one row that had reached the published CSV (it was the sole member of its group, so a meaningless single-SKU ISF group went with it). CSV is now 2,436 SKUs.
3. Hidden spacer anchors in docs/open-data.md (ℹ️ no change) — these use style="visibility:hidden", which already removes them from the tab order and the accessibility tree, so they aren't surfaced to AT. They're also the established layout pattern used by every tile in this file, so changing only the new tile would be inconsistent. Happy to do a site-wide pass separately if desired.

On the column schema (raised in review discussion): the dataset intentionally uses the canonical three-column Catalogs API schema (InstanceSizeFlexibilityGroup, ArmSkuName, Ratio).

• This is a 1:1 match for the deprecated isfratioblob.csv (Optimization Engine) — that file was also exactly these 3 columns.
• The deprecated AutofitComboMeterData.csv (Power BI) had 9 columns, but the 6 extras (ResourceLocation, ComboMeter/HardwareMeter/SoftwareMeter meter IDs, CompositeKey, NormalizedSKU) are Cost Management connector artifacts — they are not ISF data and the Catalogs API does not expose them, so they can't be reproduced here regardless of column count.
• Consequence for the deferred Power BI repoint: the model currently joins via x_CommitmentDiscountKey (derived from those connector columns), so the repoint will need to join on ArmSkuName instead. I've documented this in src/open-data/README.md.

[RolandKrummenacher]

Update on the auth setup — no new secrets/SP needed.

The earlier note said maintainers would need to create an OIDC service principal and AZURE_* secrets. On closer inspection, the repo already has that identity: the aoe-prod environment holds AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_SUBSCRIPTION_ID (the same OIDC principal the AOE deployment workflows use), and it has no protection rules, so a scheduled job can use it.

I've updated the workflow to reuse that identity rather than create a parallel one:

• environment: aoe-prod
• the repo's established azure/login@hf_447_release + enable-AzPSSession: true pattern, with an explicit Az.Accounts install

So there's nothing to provision. The only residual is a one-time confirmation that the aoe-prod principal can read the Catalogs API on its subscription — i.e. it has Microsoft.Capacity/catalogs/read (a deployment principal with Contributor/Owner already does) and the subscription has the Microsoft.Capacity resource provider registered (done). Easiest way to confirm is a manual workflow_dispatch once this is merged.

If you'd prefer the open data workflow not to borrow the AOE-named environment, a maintainer can create a dedicated opendata environment with the same federated credential + secrets and I'll repoint to it — purely cosmetic.

[flanakin]

🤖 [AI][Claude Code] PR Review

Summary: Clean, well-structured addition that closely mirrors the established Update-CommitmentDiscountEligibility sibling pattern (cache-merge, paging, retry, weekly PR workflow). The earlier Copilot items (in-place 429/5xx retry, placeholder-SKU filtering) are already addressed in-tree, and the generator/CSV logic looks correct. Two non-blocking findings below.

⚠️ Should fix (1)

1. No changelog entry. The sibling Commitment discount eligibility dataset shipped with an ### [Open data] changelog section; this is a comparably substantial external-facing dataset (2,436 SKUs, new download tile, weekly workflow) with no entry in the current docs-mslearn/toolkit/changelog.md. Add an ### [Open data] entry under the next/unreleased version.

💡 Suggestions (1)

1. -Normalize is typed [bool] (requires -Normalize $true); [switch] would be more idiomatic PowerShell. Optional — the [bool] form does allow scripted toggling.

[github-actions]

⚠️ Open Data CI task failed: https://github.com/microsoft/finops-toolkit/actions/runs/28363541029

[helderpinto]

The earlier note said maintainers would need to create an OIDC service principal and AZURE_* secrets. On closer inspection, the repo already has that identity: the aoe-prod environment holds AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_SUBSCRIPTION_ID (the same OIDC principal the AOE deployment workflows use), and it has no protection rules, so a scheduled job can use it.

The aoe-prod principal belongs to a personal Entra ID tenant associated to a MSDN benefits subscription. Given that the ISF workflow is not going to create resources, I don't anticipate issues, but if the subscription type can be a problem, we should switch to a more permanent subscription type, such as PAYG/enterprise.

So there's nothing to provision. The only residual is a one-time confirmation that the aoe-prod principal can read the Catalogs API on its subscription — i.e. it has Microsoft.Capacity/catalogs/read (a deployment principal with Contributor/Owner already does) and the subscription has the Microsoft.Capacity resource provider registered (done). Easiest way to confirm is a manual workflow_dispatch once this is merged.

The aoe-prod principal can read the Catalogs API on its subscription and the subscription has the Microsoft.Capacity resource provider registered.

[RolandKrummenacher]

Two follow-ups:

Open Data CI failure was transient — now green. The failure wasn't in the data or generator: the Build Open Data step regenerates the Get-OpenData* helper functions and auto-commits them, and its git push was rejected (fetch first) because concurrent commits had moved the branch head mid-run. Re-ran the job with no concurrent push and both steps pass (Build Open Data ✓, Test Open Data ✓). The two earlier Open Data CI runs on this branch had also passed, confirming it was a one-off race. No generated function is committed (consistent with the sibling Commitment discount eligibility dataset — git commit -a skips the untracked generated files).

Thanks @helderpinto for confirming the aoe-prod principal can read the Catalogs API and that the subscription has Microsoft.Capacity registered — that clears the only residual, so the post-merge workflow_dispatch is now just a nice-to-have sanity check rather than a gate. Noted on the MSDN-benefits subscription: since this workflow is read-only (no resource creation, one lightweight catalog read per week), the subscription type shouldn't matter, but easy to point at a PAYG/enterprise sub later if you'd prefer.