Skip to content

Add FOCUS 1.4-preview ingestion (IngestionSetup_v1_4.kql)#2132

Draft
flanakin wants to merge 4 commits intoflanakin/focus14from
flanakin/focus14-phase4-ingestion-1.4
Draft

Add FOCUS 1.4-preview ingestion (IngestionSetup_v1_4.kql)#2132
flanakin wants to merge 4 commits intoflanakin/focus14from
flanakin/focus14-phase4-ingestion-1.4

Conversation

@flanakin
Copy link
Copy Markdown
Collaborator

@flanakin flanakin commented May 6, 2026

Summary

Phase 4 of FOCUS 1.4 work (parent: #2120). Adds preview-grade FOCUS 1.4 ingestion. FOCUS 1.4 is still in working_draft so this schema is labeled preview and may change before the spec ratifies.

Stacked on phase 3 (#2130).

Files

  • NEW: `IngestionSetup_v1_4.kql` (copied from v1_3 with renames + 1.4 deltas).
  • MODIFIED: `IngestionSetup_RawTables.kql` — `ContractCommitment_raw` gains the 14 new FOCUS 1.4 columns.
  • MODIFIED: `app.bicep` and `.build.config` — wires `IngestionSetup_v1_4.kql` into the deployment.

FOCUS 1.4 deltas applied

Cost and Usage

  • Removed `ProviderName` and `PublisherName` from `Costs_final_v1_4` table and projection (deprecated in 1.3, removed in 1.4).
  • Internal use of `ProviderName`/`PublisherName` retained inside the transform so `ServiceProviderName`/`HostProviderName` fall back to those values when source data still uses the old columns.

Contract Commitment (added 14 columns)

  • BenefitCategory, ContractCommitmentApplicability (JSON), Created, DiscountPercentage, DurationType, FulfillmentInterval, LastUpdated, LifecycleStatus, Model, OfferCategory, PaymentInterval, PaymentModel, PaymentUpfrontPercentage, PricingCurrencyContractCommitmentCost.
  • The transform uses `column_ifexists` for the new columns so partial 1.3-shaped source data still ingests cleanly.

Out of scope

  • HubSetup_v1_4.kql -> phase 5
  • New Invoice dataset (mentioned in 1.4 working_draft) -> deferred until spec stabilizes

Closes #2131

Test plan

  • Pester unit tests pass: 192/192 in HubsIngestionQueries.Tests.ps1
  • Bicep template compiles cleanly
  • Manual smoke test once Cost Management ships any FOCUS 1.4-preview export.

flanakin and others added 4 commits May 6, 2026 08:07
Adds IngestionSetup_v1_3.kql with Costs/Prices/CommitmentDiscountUsage/
Recommendations/Transactions transforms and final tables renamed for
FOCUS 1.3. Cost and Usage gains 8 new FOCUS 1.3 columns:
AllocatedMethodId/Details/ResourceId/ResourceName/Tags (data-generator
split cost allocation), ContractApplied (per-row contract commitment
application), ServiceProviderName + HostProviderName (replacing the
deprecated ProviderName/PublisherName, with empty-fallback for back
compat).

Costs_raw now carries the new columns so downstream v1_2 transforms
keep working and v1_3 transforms can read them directly. The v1_3
file is wired into the Bicep deployment alongside v1_0 and v1_2.

The unversioned Costs() function still aliases to v1_2 — phase 2
adds Costs_v1_3() and retargets Latest.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds HubSetup_v1_3.kql with Costs_v1_3, Prices_v1_3,
CommitmentDiscountUsage_v1_3, Recommendations_v1_3, and
Transactions_v1_3 that union the new Costs_final_v1_3 with the
existing Costs_final_v1_2 and Costs_final_v1_0 tables.

For v1_2 data unioned into the v1_3 view, the 8 new FOCUS 1.3 columns
default to null/empty and ServiceProviderName/HostProviderName are
populated from the deprecated ProviderName/PublisherName for back
compat. The same defaults apply to the v1_0 union arm on top of the
existing v1_0 -> v1_2 conversion.

HubSetup_Latest.kql aliases now point to *_v1_3 functions so the
unversioned Costs(), Prices(), etc. return the latest schema.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the FOCUS 1.3 Contract Commitment supplemental dataset:
- ContractCommitment_raw table in IngestionSetup_RawTables.kql with
  the 14 mandatory FOCUS 1.3 columns plus standard hubs source
  metadata.
- ContractCommitment_transform_v1_3 + ContractCommitment_final_v1_3
  in IngestionSetup_v1_3.kql, with update policy mapping raw to final.
- ContractCommitment_v1_3 hub function in HubSetup_v1_3.kql that
  reads the final table directly (no older versions to union).
- Unversioned ContractCommitment alias in HubSetup_Latest.kql.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds IngestionSetup_v1_4.kql as a preview schema for FOCUS 1.4
working_draft. Cost and Usage drops the deprecated ProviderName
and PublisherName columns (removed in 1.4 per spec). Contract
Commitment expands by 14 new columns: BenefitCategory,
ContractCommitmentApplicability (JSON), Created, DiscountPercentage,
DurationType, FulfillmentInterval, LastUpdated, LifecycleStatus,
Model, OfferCategory, PaymentInterval, PaymentModel,
PaymentUpfrontPercentage, PricingCurrencyContractCommitmentCost.

The internal ProviderName/PublisherName fix-up logic in the Costs
transform stays so HostProviderName / ServiceProviderName can fall
back to those values when source data does not yet supply them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@flanakin flanakin requested a review from MSBrett as a code owner May 6, 2026 15:39
Copilot AI review requested due to automatic review settings May 6, 2026 15:39
@microsoft-github-policy-service microsoft-github-policy-service Bot added the Needs: Review 👀 PR that is ready to be reviewed label May 6, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added the Tool: FinOps hubs Data pipeline solution label May 6, 2026
@flanakin flanakin changed the base branch from flanakin/focus14-phase3-contractcommitment to flanakin/focus14 May 6, 2026 23:00
@flanakin flanakin marked this pull request as draft May 6, 2026 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs: Review 👀 PR that is ready to be reviewed Tool: FinOps hubs Data pipeline solution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants