focus-data-toolkit 0.13.0

Prepared in source; publishing is a separate release step.

• Correct period-subscription effective costs and derive illustrative 10% taxes
  from earlier untaxed usage with exact amount and currency lineage.
• Stabilize offer/SKU/price identities and public prices; correct recognized storage
  properties, AWS namespaces and owning subaccounts, including allocations.
• Model partially used fleets of 500 machines, exact annual capacity and complete
  budget-aware groups. Add targeted 1.3 negotiated contracts, shared contract parents
  and a common generation registry; Contract Commitment APIs accept include_credits.
• Omit inapplicable ContractApplied metric properties while retaining legacy input
  compatibility. Preserve exact invoice aggregates and widen affected Parquet cost
  columns to decimal128(38,17) without silent rounding.
• Replace FAIL-only official gating with complete inventories, violation counts,
  resource/source/data fingerprints, archived reports and separately reviewed
  candidate evidence. Public validate --official now rejects false-green exits.
• New byte baseline: regenerated provider goldens, archived before examples,
  independent corruption tests and reproducible statistics. See
  generator corrections for assumptions, residual
  official failures and the deliberate differences from upstream PRs 6 and 7.

focus-data-toolkit 0.12.0

Back-ports the reviewed FOCUS-Sample-Data conformance fixes into the generator engine.

This release carries a deliberate reproducibility break: every golden fixture was
regenerated once, so synthetic output bytes differ from 0.11.0 for identical generation
parameters. That is a new byte baseline, not a regression — see
docs/versioning.md; pin an exact version if you need byte stability.

Fixed

• Exact cost arithmetic in the generators. Costs are now exact products of their
  factors: the unit price (10 dp) and quantity (4 dp) are quantised, never the product,
  so ListCost == ListUnitPrice × PricingQuantity and ContractedCost == ContractedUnitPrice × PricingQuantity hold under exact Decimal equality on every
  priced row (previously ~46% of usage rows carried a ≤ 5e-7 rounding delta). Products
  are display-trimmed to 6 decimals only when lossless. The Parquet decimal registry
  default widens from (38, 12) to (38, 14) to hold these products exactly.
• Three prices kept apart on commitment-covered usage. ContractedUnitPrice now
  carries only the negotiated rate; the commitment discount shows only between
  ContractedCost and EffectiveCost, giving the strict EffectiveCost < ContractedCost ≤ ListCost ordering on Used rows (previously the commitment rate was
  folded into the contracted price and ContractedCost == EffectiveCost).
• FOCUS erratum #3 ContractApplied casing. The 1.3 generators emit the canonical
  ContractId / ContractCommitmentId element keys of the 1.3.0.1 rule model, and the
  1.3 parser accepts both casings — legacy pre-erratum ContractID /
  ContractCommitmentID input is normalized and surfaced once per conversion as the
  new catalogued FDT-CA-001 warning (compatibility is never silent equivalence; mixed
  casings in one element are rejected as ambiguous). to_json(..., version="1.3") emits
  the canonical casing. Supersedes the 0.2.0 note that recorded the uppercase casing as
  the fix.
• Full 1.2 billing identity on commitment groups. 1.2 commitment usage rows now copy
  all seven identity keys from the purchase (as 1.3 already did), so each
  BillingAccountId maps to exactly one BillingAccountName and one InvoiceId
  (previously name and invoice diverged within every 1.2 commitment group).
• PricingCurrency on 1.2 Tax/Credit rows. Tax and Credit rows carry
  PricingCurrency and PricingCurrencyEffectiveCost in 1.2 as in 1.3, and
  same-currency pricing columns are exact mirrors instead of re-quantised views.

Added

• Generated-data conformance suite (tests/test_generated_conformance.py): the
  upstream FOCUS-Sample-Data checker catalogue (24 assertions per provider for 1.2,
  36-37 for 1.3) ported onto the toolkit's generators and run against both fresh
  generation and the committed golden fixtures — exact Decimal equality throughout.
• Official-validator CI gate (scripts/validate_official_samples.py + the
  official-validation job in ci.yml): the nine generated outputs (3 providers ×
  1.2 Cost and Usage, × 1.3 Cost and Usage + Contract Commitment) are validated by the
  official FinOps focus-validator, pinned to 2.2.1, with
  --applicability-criteria ALL so the conditional rules for the capabilities these
  datasets exercise run too, offline via the packaged / SHA-256-pinned release rule
  models. Each (version, provider, dataset) run is compared against its exact
  expected artifact set — per-rule justifications printed on every run, each claim
  pinned by a data-side conformance test; an unexpected failure or a stale
  allowlist entry (an expected artifact that stops failing) breaks the build. The
  1.3 Contract Commitment dataset validates with zero artifacts.
• scripts/regenerate_golden_fixtures.py: replays the exact golden grid of
  tests/test_generator_golden.py, replacing the ad-hoc regeneration procedure.
• CI: dependency-review job in .github/workflows/security.yml (PR-only, informational).
  Now that the repository Dependency Graph is enabled, actions/dependency-review-action gates a
  PR's dependency diff against the GitHub Advisory database and fails on a HIGH+ vulnerability.
  It complements pip-audit (which scans the installed Linux/py3.12 environment every run) by
  covering the full locked graph at PR time — including platform-/version-conditional packages
  (e.g. the Windows-only tzdata) that the single-platform install never exercises — ahead of
  Dependabot's post-merge alerts. Least privilege (contents: read), guarded to pull_request
  events, SHA-pinned; deliberately not a required check.

Changed

• Commitment discounts are modelled per charge period and reconcile exactly. Each
  commitment now emits whole per-period blocks: a Recurring Purchase row
  (BilledCost = the per-period fee, EffectiveCost = 0, explicit
  CommitmentDiscountQuantity/Unit — committed spend in USD or capacity in the
  native unit), Used rows for consumed capacity, and one Unused row absorbing the
  use-it-or-lose-it remainder — so sum(Usage.EffectiveCost) == sum(Purchase.BilledCost) holds under exact equality per charge period and per
  billing period (previously one One-Time all-upfront purchase with a handful of
  covered hours and no reconciliation). The provider terms are NoUpfront
  accordingly (payment-option metadata, SKU names and purchase descriptions — a
  recurring fee contradicts all-upfront). Spend commitments price a monetary
  block on their Purchase and Unused rows: PricingUnit is the currency, the
  unit prices are exactly 1.00 and the priced quantity is the committed/unused
  spend itself (Used rows keep the consuming resource's native pricing).
  ContractApplied is attached to every 1.3 commitment row — Purchase
  (ContractCommitmentId == ResourceId per rule O-039-C), Used and Unused alike
  — with all five element keys always present (rule O-007-M) and exactly one
  metric branch per category: a spend commitment applies a cost alone, a usage
  commitment applies the measured quantity in its native unit alone, so the
  quantity branch survives the 1.4 oneOf migration instead of being demoted to
  x_ custom keys.
• The Contract Commitment dataset carries term totals and negotiated terms. Costs
  and quantities are the 1-year term totals; Spend commitments leave quantity/unit
  empty while Usage commitments carry a real quantity in its native unit; the contract
  period encloses the commitment period by 90 days. Three negotiated non-discount
  terms per provider (minimum spend, negotiated rate card, usage commitment) share one
  multi-commitment ContractId and are reachable from Cost and Usage exclusively
  through ContractApplied — the FOCUS-defined dataset relationship — never via
  CommitmentDiscountId equality. On-demand 1.3 usage rows reference them with
  cross-dataset unit coherence: the rate card and minimum spend apply a cost on every
  row (unit-agnostic), while the usage commitment — contracted in Hours — receives
  quantities only from usage of the commitment-eligible compute service, measured in
  that same unit (an element applied to a Usage-category commitment always matches
  its ContractCommitmentUnit).
• Split Cost Allocation rows are coherent groups. One shared host charge is fully
  allocated to 2-3 distinct workloads in a single charge period: AllocatedRatio
  values sum to exactly 1 and every cost column conserves the host amount exactly
  (quantity shares absorb the residue; each row's costs stay exact unit-price ×
  quantity products). The residue arithmetic is shared with
  generators/scenarios.py via the new generators/engine/allocation_math module.
• Docs: docs/runner.md rewritten as a step-by-step guide, and several claims corrected (no
  runtime code changed). The page now walks a newcomer from docker pull to a validated FOCUS 1.4
  output — prerequisites, a mount-free first run, directory setup, a six-step walkthrough with the
  real command output, Podman, GitHub Actions / Kubernetes / cron examples, and a troubleshooting
  table — before the reference sections. Corrections to previously published statements: the disk
  budgets apply to the streaming path only (the eager CSV conversion ignores them and cannot
  exit 5); the exit-code table is the convert contract, not a global one; the image ships the
  [parquet] extra only, so validate --official and ui are unavailable in the container; the
  image's CycloneDX SBOM is retained as a workflow artifact, not attached to the release or
  pushed to the registry; trivy fails on HIGH/CRITICAL findings that have a fix available
  (ignore-unfixed: true). Also documents FOCUS_TOOLKIT_LOG_LEVEL (with its current lack of
  effect in the Runner), the _run.json sidecar, focus-toolkit clean after a hard kill, and the
  TMPDIR vs FOCUS_TOOLKIT_WORK_DIR distinction for validate-bundle. Two further scope
  corrections: the cooperative cancel (SIGTERM → exit 130, nothing published) is a property of the
  streaming path — the eager CSV conversion installs no signal handler, so docker stop there
  terminates it mid-flight (exit 143) and can leave staging behind; and clean takes only --out,
  so it does not sweep fdt-<run_id> scratch orphaned under FOCUS_TOOLKIT_WORK_DIR by a
  killed streaming run. docs/compatibility.md now points Windows readers to the Runner for the
  streaming path.
• CI (risk-based audit follow-up; no runtime code changed). The container SIGTERM smoke
  is now deterministic: it waits for the conversion to observably start before stopping, and
  a run that finishes before the signal lands fails as inconclusive instead of passing
  silently. The bounded-memory streaming test (-m slow, ~10 min under tracemalloc) now runs
  automatically in a new scale.yml — on streaming-e...

focus-data-toolkit 0.11.0

First stable release. Same feature set as 0.11.0rc1, promoted to a final release now that the
embedded FOCUS 1.4 model's provenance is complete — the FinOps Foundation source workbook is
hashed (source.artifact_sha256) and the committed model was reproduced from it byte-for-byte by
the pinned extractor — so it installs with a plain pip install focus-data-toolkit (no --pre).

Highlights (full details in the 0.11.0rc1 entry below):

• Studio — a local web UI (focus-toolkit ui, extra [studio]) over the same Core.
• Runner — a containerised batch image (GHCR) whose entrypoint is the focus-toolkit CLI.
• Core (Lot A) — progress/cancellation, separate work/output disk budgets, --exit-policy, and
  the detect / validate-bundle / version commands.

Changed

• Model provenance: partial → complete (source workbook hashed and end-to-end reproduction
  verified; see docs/model-provenance.md).
• Release workflow (release.yml): the GitHub Release step is idempotent — if a release for the
  tag already exists (e.g. created via the "Draft a new release" UI), it updates that release in
  place and attaches the attested assets instead of failing.

focus-data-toolkit 0.11.0rc1

First release candidate published to PyPI — a pre-release (marked as such per the honesty gate,
because the embedded FOCUS model's provenance is partial; see
docs/model-provenance.md). It bundles the three deployment access
methods on the single Core: the CLI/SDK, the containerised Runner (Lot B) and the local
Studio web UI (Lot C), plus the Core progress/cancellation, disk-budget and CLI additions
(Lot A). Install with pip install --pre focus-data-toolkit (pre-releases are not selected by a
plain pip install).

Added — Studio: local web UI (deployment Lot C)

• focus-toolkit ui launches a local web app (FastAPI, behind the optional [studio] extra;
  the command imports it lazily so a core install is unaffected) over the same Core — every
  operation drives the same SDK the CLI/Runner use, so its manifests, diagnostics and checksums are
  identical. Detect a source, pick a file under --root / upload (capped) / generate synthetic
  data, convert (strict|synthetic, CSV|Parquet) with live per-phase progress and cancel,
  preview a sampled page (the full file is never loaded), and download datasets, manifest,
  diagnostics (JSON/CSV), SHA256SUMS and an HTML summary.
• Security: binds 127.0.0.1 by default (a non-loopback --host is refused without
  --allow-remote); a fresh per-start token is required on every API call; Host/Origin are
  validated (anti DNS-rebinding / CSRF); file access is confined to the allowlisted --root;
  uploads stream to disk and are size-capped. No telemetry, no external upload.
• Path confinement: resolve_within_root walks real directory entries (matching each component
  by name, never concatenating the user string into a path) and canonicalises every matched
  entry with Path.resolve — a symlink, Windows junction or reparse point whose real target
  escapes --root is refused, while a link that stays inside is followed; absolute, drive-relative
  and UNC paths and .. traversal are rejected.
• Bounded by design: one conversion at a time by default (extra submissions queue); per-job
  scratch under a work dir with TTL + startup cleanup; generation is row-capped in the UI (use the
  CLI/Runner for very large synthetic sets). New extras studio and studio-all; see
  docs/studio.md.

Added — Runner: containerised batch image (deployment Lot B)

• OCI image (Dockerfile) whose entrypoint is the focus-toolkit CLI — a container run
  equals a CLI run (same manifests, diagnostics, checksums, exit codes; no FOCUS logic
  duplicated). Batch-only (no HTTP server). Multi-stage build on a digest-pinned
  python:3.12-slim-bookworm, bundling the [parquet] extra; non-root (uid 65532),
  read-only-rootfs compatible (only /work and /output written), FOCUS_TOOLKIT_WORK_DIR=/work.
  Exec-form entrypoint so docker stop (SIGTERM) cancels cleanly (exit 130, nothing partial
  published). Volumes: /input (ro), /output, /work. See docs/runner.md.
• Container CI (.github/workflows/container.yml): builds the image on every PR / push to
  main (no publish) and runs docker run smoke tests — non-root uid, read-only-rootfs streaming
  Parquet convert, exit codes, SIGTERM handling — plus a trivy scan (fails on HIGH/CRITICAL). A
  fast static test (tests/test_container.py) enforces the base-image digest pin, non-root user
  and exec-form entrypoint.
• Container release (.github/workflows/release-container.yml): on a v* tag, runs the same
  release gates as the PyPI flow (tag matches __version__; provenance-honesty gate), builds a
  candidate image, scans it (trivy) before any public tag is assigned, then publishes to
  ghcr.io/guymano/focus-data-toolkit — immutable <version> and sha-<full-commit> tags
  plus a rolling <major>.<minor> alias (PEP 440 tag parsing; no latest) — generates a
  CycloneDX SBOM, attests build provenance and signs with cosign (keyless OIDC), in a
  reviewer-gated ghcr environment. All actions pinned by commit SHA.

Added — progress, cancellation, disk budgets & pipeline ergonomics (deployment Lot A)

• Progress reporting: the streaming engine (convert_files) accepts an optional
  progress callback receiving throttled ProgressEvents per phase (READING,
  TRANSFORMING, AGGREGATING, WRITING, VALIDATING, PUBLISHING) with a completed
  count, an optional total, a unit (rows/bytes) and a message — derived without
  materialising data (CSV byte cursor / Parquet footer row count). focus-toolkit convert --progress renders a single throttled status line on stderr. All hooks are opt-in and
  keyword-only; output is byte-identical with or without them.
• Cooperative cancellation: convert_files(..., cancel=...) checks a predicate between
  rows and validation passes and raises ConversionCancelled — the atomic staging directory
  is removed, so nothing partial is ever published. The CLI maps SIGINT/SIGTERM to a clean
  cancel (exit code 130), so Ctrl-C and docker stop unwind cleanly instead of dying
  mid-write.
• Separate disk budgets (focus_data_toolkit.runtime): the scratch filesystem and the
  output filesystem are budgeted independently via FOCUS_TOOLKIT_WORK_DIR,
  FOCUS_TOOLKIT_MAX_WORK_BYTES, FOCUS_TOOLKIT_MIN_WORK_FREE_BYTES and
  FOCUS_TOOLKIT_MIN_OUTPUT_FREE_BYTES (FOCUS_TOOLKIT_LOG_LEVEL too). A best-effort
  pre-flight (estimate with a safety margin) plus periodic in-run checks fail fast with a
  structured FDT-IO-005 (output) / FDT-IO-006 (work / temp budget) diagnostic and CLI
  exit code 5, instead of a raw OSError mid-run. WORK_DIR relocates the SQLite
  aggregation + bundle-spill scratch off the output disk (business artifacts stay
  byte-identical; scratch is always cleaned up).
• Pipeline-friendly exit codes: focus-toolkit convert --exit-policy pipeline maps the
  functional-but-complete outcomes (3 = strict incomplete, 4 = synthetic assumptions) to 0,
  so orchestrators (Kubernetes / Airflow / Jenkins / AWS Batch) don't flag a legitimate run
  failed. The default detailed policy keeps the historic codes; full status stays in the
  manifest and _run.json.
• New CLI commands: focus-toolkit detect (dataset/version of a file header, text/JSON),
  focus-toolkit validate-bundle (cross-dataset validation gate over explicit per-dataset
  files or an auto-detected --directory; ambiguous combinations refused), and
  focus-toolkit version. New SDK exports: ProgressEvent, ConversionCancelled.

Added — provider-native supplement adapters

• Adapters translate documented cloud-provider export formats into the
  canonical supplement kinds automatically, so a client passes native exports
  straight to convert / supplements validate without renaming anything.
  First adapters (AWS): aws-invoice-summary (Invoicing API InvoiceSummary,
  incl. nested Entity.InvoicingEntity, DueDate, PurchaseOrderNumber) →
  invoice; aws-savings-plans (Savings Plans inventory: paymentOption,
  state, start) → contract_commitment. Each adapter is a vendored,
  versioned JSON mapping table with official-doc provenance
  (supplement/adapters/adapters_provenance.json, sha256-verified); the format
  is auto-detected from the header (force with FILE:<adapter-name>).
  Translated rows flow through the unchanged supplement validation and carry
  ENRICHED lineage attributed as supplement:<adapter>@<version>:<file>. An
  adapter only maps fields its table describes (residual gaps are reported, not
  guessed); an unrecognized export falls back to the generic FOCUS-named path.
  New command: fdt supplements adapters. Adapters ship for AWS
  (aws-invoice-summary, aws-savings-plans), Azure (azure-invoice —
  Billing Invoices REST API; InvoiceStatus Due/OverDue/Paid → Issued,
  Void → Voided) and GCP (gcp-compute-commitments — Compute Engine
  regionCommitments; status and CUD payment facts → contract_commitment).

Added — supplemental client data (promise #3)

• Gap analysis (fdt gaps): reports, per FOCUS 1.4 dataset, exactly which
  columns block strict production for a given 1.2/1.3 source — computed from
  the converter's own provenance rules and annotated from the embedded model —
  plus ready-to-fill CSV templates per supplement kind. Missing mandatory
  source columns are reported as source-completeness gaps.
• Supplement bundles: clients supply the missing provider-issued facts as
  sidecar files (CSV/JSON, gzip ok; kinds billing_period, invoice,
  invoice_line, contract_commitment; kind auto-detected from the header or
  forced with FILE:KIND). Supplements are validated against the source and
  the model before any use (FDT-SUPP-0xx: duplicate keys, unknown columns,
  format/allowed-value violations, orphans, BilledCost reconciliation
  conflicts, per-column coverage). Pre-flight command:
  fdt supplements validate.
• ENRICHED conversion: convert_to_focus_1_4(..., supplements=...)
  applies supplied facts with ENRICHED lineage and full attribution
  (supplement:<kind>:<file> + sha256 in the new manifest supplements
  section). At full coverage, strict mode now produces all four FOCUS 1.4
  datasets with nothing invented; partial coverage keeps the dataset
  NOT_PRODUCED with per-value counters showing how close it is. In strict
  mode uncovered nullable assumed columns are emitted empty (synthetic
  defaults never leak); real issuer-assigned InvoiceDetailIds replace the
  locally generated back-links.

Added — capability profiles

• New CapabilityProfile (focus_data_toolkit.model.capabilities): an
  explicit, validated declaration of the FOCUS applicability conditions a
  source supports (SupportsUnitPricing,
  SupportsMultiplePricingCategories). The lint...