Skip to content

v1.2.0

Choose a tag to compare

@mbettan mbettan released this 15 Jul 03:25
· 60 commits to main since this release
79fd81d

This release consolidates multiple enhancements, security hardening updates, and critical bug fixes to ensure data correctness, cost safety compliance, and comprehensive org-wide auditing.

🔑 Key Highlights

1. 🏗️ Organization-Level Schema Auditing

Engineered a dynamic discovery layer to seamlessly scan the entire GCP organization for missing partitions and clusters without hitting API limits.

2. 🎯 Project Focus Scope Added

Introduced a global focus_projects parameter to scope analysis to specific GCP projects rather than the entire organization.

3. 🛡️ Security & XSS Hardening

Resolved DOM-based XSS vulnerabilities and removed all hardcoded fallback demo data to ensure true empty states.

4. 🐛 Org-Wide Bug Fixes

Fixed MV Cost Auditor phantom counts, Cost Attribution math, and corrected Slot Utilization concurrency to reflect true org-wide demand.

5. 📊 Metadata in Job Analysis

The analyze_jobs response now includes sample metadata, enabling accurate explanations of sampling bias.

6. 🔎 AI Doctor Enhancements

Parameterized AI Doctor job IDs to prevent string interpolation risks and improved the underlying prompt context.

7. 🚀 Fluid Scaling Guardrails

Verified that user-configured safety caps (max_bytes_billed_gb) are strictly enforced across all fluid scaling backend API calls.

8. 🧩 UI/UX Refinements

Promoted the Schema Optimizer to its own navigation group and added collapsible educational panels for power users.

✨ Features & Enhancements

  • Organization-Level Schema Auditing: Engineered a dynamic discovery and UNION ALL orchestration layer to seamlessly scan the entire GCP organization for missing partitions and clusters without querying empty projects or hitting API limits.
  • Versioned Static Assets: Static assets served with version query parameters now receive long-lived caching headers (Cache-Control: public, max-age=31536000, immutable), reducing repeat page-load latency.
  • Metadata in Job Analysis: The analyze_jobs response now includes sample metadata, enabling the frontend to accurately explain sampling bias (e.g., top N jobs by bytes billed).
  • Logging & Observability: RotatingFileHandler is now gated behind an ENABLE_FILE_LOG environment variable to optimize memory in Cloud Run environments.
  • Refactors: Centralized standard date constants (DAYS_PER_MONTH), eliminated redundant .groupby operations in fluid scaling, and parameterized job IDs for AI Doctor instead of using string interpolation.

🛡️ Security & Hardening

  • XSS Vulnerability Fixes: Resolved a DOM-based Stored XSS in the Slots Profiler caused by a sanitizer whitelist bypass, and a full-application XSS vulnerability via snapshot hydration bypass. All imported snapshots and table rendering functions now properly HTML-escape user data.
  • Data Integrity & Honest Empty States: Removed all hardcoded fallback demo data (including mock project names, dummy schema names, and fabricated dollar amounts) from backend endpoints. Empty queries now return true empty states.
  • Focus Filter Safeguards: Removed the focus_projects parameter from Capacity Planning and Active Assist endpoints where it previously distorted capacity totals or was ignored. Focus filtering remains fully active on 19 other org-level endpoints.
  • Environment Variables: Ensured .env loaders only fill in missing variables and no longer override Cloud Run/GKE injected credentials or configurations.
  • Validation Guardrails:
    • Added _safe_ident() validation to BQ-sourced project IDs before SQL interpolation.
    • Bounded numeric parameters (e.g., capping lookback_days to 90) to prevent unbounded org-wide scans or division-by-zero crashes.

🐛 Bug Fixes

  • MV Cost Auditor: Fixed a bug where MV refresh patterns were silently missed across projects. The auditor now discovers all projects with MV activity and keys the inventory correctly by project, preventing phantom counts and accurately measuring org-wide MV costs.
  • Slot Utilization Concurrency: Changed the aggregation method to sum across all concurrent jobs before taking quantiles. Capacity metrics now correctly reflect true org-wide concurrent demand rather than the single largest job's peak.
  • Cost Attribution & Waste Rules:
    • Rejected focus_projects filters in cost attribution, which previously corrupted waste allocation math.
    • Fixed a bug where waste silently vanished under Rule B (Central Dump) if the central project wasn't configured.
    • Corrected the exclusion of NULL statement_type rows that artificially inflated wasted direct usage.
  • BI Engine & On-Demand Costs: BI Engine savings estimates are now only applied to on-demand jobs (not Editions). On-Demand cost simulations no longer erroneously include failed queries that aren't billed by BigQuery.
  • Data Skew & DML Auditor: Skew analysis now filters out SCRIPT parent/child jobs to avoid double-counting stages. DML auditor now requires state = 'DONE' to prevent in-flight jobs from inflating wasted slot hours.
  • Fluid Scaling Caps: Verified that the user-configured max_bytes_billed_gb safety cap is consistently forwarded to and enforced by all fluid scaling BigQuery API calls.
  • HBO Validation: Projects failing access verification now report as unknown status rather than false positives for being enabled.
  • Time-Travel DDL: Invalid combinations of time_travel_rescale (without setting hours) or non-integer float hours are now strictly rejected to prevent generating unusable BigQuery DDL.
  • Static Schema Estimates: Replaced a fabricated row count estimate (based on byte size) with the actual total_rows metric from INFORMATION_SCHEMA.TABLE_STORAGE.
  • Exceptions & File Paths: Addressed silent failures in schema and active assist audits that previously returned empty arrays on 403/404 errors. Corrected cost_attribution_config.json relative path resolution for Docker compatibility.