Skip to content

v0.40.0 — Web UI v2, onboarding, shared code

Choose a tag to compare

@padak padak released this 18 Apr 19:06
· 12 commits to main since this release

First substantial release past the Phase 1 gate. DuckDB engine + web UI + onboarding material are all in place. Snowflake / BigQuery / JDBC adapters are still Phase 2 (stub only).

Highlights

Web UI v2 (PRs #8–#13)

juncture web went from a glance-only DAG render to a self-contained debugging surface — source browser, governance, reliability, portfolio, and a one-click LLM knowledge export.

  • DAG tab with kind-distinguishable shapes (seed / SQL / Python), border-encoded last-run status, and PII ring propagation from flagged seeds.
  • Metadata sidebar with Source / Schema / Tests tabs, Prism syntax highlighting, and a reliability sparkline (p50 / p95 / 30-day success).
  • Runs tab with per-model drawer showing statement-level errors classified into buckets (type_mismatch / conversion / missing_object / …).
  • Seeds, Portfolio, Reliability, Project tabs for the CDO / governance lens.
  • Download buttons: manifest.json, manifest.openlineage.json, and llm-knowledge.json (single-shot project snapshot for paste-into-Claude workflows).

Shared code idioms

  • Jinja macros (macros/**/*.sql, PR #15) — every {% macro %} is a global. Define my_date() / is_vip() once; every model that calls it follows a single-point change. dbt-style UX without the Jinja surface.
  • Ephemeral models — named DAG nodes that behave like inlined CTEs; reuse a dimension across multiple marts without materializing a table.

Profiles (PR #18)

Named overlays on juncture.yaml for dev / staging / prod. Per-key merge on vars: and connections.<name>:; --profile / JUNCTURE_PROFILE / top-level profile: precedence. Unlocks kbagent-branch per-schema mapping without editing SQL.

Onboarding track (PR #16)

  • docs/TUTORIAL.md — four-level walkthrough (L1 zero → L2 Python in the DAG → L3 macros + ephemeral → L4 external --var).
  • examples/tutorial_shop/ — working L4 reference project (20 orders × 8 customers, two macros, ephemeral dimension, two marts, pandas cohort).
  • make examples runs it both with defaults and with a --var override so the tutorial's L4 claim is continuously verified.

CLI structure (PR #14)

Grouped top-level CLI into sql / migrate / debug sub-apps. Core init / compile / run / test / docs / web stay top-level. juncture translate → juncture sql translate, juncture diagnostics → juncture debug diagnostics, etc.

Governance surface (PR #11 / #12)

schema.yml gains owner, team, criticality, sla.freshness_hours, docs, consumers. Seeds carry pii, retention_days, source_system, source_locator. Surfaced in the web UI's Portfolio, Reliability, and Metadata tabs.

Migration tooling

Shipped during the pilot migration of a production Keboola pipeline (208 parquet seeds × 374 SQL statements E2E on DuckDB):

  • juncture migrate keboola — raw Keboola config JSON → Juncture project.
  • juncture migrate sync-pull — kbagent filesystem layout → Juncture project with EXECUTE materialization and symlinked parquet seeds.
  • juncture sql split-execute — rewrites an EXECUTE monolith into one .sql per CTAS target with inferred ref() dependencies.
  • juncture sql translate — schema-aware SQLGlot translation with harmonize_case_types / harmonize_binary_ops / fix_timestamp_arithmetic AST passes.
  • juncture run --continue-on-error — EXECUTE keeps running after failing statements and collects every primary error into one RunReport.
  • juncture debug diagnostics — regex-based error classifier; buckets and fix hints.

Bug fixes

  • --var CLI flag now reaches SQL Jinja rendering (previously flowed only to Python ctx.vars()). Project.load() takes run_vars and merges them into config.vars before the first Jinja pass.

Breaking changes

  • Top-level juncture translate / juncture sanitize / juncture split-execute / juncture migrate-keboola / juncture migrate-sync-pull / juncture diagnostics now live under sql / migrate / debug sub-apps. The old command paths stay as deprecated aliases and emit a deprecation notice.

Doc house-keeping

  • docs/ trimmed to five public files: VISION.md, TUTORIAL.md, CONFIGURATION.md, ROADMAP.md, DESIGN.md (+ images/). Development-track material (STATUS.md, STRATEGY.md, RESEARCH.md, MIGRATION_TIPS.md, BENCHMARKS.md, rfcs/, src.md) moved to docs/priv/ (gitignored).
  • README rewritten from a new-user perspective with hero + Runs screenshots.

What's next

Phase 2: Snowflake / BigQuery / JDBC adapters + real SAPI upload in the Keboola component. Track in docs/ROADMAP.md.