diff --git a/AGENTS.md b/AGENTS.md index 64e40d6..3d1624a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -511,7 +511,7 @@ LIGHTDASH_TRINO_HOST=host.docker.internal # Trino host override for Docker ## AI Agent Integration -When `dj.codingAgent` is `true`, the extension generates a project-tailored `AGENTS.md` at `.agents/dj/AGENTS.md` and copies agent-agnostic skill directories from [`templates/`](templates/) to `.agents/skills/` at workspace activation, following the [Agent Skills](https://agentskills.io) open standard (each skill is a folder with a `SKILL.md`). The agent code lives in [`src/services/agent/`](src/services/agent/); skill files are written by the Dbt service. +When `dj.codingAgent` is `true`, the extension copies the DJ agent context to `.agents/dj/` — a hub `AGENTS.md` plus on-demand `reference/` files — and copies the agent-agnostic skill directories from [`templates/`](templates/) to `.agents/skills/` at workspace activation, following the [Agent Skills](https://agentskills.io) open standard (each skill is a folder with a `SKILL.md`). The templates live under [`templates/_agents-dj/`](templates/_agents-dj/) and [`templates/skills/`](templates/skills/); both are deployed by the Dbt service ([`src/services/dbt.ts`](src/services/dbt.ts)), which strips the leading `_` from `_AGENTS.md` / `_SKILL.md` on copy. ## Additional Resources diff --git a/CHANGELOG.md b/CHANGELOG.md index bab314f..e07bbfb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,11 +2,24 @@ ## 2.1.0 +### Agent skills + +- **New `dj-govern-model` skill.** Runs a read-only governance audit across a model, folder, dependency tree, or the whole workspace — ownership, PII / classification / compliance tagging, registered-group conformance, and prod-write posture — and points you to the skill that fixes each gap. Try _"Which models in the finance group have no owner?"_ +- **New `dj-create-source` skill.** Registers a raw Trino table as a DJ source by reading its exact column types from the warehouse, so a model that reads a not-yet-defined `catalog.schema.table` builds instead of failing. Authoring skills detect the missing source and offer to create it, or run the `DJ: Create Source` command yourself. +- **New `dj-run-dbt`, `dj-run-trino`, and `dj-git-workflow` skills.** Compile and test models, run read-only Trino queries, and commit your `.model.json` sources together with their generated SQL/YAML — activating the right Python environment and keeping warehouse writes behind explicit confirmation. +- **Skills confirm the target and stay read-only by default.** Creating or converting a model asks which dbt project to use, requires the `group` to be registered in `models/groups.yml`, and offers optional `owner` / `pii` / `classification` / `compliance` tags (skip them and nothing is written). Anything that writes to the warehouse or production always needs your explicit confirmation. +- **Model creation builds missing upstreams and defaults BI work to marts.** A dashboard or metrics request scaffolds a mart and offers to build any missing staging / intermediate / source layers upstream-first, then reminds you to re-sync so the new columns resolve. +- **Leaner agent context.** The generated `.agents/dj/AGENTS.md` is now a slim hub, with deep reference material split into on-demand files under `.agents/dj/reference/` that skills open only when needed. + ### UX improvements -- **Data Modeling canvas selects keep dropdown text sized with zoom.** Single- and multi-select menus in the visual editor stay inside the canvas transform so option text matches node chrome when you zoom in or out; truncated labels show the full value on hover. +- **Data Modeling dropdowns scale with zoom.** Single- and multi-select menus in the visual editor now size their text to the zoom level so it matches the node, and truncated labels show the full value on hover. - **Monochrome sidebar icon.** Switched the Activity Bar icon to a monochrome SVG so it remains visible across light, dark, and high-contrast themes in remote workspaces. +### Documentation + +- **New [Agent Skills](docs/AGENT_SKILLS.md) guide** — catalogs every DJ agent skill and when to reach for it. + ## 2.0.2 - **Python model run tracking in Airflow.** DJ-generated `etl_helper.py` records each python model run (success, error, skipped, upstream_failed) to a Trino meta table configured via `run_tracking` (`catalog`, `schema`, and `table` are required in the `dj_python_source_config` Airflow Variable). Mapped tasks reconcile failures that occur before model code runs, and expose helpers your DAG can call for end-of-run reconciliation and `[Python Models]` failure email summaries. diff --git a/README.md b/README.md index 8fa8e32..f38bedc 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,7 @@ DJ is a VS Code extension that revolutionizes dbt development through a structur - **Data Catalog Integration**: Browse Trino catalogs and execute queries directly in VS Code. - **BI Integration**: Built-in Lightdash support for creating dashboards from dbt models. - **11 Model Types**: Complete coverage from staging to marts with pre-built templates. +- **AI Agent Skills**: Agent-agnostic skills that guide AI coding assistants through common DJ tasks. ## Supported Stack @@ -297,6 +298,7 @@ DJ supports the following model types: - **[Lineage Visualization](docs/LINEAGE.md)** - Model and column-level lineage - **[Integrations Guide](docs/integrations/README.md)** - dbt, Trino, and Lightdash integration - **[Model Types Reference](docs/models/README.md)** - All 11 model types with examples +- **[Agent Skills](docs/AGENT_SKILLS.md)** - AI agent skills for common DJ tasks ## Support & Community diff --git a/docs/AGENT_SKILLS.md b/docs/AGENT_SKILLS.md new file mode 100644 index 0000000..ee24346 --- /dev/null +++ b/docs/AGENT_SKILLS.md @@ -0,0 +1,194 @@ +# Agent Skills + +DJ ships **AI agent skills** — packaged instructions that guide AI coding assistants (Claude Code, Cursor, GitHub Copilot, Cline, Windsurf, and others) through common DJ (Data JSON) Framework tasks: creating and refactoring models, registering sources, authoring Lightdash dashboards, running dbt and Trino commands, diagnosing slow Trino queries, resolving merge conflicts, committing your work, and more. + +This page explains what the skills are, how to turn them on, and catalogs the 17 skills DJ provides today. + +## What are DJ Agent Skills? + +- **Agent-agnostic.** Skills are plain-Markdown files that follow the [Agent Skills open standard](https://agentskills.io) — each skill is a folder containing a `SKILL.md` (plus optional `references/` and `scripts/`). Any AI coding tool that understands the standard can use them; there is no per-agent configuration. +- **Task-focused.** Each skill encodes DJ's conventions for a single job, so the assistant produces framework-correct output — the right model `type`, valid JSON against the schemas, and the single-source-of-truth rules — instead of guessing. +- **Paired with a framework reference.** Alongside the skills, DJ generates `.agents/dj/AGENTS.md` — a slim, project-tailored hub — plus on-demand `.agents/dj/reference/` files (model types, materialization, Lightdash config, CTEs, running dbt/Trino, git workflow, and more) that the skills read as needed. + +## Enabling skills + +Skills are opt-in via a single setting. + +1. Set `dj.codingAgent` to `true` in the VS Code settings UI, or in `.vscode/settings.json`: + + ```json + { "dj.codingAgent": true } + ``` + +2. Run **`DJ: Refresh Projects`** from the Command Palette. (`dj.codingAgent` takes effect on refresh — see [When Settings Take Effect](SETTINGS.md#when-settings-take-effect).) + +DJ then writes, at your workspace root: + +- `.agents/dj/AGENTS.md` — the framework reference hub +- `.agents/dj/reference/*.md` — on-demand deep-reference files the hub and skills open when needed +- `.agents/skills//SKILL.md` — one folder per skill, with any bundled `references/` and `scripts/` + +Point your AI coding tool at the workspace and the skills become available. Most skills also rely on the `.dj/schemas/` directory (the JSON schemas DJ maintains in every workspace) for exact model shapes. + +> Legacy string values (`"github-copilot"`, `"claude-code"`, `"cline"`) are still accepted but deprecated — skills are now agent-agnostic. For details, see [AI & Coding Agents](SETTINGS.md#ai--coding-agents) in the Settings reference. + +## How skills work + +- **Just ask.** Describe your task in natural language ("create a mart for daily orders", "why is this query slow?") and the assistant matches it to a skill via that skill's _Use when…_ description. You can also name a skill directly. +- **Progressive disclosure.** A skill loads its `SKILL.md` first and pulls in `references/` or runs `scripts/` only when needed, keeping the assistant focused. +- **Single source of truth.** Skills edit only the JSON sources of truth — `.model.json`, `.source.json`, `.python.json` — and never hand-edit the generated `.sql` / `.yml` / `.python.py`, which DJ regenerates via JSON Sync. +- **You stay in control of DJ commands.** Skills can't run VS Code commands themselves; they'll ask you to run things like **`DJ: Sync to SQL and YML`** or **`DJ: Refresh Projects`** at the right moment. +- **Some skills are read-only.** `dj-review-python-model`, `dj-govern-model`, and `dj-trino-analyzer` produce reports and change nothing. + +## The skills + +DJ provides 17 skills, grouped below by what they help you do. + +### Setup & onboarding + +#### `dj-initialize` + +Interactive wizard that sets up and configures the DJ Framework in an existing dbt project — Python virtual environment, `dbt_project.yml` vars, `models/groups.yml`, `.vscode/settings.json`, and optional Trino, Lightdash, and Airflow integrations — one step at a time. + +- **Use when:** you want to set up DJ in an existing dbt project, configure required settings, or diagnose why DJ is not working correctly. +- **Example prompt:** _"Set up the DJ framework in this dbt project."_ + +### Authoring SQL models + +#### `dj-create-new-model` + +Scaffolds a new `.model.json` for any layer — staging, intermediate, or mart — including joins, CTEs, rollups, subqueries, and aggregations. This is DJ's primary model-authoring reference; the other authoring skills defer to it. + +- **Use when:** you want to create, add, or scaffold a dbt model. +- **Example prompt:** _"Create a mart that summarizes daily order totals per customer."_ +- **Bundled reference:** `mart-lightdash-recipes.md` — recipes for marts that back a Lightdash explore. + +#### `dj-convert-sql-to-model` + +Converts an existing SQL query into a **new** `.model.json`, mapping SQL patterns to the right model `type` and column definitions. It only creates new files — it never overwrites existing JSON, SQL, or YAML. + +- **Use when:** you have a working SQL query (often from a `.draft.sql` file) and want to formalize it as a DJ/dbt model. +- **Example prompt:** _"Convert this draft.sql into a DJ model."_ + +#### `dj-create-source` + +Registers a raw Trino table as a DJ `.source.json` by introspecting its exact column types (`SHOW COLUMNS`), so a model that reads a not-yet-defined `catalog.schema.table` builds instead of failing. The model-authoring and SQL-conversion skills detect a missing source and defer to this one; data types are read from the warehouse rather than guessed. + +- **Use when:** a model needs a raw `catalog.schema.table` that isn't defined as a source yet, or you want to add a table or columns to an existing source. +- **Example prompt:** _"Register the raw orders table as a DJ source."_ + +### Python ETL models + +#### `dj-create-python-model` + +Scaffolds a `.python.json` for a pre-dbt Python ETL pipeline that extracts data from external sources (APIs, databases, files) and loads it into Iceberg tables for downstream dbt models to consume. It favors Trino SQL for transforms, using pandas only for ingestion and Python-only logic. + +- **Use when:** you want to create a Python model, ETL pipeline, data ingestion, API fetch, CSV import, or any pre-dbt Python data processing task. +- **Example prompt:** _"Create a Python model that fetches the Backstage API into an Iceberg table."_ +- **Bundled references:** `etl-patterns.md` (per-stage code templates), `worked-example.md` (a complete end-to-end pipeline). + +#### `dj-review-python-model` + +**Read-only** audit of a Python model (`.python.py` + `.python.json`) for framework compliance, lineage readiness, downstream integration, and performance, producing a structured report before you productionize it. + +- **Use when:** you want to review, audit, validate, or check a Python model for production readiness. +- **Example prompt:** _"Review this Python model for production readiness."_ +- **Bundled reference:** `review-checklist.md` — pass/fail examples and edge cases for every check. + +### Lightdash BI & AI hints + +#### `dj-create-lightdash-yaml` + +Authors brand-new Lightdash chart and dashboard YAML (Dashboards-as-Code) from scratch for a DJ-managed explore, then uploads it. Field IDs are resolved mechanically rather than guessed from labels, and model-level required filters are honored. + +- **Use when:** you want to build, author, or scaffold a chart or dashboard that does not exist yet (no prior `lightdash download`) and then upload it. +- **Example prompt:** _"Build a Lightdash dashboard for the customer_orders explore."_ +- **Bundled reference & script:** `lightdash-as-code-authoring.md`; `get_explore_fields.py`, a read-only helper that lists an explore's dimension and metric field IDs. + +#### `dj-edit-lightdash-yaml` + +Makes minimal-diff edits to Lightdash chart/dashboard YAML that already exists locally (downloaded or previously authored) — filters, sorts, axes, table config, tiles, dashboard filters — before you re-upload. + +- **Use when:** you want to tweak existing Lightdash YAML before re-uploading via the `DJ: Lightdash - Dashboards as Code` webview. +- **Example prompt:** _"Change this chart's date filter to the last 30 days."_ +- **Bundled reference:** `lightdash-as-code-fields.md` — field-ID derivation and upload flags. + +#### `dj-update-ai-hints` + +Adds or updates Lightdash `ai_hint` values across a model's full dependency tree, typically driven from an Excel sheet — updating existing hints in place without adding new columns or metrics. + +- **Use when:** you're working with AI hints in model or source JSON files. +- **Example prompt:** _"Update the AI hints for this model's dependency tree from ai_hints.xlsx."_ + +### Refactoring & maintenance + +#### `dj-review-and-refactor-model` + +Reviews a `.model.json` (or a folder, dependency tree, or the whole workspace) and modernizes it to newer DJ capabilities — materialization shorthand, `lightdash.*` over `meta.*`, `from.rollup`, `exclude_framework_artifacts`, the `"dims"` shorthand, inline subqueries, and more. It presents all findings first and applies only what you approve. + +- **Use when:** you want to review, audit, modernize, refactor, clean up legacy patterns, adopt newer DJ capabilities, or upgrade `.model.json` files. +- **Example prompt:** _"Review and modernize the models in this folder."_ +- **Bundled reference:** `refactor-catalog.md` — detection heuristics and before/after examples for each pattern. + +#### `dj-migrate-ephemerals-to-ctes` + +Detects legacy ephemeral models and inlines them as Common Table Expressions (CTEs) inside their downstream consumer, then removes the now-redundant file — dissolving trivial intermediate layers. + +- **Use when:** you want to migrate, inline, flatten, consolidate, or remove ephemeral models, or standardize trivial transformations into inline CTEs. +- **Example prompt:** _"Inline the ephemeral models under intermediate/ as CTEs."_ +- **Bundled reference:** `transformation-matrix.md` — per-type inline recipes and CTE-naming rules. + +#### `dj-resolve-merge-conflicts` + +Resolves git merge, rebase, or cherry-pick conflicts the DJ way — hand-merging only the `.model.json` / `.source.json` sources of truth and regenerating the `.sql` / `.yml` siblings (never hand-merging generated files). It also helps when an incoming branch is old or diverged and you must choose between a full merge and porting specific models. + +- **Use when:** you hit conflicts in DJ files while merging/rebasing/cherry-picking, or say "resolve the merge conflicts" or "help me rebase". +- **Example prompt:** _"Resolve the merge conflicts in these .model.json files."_ +- **Bundled reference:** `staleness-and-porting.md` — the staleness assessment and guided-port recipe. + +### Governance + +#### `dj-govern-model` + +**Read-only** audit of governance posture across a model, folder, dependency tree, or the whole workspace — ownership coverage, PII / classification / compliance tagging, registered-group conformance, and prod-write posture. It reports gaps and points you at the skill to fix each one; it never edits files and never forces a project to adopt metadata it hasn't chosen. + +- **Use when:** you want to review data ownership, check PII / sensitivity / compliance tagging, find models with no owner, or assess governance coverage for a group or project. +- **Example prompt:** _"Which models in the finance group have no owner or PII tags?"_ + +### Performance diagnostics + +#### `dj-trino-analyzer` + +**Read-only** diagnosis of Trino query performance from the `QueryInfo` JSON that DJ's Query Control Center writes to `.dj/diagnostics/`. It explains slowness — broadcast-join blow-ups, data skew, blocked time, object-store scan latency, and more — and suggests `.model.json` knobs; it never edits generated SQL. Run **`DJ: Analyze Trino Query with AI`** first to produce the diagnostics. + +- **Use when:** a query is slow, you want to understand a query plan, compare two queries (for example before vs. after a config change), or investigate a specific Trino query ID. +- **Example prompt:** _"Explain why Trino query 20260101_120000_00001_abcde is slow."_ +- **Bundled references:** a six-file Trino field reference — `query-info.md`, `query-stats.md`, `stage-and-task-stats.md`, `operator-stats.md`, `types-and-enums.md`, and `recipes.md`. + +### Running dbt, Trino & git + +#### `dj-run-dbt` + +Runs a dbt command from the terminal — `compile` / `parse` / `ls` / `deps` / `docs generate` / `test`, or a warehouse-writing `run` / `build` / `seed` / `snapshot` — after activating the project's Python virtual environment and running from the dbt project directory. Read-only commands run freely; warehouse writes need explicit confirmation and never target production. + +- **Use when:** you want to compile, parse, list, test, build, or otherwise run the dbt CLI, or refresh the manifest. +- **Example prompt:** _"Compile this model."_ + +#### `dj-run-trino` + +Runs a **read-only** Trino query from the terminal to inspect warehouse data or schema — `SELECT` / `SHOW` / `DESCRIBE` / `EXPLAIN`, always with a `LIMIT` — resolving the CLI from `dj.trinoPath` and the connection from the `TRINO_*` environment. Any DDL/DML needs explicit confirmation and never hits production. + +- **Use when:** you want to query Trino, preview rows, `DESCRIBE` / `SHOW` a table, or sanity-check a value. For diagnosing captured query performance, use `dj-trino-analyzer` instead. +- **Example prompt:** _"Preview 20 rows from the orders table."_ + +#### `dj-git-workflow` + +Commits DJ work the right way — staging each `.model.json` / `.source.json` together with its generated `.sql` / `.yml` after a sync, ignoring DJ's local `.dj/` state, and following the downstream project's own commit conventions. It stops at the commit and guards against staging secrets; pushing needs your go-ahead. For merge conflicts, use `dj-resolve-merge-conflicts` instead. + +- **Use when:** you want to commit, stage, branch, or check in your DJ models, or ask what should be committed. +- **Example prompt:** _"Commit these model changes."_ + +## Feedback & more + +- The full framework reference the skills build on is generated to `.agents/dj/AGENTS.md` in your workspace once `dj.codingAgent` is enabled. +- Questions or ideas? Open a [GitHub Discussion](https://github.com/Workday/dj/discussions) or [Issue](https://github.com/Workday/dj/issues). diff --git a/docs/SETTINGS.md b/docs/SETTINGS.md index 00f0c64..50cc297 100644 --- a/docs/SETTINGS.md +++ b/docs/SETTINGS.md @@ -239,6 +239,7 @@ Takes effect on next `DJ: Sync to SQL and YML`. - Writes `AGENTS.md` to `.agents/dj/` and skill files to `.agents/skills/` at the workspace root - Legacy string values (`"github-copilot"`, `"claude-code"`, `"cline"`) still accepted but deprecated - Skills are agent-agnostic markdown files usable by any AI coding tool +- See **[Agent Skills](AGENT_SKILLS.md)** for the full catalog of shipped skills --- diff --git a/package-lock.json b/package-lock.json index 71bfd9e..554d241 100644 --- a/package-lock.json +++ b/package-lock.json @@ -13301,9 +13301,9 @@ "license": "Apache-2.0" }, "node_modules/ts-jest": { - "version": "29.4.11", - "resolved": "https://registry.npmjs.org/ts-jest/-/ts-jest-29.4.11.tgz", - "integrity": "sha512-IrFl7l9AuB/qrNw5quqvAv/hmKMb8dhWOH4jQOGo0Oq8tCeo1O86/iTFG1FaRimgUkF13l4PcepO8ATFT6Ns4g==", + "version": "29.4.12", + "resolved": "https://registry.npmjs.org/ts-jest/-/ts-jest-29.4.12.tgz", + "integrity": "sha512-Ov6ClY53Fflh6BGAnY2DlTq1hYDrTycz2PVTXBWFW2CU+9zrEqAp9fWdGXl42EXO5RLSFAcAZ2JFKbP+zBTFfw==", "dev": true, "license": "MIT", "dependencies": { @@ -13313,7 +13313,7 @@ "json5": "^2.2.3", "lodash.memoize": "^4.1.2", "make-error": "^1.3.6", - "semver": "^7.8.0", + "semver": "^7.8.5", "type-fest": "^4.41.0", "yargs-parser": "^21.1.1" }, diff --git a/src/services/__tests__/agent.test.ts b/src/services/__tests__/agent.test.ts index fe303d6..a5938c9 100644 --- a/src/services/__tests__/agent.test.ts +++ b/src/services/__tests__/agent.test.ts @@ -3,10 +3,8 @@ import * as fs from 'fs'; import * as path from 'path'; const SKILLS_DIR = path.resolve(__dirname, '../../../templates/skills'); -const AGENTS_TEMPLATE = path.resolve( - __dirname, - '../../../templates/_AGENTS.md', -); +const AGENTS_DJ_DIR = path.resolve(__dirname, '../../../templates/_agents-dj'); +const AGENTS_TEMPLATE = path.join(AGENTS_DJ_DIR, '_AGENTS.md'); describe('Skills', () => { const skillDirs = fs @@ -85,4 +83,39 @@ describe('Skills', () => { expect(content).toBeTruthy(); expect(content).toContain('DJ (Data JSON) Framework'); }); + + test('_agents-dj relative markdown links resolve to bundled files', () => { + const mdFiles: string[] = []; + const walk = (dir: string) => { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) { + walk(full); + } else if (entry.name.endsWith('.md')) { + mdFiles.push(full); + } + } + }; + walk(AGENTS_DJ_DIR); + + for (const mdFile of mdFiles) { + const content = fs.readFileSync(mdFile, 'utf-8'); + const links = [...content.matchAll(/\]\(([^)\s]+)\)/g)].map((m) => m[1]); + for (const target of links) { + // Skip external URLs (scheme prefix) and in-page anchors. + if (/^[a-z][a-z+.-]*:/i.test(target) || target.startsWith('#')) { + continue; + } + const resolved = path.resolve( + path.dirname(mdFile), + target.split('#')[0], + ); + const link = { in: path.relative(AGENTS_DJ_DIR, mdFile), target }; + expect({ ...link, exists: fs.existsSync(resolved) }).toEqual({ + ...link, + exists: true, + }); + } + } + }); }); diff --git a/src/services/agent/utils.ts b/src/services/agent/utils.ts deleted file mode 100644 index 06abea8..0000000 --- a/src/services/agent/utils.ts +++ /dev/null @@ -1,12 +0,0 @@ -import * as fs from 'fs'; -import * as path from 'path'; - -let agentsMdCache: string | undefined; - -export function generateAgentsMd(): string { - agentsMdCache ??= fs.readFileSync( - path.resolve(__dirname, '../../templates/_AGENTS.md'), - 'utf-8', - ); - return agentsMdCache; -} diff --git a/src/services/constants.ts b/src/services/constants.ts index 58aa846..669016d 100644 --- a/src/services/constants.ts +++ b/src/services/constants.ts @@ -11,6 +11,10 @@ export const BASE_MACROS_PATH = path.join(__dirname, '../../macros'); export const BASE_TESTS_PATH = path.join(__dirname, '../../macros/tests'); export const BASE_SCHEMAS_PATH = path.join(__dirname, '../../schemas'); export const BASE_SKILLS_PATH = path.join(__dirname, '../../templates/skills'); +export const BASE_AGENTS_DJ_PATH = path.join( + __dirname, + '../../templates/_agents-dj', +); // Extension constants export const OUTPUT_CHANNEL_NAME = 'DJ'; diff --git a/src/services/dbt.ts b/src/services/dbt.ts index 153dc4c..a6ad3dd 100644 --- a/src/services/dbt.ts +++ b/src/services/dbt.ts @@ -1,4 +1,3 @@ -import { generateAgentsMd } from '@services/agent/utils'; import type { Coder } from '@services/coder'; import type { CoderFileInfo } from '@services/coder/types'; import { @@ -7,6 +6,7 @@ import { isDbtProjectNameConfigured, } from '@services/config'; import { + BASE_AGENTS_DJ_PATH, BASE_AIRFLOW_PATH, BASE_MACROS_PATH, BASE_SKILLS_PATH, @@ -1853,7 +1853,10 @@ ${macro.macro_sql}`; } /** - * Write AGENTS.md to the workspace root's .agents/dj/ directory for AI coding agents. + * Write the DJ agent context to the workspace root's .agents/dj/ directory + * for AI coding agents. Deploys the hub `AGENTS.md` plus its `reference/` + * files (loaded on demand), mirroring the Agent Skills progressive-disclosure + * layout (https://agentskills.io). */ async writeAgentsMd(): Promise { const { codingAgent } = getDjConfig(); @@ -1863,17 +1866,8 @@ ${macro.macro_sql}`; try { this.log.info('WRITING AGENTS.MD'); - const agentsMdContent = generateAgentsMd(); - const agentsMdPath = path.join( - WORKSPACE_ROOT, - '.agents', - 'dj', - 'AGENTS.md', - ); - await vscode.workspace.fs.writeFile( - vscode.Uri.file(agentsMdPath), - Buffer.from(agentsMdContent), - ); + const targetDir = path.join(WORKSPACE_ROOT, '.agents', 'dj'); + await this.copyTemplateDirectoryRecursive(BASE_AGENTS_DJ_PATH, targetDir); } catch (err) { this.log.error('Error writing AGENTS.md:', err); } @@ -1910,7 +1904,7 @@ ${macro.macro_sql}`; skillDirName, ); - await this.copySkillDirectoryRecursive(sourceDir, targetDir); + await this.copyTemplateDirectoryRecursive(sourceDir, targetDir); } catch (err: unknown) { this.log.error(`Error writing skill ${skillDirName}:`, err); } @@ -1946,16 +1940,17 @@ ${macro.macro_sql}`; } /** - * Recursively copy a skill directory (files + subdirectories) into the - * workspace target. Subdirectories like `references/`, `scripts/`, `assets/` - * are part of the Agent Skills open standard (https://agentskills.io) for - * progressive disclosure of skill content. + * Recursively copy a template directory (files + subdirectories) into the + * workspace target. Subdirectories like `reference/`, `references/`, + * `scripts/`, `assets/` are part of the Agent Skills open standard + * (https://agentskills.io) for progressive disclosure of content. * * Leading underscores on template filenames are stripped (e.g. `_SKILL.md` - * → `SKILL.md`) so they aren't picked up as skills inside this repo's own - * `templates/skills` directory. Subdirectory names are preserved verbatim. + * → `SKILL.md`, `_AGENTS.md` → `AGENTS.md`) so they aren't picked up as + * agent files inside this repo's own `templates` directory. Subdirectory + * names are preserved verbatim. */ - private async copySkillDirectoryRecursive( + private async copyTemplateDirectoryRecursive( sourceDir: string, targetDir: string, ): Promise { @@ -1978,7 +1973,10 @@ ${macro.macro_sql}`; await vscode.workspace.fs.writeFile(targetPath, content); } else if (entryType === vscode.FileType.Directory) { const targetSubDir = path.join(targetDir, entryName); - await this.copySkillDirectoryRecursive(sourceEntryPath, targetSubDir); + await this.copyTemplateDirectoryRecursive( + sourceEntryPath, + targetSubDir, + ); } } } diff --git a/templates/_AGENTS.md b/templates/_AGENTS.md deleted file mode 100644 index 531dc10..0000000 --- a/templates/_AGENTS.md +++ /dev/null @@ -1,1315 +0,0 @@ -# AGENTS.md — DJ (Data JSON) Framework Guide - -> This file is auto-generated by the Workday DJ (Data JSON) Framework VS Code extension. -> It provides LLMs with the context needed to create and modify `.model.json` and `.source.json` files in this dbt project. - -## Overview - -This project uses the **DJ (Data JSON) Framework** — a JSON-based abstraction layer on top of dbt. Instead of writing raw SQL and YML files by hand, developers author `.model.json` and `.source.json` files. The DJ (Data JSON) Framework then **auto-generates** the corresponding `.sql` and `.yml` files via a process called "JSON Sync." You should **never** manually edit the generated `.sql` or `.yml` files — only edit the `.model.json` and `.source.json` files. - -All JSON files use the **JSONC** format (JSON with Comments). Trailing commas are allowed. Preserve any existing comments when editing files. - ---- - -## Project Structure - -The dbt project root (where `dbt_project.yml` lives) contains the following structure: - -```text -/ -├── models/ -│ ├── staging/ # stg__ models (stg_select_source, stg_select_model, stg_union_sources) -│ │ └── / -│ │ └── / -│ │ ├── .model.json -│ │ ├── .sql (auto-generated, do NOT edit) -│ │ └── .yml (auto-generated, do NOT edit) -│ ├── intermediate/ # int__ models (int_select_model, int_join_models, int_union_models, etc.) -│ │ └── / -│ │ └── / -│ ├── marts/ # mart__ models (mart_select_model, mart_join_models) -│ │ └── / -│ │ └── / -│ └── sources/ # source definitions -│ └── / -│ ├── __.source.json -│ └── __.yml (auto-generated, do NOT edit) -├── seeds/ -│ └── / -│ └── seed____.csv -├── macros/ -└── dbt_project.yml -``` - ---- - -## Model Naming Convention - -Model names follow the pattern: `______` - -- **layer**: Derived from the model `type` field (`stg`, `int`, or `mart`) -- **group**: Team or project classification (e.g., `finance`, `analytics`, `sales`) -- **topic**: Subject area within the group (e.g., `billing`, `orders`, `customers`) -- **name**: Descriptive name for the model (e.g., `daily_summary`, `account_hierarchy`) - -## Model Types - -### 1. `stg_select_source` — Staging: Select from a Source - -Selects columns from a raw data source table. - -```jsonc -{ - "type": "stg_select_source", - "group": "my_group", - "topic": "my_topic", - "name": "raw_data_conformed", - "materialized": "incremental", // optional: "incremental" or "ephemeral" - "from": { - "source": "my_database__my_schema.my_table", // format: __. - }, - "select": [ - "account_id", // simple column reference (string) - "region", - { - "name": "cost", // column with additional config - "type": "fct", // "dim" (dimension) or "fct" (fact/measure) - "expr": "CAST(cost AS double)", // optional SQL expression override - }, - { - "name": "event_date", - "type": "dim", - "data_type": "date", // optional Trino data type - }, - ], - "where": { - // optional filter - "and": [{ "expr": "cost > 0" }], - }, -} -``` - -**Required fields**: `type`, `group`, `topic`, `name`, `from.source`, `select` - -### 2. `stg_select_model` — Staging: Select from Another Model - -Selects from another model (commonly used for seeds). - -```jsonc -{ - "type": "stg_select_model", - "group": "my_group", - "topic": "my_topic", - "name": "lookup_mapping", - "from": { - "model": "seed__my_topic__lookup_mapping", - }, - "select": ["key_column", "value_column"], -} -``` - -**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `select` - -### 3. `stg_union_sources` — Staging: Union Multiple Sources - -Unions multiple source tables. - -```jsonc -{ - "type": "stg_union_sources", - "group": "my_group", - "topic": "my_topic", - "name": "combined_accounts", - "from": { - "source": "my_database__my_schema.accounts_us", - "union": { - "sources": [ - "my_database__my_schema.accounts_eu", - "my_database__my_schema.accounts_apac", - ], - }, - }, - "select": ["account_id", "account_name"], -} -``` - -**Required fields**: `type`, `group`, `topic`, `name`, `from.source`, `from.union.sources` - -### 4. `int_select_model` — Intermediate: Select from a Model - -Transforms data from a single upstream model. Supports optional `from.rollup` for time-grain re-aggregation (provides `int_rollup_model` functionality with more control over columns). - -```jsonc -{ - "type": "int_select_model", - "group": "my_group", - "topic": "my_topic", - "name": "daily_summary", - "materialized": "incremental", - "from": { - "model": "stg__my_group__my_topic__raw_data_conformed", - // optional: re-aggregate to coarser time grain - "rollup": { - "interval": "day", // "day", "hour", "month", "year" - }, - }, - "select": [ - "account_id", - { - "name": "cost", - "type": "fct", - "agg": "sum", // auto-creates aggregation columns: sum, count, min, max, hll, tdigest - }, - { - "name": "datetime", - "interval": "day", // interval column: "day", "hour", "month", "year" - }, - ], - "group_by": [ - { "type": "dims" }, // group by all dimension columns - ], - "where": "cost > 0", // simple string where clause -} -``` - -**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `select` - -### 5. `int_join_models` — Intermediate: Join Multiple Models - -Joins a primary model with one or more additional models. Supports optional `from.rollup` for time-grain re-aggregation alongside joins. - -```jsonc -{ - "type": "int_join_models", - "group": "my_group", - "topic": "my_topic", - "name": "enriched_daily", - "materialized": "incremental", - "from": { - "model": "int__my_group__my_topic__daily_summary", - // optional: re-aggregate to coarser time grain - "rollup": { - "interval": "day", // "day", "hour", "month", "year" - }, - "join": [ - { - "model": "int__my_group__other_topic__dimension_table", - "type": "inner", // "left", "inner", "right", "full", "cross" - "on": { - "and": [ - "account_id", // shorthand: join on same column name - "event_date", - { "expr": "a.region = b.region" }, // or explicit SQL expression - ], - }, - }, - ], - }, - "select": [ - { - "model": "int__my_group__my_topic__daily_summary", - "type": "dims_from_model", // "all_from_model", "dims_from_model", "fcts_from_model" - "include": ["account_id", "region"], // optional: filter which columns - }, - { - "model": "int__my_group__other_topic__dimension_table", - "type": "dims_from_model", - }, - { - "name": "allocated_cost", - "type": "fct", - "expr": "sum(a.cost * b.ratio)", - }, - ], - "group_by": [{ "type": "dims" }], -} -``` - -**Required fields**: `type`, `group`, `name`, `from.model`, `from.join`, `select` - -### 6. `int_union_models` — Intermediate: Union Multiple Models - -```jsonc -{ - "type": "int_union_models", - "group": "my_group", - "topic": "my_topic", - "name": "all_providers_daily", - "from": { - "model": "int__my_group__provider_a__daily", - "union": { - "models": [ - "int__my_group__provider_b__daily", - "int__my_group__provider_c__daily", - ], - }, - }, -} -``` - -**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `from.union.models` - -### 7. `int_rollup_model` — Intermediate: Time-based Rollup - -Aggregates data to a coarser time interval. - -```jsonc -{ - "type": "int_rollup_model", - "group": "my_group", - "topic": "my_topic", - "name": "daily_from_hourly", - "materialized": "incremental", - "from": { - "model": "int__my_group__my_topic__hourly_summary", - "rollup": { - "interval": "day", // "day", "hour", "month", "year" - }, - }, -} -``` - -**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `from.rollup.interval` - -### 8. `int_lookback_model` — Intermediate: Trailing Window Aggregation - -Aggregates over a trailing number of days. - -```jsonc -{ - "type": "int_lookback_model", - "group": "my_group", - "topic": "my_topic", - "name": "trailing_30d", - "materialized": "incremental", - "from": { - "model": "int__my_group__my_topic__daily_summary", - "lookback": { - "days": 30, - "exclude_event_date": false, // optional - }, - }, - "select": ["account_id", { "name": "cost", "type": "fct", "agg": "sum" }], - "group_by": [{ "type": "dims" }], -} -``` - -**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `from.lookback.days`, `select` - -### 9. `int_join_column` — Intermediate: Cross Join on Unnested Column - -Cross joins a model with an unnested array column. - -**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `from.join.column`, `select` - -### 10. `mart_select_model` — Mart: Select from a Model - -Final business-ready model selecting from an intermediate model. - -```jsonc -{ - "type": "mart_select_model", - "group": "my_group", - "topic": "my_topic", - "name": "accounts_daily", - "from": { - "model": "int__my_group__my_topic__daily_summary", - }, - "select": ["account_id", "cost_sum"], -} -``` - -**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `select` - -### 11. `mart_join_models` — Mart: Join Multiple Models - -Final business-ready model that joins multiple intermediate models. Same join syntax as `int_join_models`. - -**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `from.join`, `select` - -### Advanced: CTEs, rollup, shorthands, subqueries - -For **`int_select_model`**, **`int_join_models`**, **`int_union_models`**, **`mart_select_model`**, **`mart_join_models`** (not staging). **Shapes and required keys** live in **`.dj/schemas/model.type..schema.json`** and **`$ref`** targets — read those first; this section is a map, not a full spec. - -- **CTEs**: Optional ordered **`ctes`**. **`model.ctes.schema.json`**, **`model.cte.schema.json`**. -- **`from`**: Each type’s **`from`** **`anyOf`** lists legal combinations (**`model`**, **`cte`**, **`join`**, optional **`rollup`** on **`int_*` select/join only** — not on marts). -- **Rollup on select/join**: Optional **`rollup`** on **`from.model`** for **`int_select_model`** and **`int_join_models`** only (not marts). Keeps a normal **`select`** / join; coarser **`interval`** triggers **re-aggregation** of declarative **`agg`/`aggs`**. **`model.from.rollup.schema.json`**. **`group_by` / `agg` / `expr`**: **#9**–**#10**. -- **Rollup inside a CTE**: Optional **`rollup`** on a CTE's **`from.model`** or **`from.cte`** (not **`from.source`**, not **`from.union`**). Re-aggregates that CTE's source to a coarser grain — same DATE_TRUNC + suffix-agg + GROUP BY behavior as the model-level rollup, but scoped to one stage of the pipeline. Available on every CTE-supporting model type. **`exclude_datetime`** / **`exclude_framework_artifacts`** at the same scope is rejected as a conflict; chained rollups (e.g. month CTE feeding a year CTE) work end-to-end. -- **Shorthands & CTE columns**: **`dims_from_*`**, **`fcts_from_*`**, **`all_from_*`** and explicit CTE column objects — **`model.select.model.schema.json`**, **`model.select.cte.schema.json`**, related **`model.select.*`**. CTE bulk selects support **`exclude`/`include`** filters and **inherit dim/fct types** from upstream. -- **`where` / `having`**: Nested **`subquery`** — **`model.subquery.schema.json`**. -- **`"dims"` shorthand**: **`group_by: "dims"`** equivalent to **`[{ "type": "dims" }]`**; join **`on: "dims"`** auto-joins on all shared dimension columns — **`model.group_by.schema.json`**. -- **Materialization**: String **`"incremental"`** / **`"ephemeral"`** or structured object with **`type`**, **`format`**, **`partitions`**, **`strategy`**, **`database`** — **`model.materialization.schema.json`**. - ---- - -## Source Files (`.source.json`) - -Source files define external database tables that staging models read from. They are placed at: -`models/sources//__.source.json` - -### Source Structure - -```jsonc -{ - "database": "my_database", // catalog/database name - "schema": "my_schema", // schema name - "tables": [ - { - "name": "my_table", // table name - "columns": [ - { - "name": "account_id", - "data_type": "varchar", // Trino data type - }, - { - "name": "cost", - "data_type": "double", - "description": "The raw cost amount", // optional - }, - ], - }, - ], -} -``` - -**Required fields**: `database`, `schema`, `tables` -**Required per table**: `name`, `columns` -**Required per column**: `name`, `data_type` - -### Source Naming - -The source name is derived as: `__` - -When referenced in a model's `from.source`, use: `__.` - -### Source ETL Configuration - -Sources can include ETL metadata in the `meta` field (at either schema or table level) to control scheduling: - -```jsonc -{ - "database": "my_database", - "schema": "my_schema", - "meta": { - "etl": { - "active": true, // whether ETL monitors this source - "backfill_start": "2024-01-01", // date to start backfilling from (YYYY-MM-DD) - "type": "event_count" // "event_count" (default) or "run_schedule" - }, - "event_datetime": { - "expr": "event_timestamp" // expression to extract event datetime - }, - "partition_date": { - "expr": "dt", // partition date expression - "interval": "day" // "day" or "month" - } - }, - "tables": [...] -} -``` - -#### ETL Types - -- **`event_count`** (default): The scheduler queries this source to detect which event dates have new or changed data, then runs downstream models only for those dates. Requires `backfill_start`. -- **`run_schedule`**: The scheduler runs downstream models on a fixed schedule regardless of data changes. Does not require `backfill_start`. - -### Source Partitions - -Sources can define partition filters to enable efficient querying: - -```jsonc -{ - "meta": { - "partitions": [ - { - "type": "event_dates", // filter by project event dates - "expr": "dt", // partition column expression - }, - { - "type": "gte", // comparison: "eq", "gt", "gte", "lt", "lte", "neq" - "expr": "created_date", - "value": "2024-01-01", - }, - ], - }, -} -``` - -### Optional Source Fields - -- `description`: Description of the source -- `freshness`: dbt freshness configuration object, or `null` to disable freshness checks for the entire source -- `loaded_at_field`: Column indicating data freshness -- `meta.portal_partition_columns`: Custom partition columns for the framework -- `meta.portal_source_count`: Custom source count configuration -- `meta.table_function`: Table function configuration -- `meta.where`: Static where clause applied whenever the source is queried -- Per-table `meta`: Table-level overrides for the same meta fields above -- Per-table `freshness`: Table-level freshness config or `null` to disable for a specific table -- Per-table `loaded_at_field`: Table-level override for the timestamp field used in freshness checks - ---- - -## Select Column Types - -### Simple String Reference - -```jsonc -"column_name" -``` - -Selects a column by name with default dimension type. - -### Named Column (`dim` or `fct`) - -```jsonc -{ - "name": "column_name", - "type": "dim", // "dim" (dimension) or "fct" (fact/measure) — default is "dim" - "data_type": "varchar", // optional: Trino data type - "description": "Description", // optional - "expr": "CAST(col AS varchar)", // optional: SQL expression override -} -``` - -### Aggregated Column - -```jsonc -{ - "name": "cost", - "type": "fct", - "agg": "sum", // "sum", "count", "min", "max", "hll", "tdigest" -} -``` - -This auto-creates an aggregation column named `_` (e.g., `cost_sum`). - -### Multi-Aggregations - -```jsonc -{ - "name": "cost", - "type": "fct", - "aggs": ["sum", "count", "min", "max"], -} -``` - -### From Another Model (in join/union models) - -```jsonc -{ - "model": "int__my_group__my_topic__daily_summary", - "type": "dims_from_model", // "all_from_model", "dims_from_model", "fcts_from_model" -} -``` - -With optional include/exclude: - -```jsonc -{ - "model": "int__my_group__my_topic__daily_summary", - "type": "dims_from_model", - "include": ["account_id", "region"], - "exclude": ["internal_id"], -} -``` - -### Named Column from Specific Model - -```jsonc -{ - "model": "int__my_group__my_topic__daily_summary", - "name": "cost", - "type": "fct", -} -``` - -### From Source (in staging models) - -```jsonc -{ - "source": "my_database__my_schema.my_table", - "type": "all_from_source", -} -``` - -### From CTE (in models with `ctes`) - -```jsonc -{ - "cte": "my_cte_name", - "type": "all_from_cte", // "all_from_cte", "dims_from_cte", "fcts_from_cte" -} -``` - -With optional include/exclude: - -```jsonc -{ - "cte": "my_cte_name", - "type": "dims_from_cte", - "include": ["account_id", "region"], -} -``` - -Named column from a CTE: - -```jsonc -{ - "cte": "my_cte_name", - "name": "cost", - "type": "fct", -} -``` - -### Interval (Datetime) - -```jsonc -{ - "name": "datetime", - "interval": "day", // "day", "hour", "month", "year" -} -``` - ---- - -## Common Optional Model Fields - -| Field | Type | Description | -| ---------------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `description` | string | Model description | -| `tags` | array | Tags for categorization, e.g. `["my_tag", "my_group"]` | -| `materialized` | string | Legacy: `"incremental"` or `"ephemeral"` (default is view-like). Prefer `materialization` instead. | -| `materialization` | string/object | Preferred. String `"incremental"` or `"ephemeral"`, or object `{ "type": "incremental", "format"?, "partitions"?, "strategy"?, "database"? }`. See Materialization section. | -| `incremental_strategy` | object | Legacy: `{ "type": "delete+insert" }` or `{ "type": "merge", "unique_key": "id" }`. Prefer `materialization.strategy`. | -| `sql_hooks` | object | `{ "pre": "SET ...", "post": "..." }` — SQL to run before/after (staging and intermediate only) | -| `partitioned_by` | array | Legacy: Column(s) to partition by. Prefer `materialization.partitions`. | -| `group_by` | string/array | `"dims"` or `[{ "type": "dims" }]` or `["col1", "col2"]` or `[{ "expr": "..." }]` | -| `where` | string/object | Filter clause — simple string or `{ "and": [...], "or": [...] }` | -| `having` | object | HAVING clause (same shape as `where`) | -| `order_by` | array | ORDER BY columns | -| `limit` | integer | LIMIT clause | -| `offset` | integer | OFFSET clause | -| `exclude_date_filter` | boolean | Skip auto date filtering | -| `exclude_daily_filter` | boolean | Skip daily partition filter | -| `exclude_portal_partition_columns` | boolean/array | Drop portal partition columns. `true` drops all; an array (e.g. `["portal_partition_hourly"]`) drops only the named ones | -| `exclude_portal_source_count` | boolean | Don't add portal source count | -| `data_tests` | array | dbt test configurations | -| `lightdash` | object | Lightdash BI tool configuration | -| `meta` | object | Free-form user-defined metadata (see "Custom Meta" section below) | - ---- - -## Lightdash Configuration - -### Model-level Lightdash - -```jsonc -{ - "lightdash": { - "table": { - "label": "Daily Cost Summary", - "group_label": "Cost Analytics", - "ai_hint": "Daily aggregated costs by account and region", - "sql_filter": "cost_sum > 0", - "required_filters": ["portal_partition_daily"], - }, - "metrics": [ - { - "name": "total_cost", - "type": "sum", - "label": "Total Cost", - "group_label": "Cost Metrics", - "sql": "${TABLE}.cost_sum", - "round": 0, - "format": "usd", - }, - ], - }, -} -``` - -### Column-level Lightdash - -```jsonc -{ - "name": "cost", - "type": "fct", - "lightdash": { - "dimension": { "hidden": true, "label": "Raw Cost", "group_label": "Cost" }, - "metrics": [ - { - "name": "total_cost", - "type": "sum", - "label": "Total Cost", - "format": "usd", - "round": 0, - }, - ], - "metrics_merge": { - "format": "usd", - "round": 0, - "group_label": "Cost Metrics", - }, - }, -} -``` - -## Lightdash Dashboards as Code - -The DJ extension also exposes Lightdash's [Dashboards as Code](https://docs.lightdash.com/guides/developer/dashboards-as-code) workflow via the `DJ: Lightdash — Dashboards as Code` command. This is **separate from** the model-level `lightdash` config above (which generates `meta.dimensions` / `meta.metrics` blocks in the dbt-managed YAML). Dashboards-as-Code lets you author the actual saved charts and dashboards (the things visible in the Lightdash UI) as version-control-friendly YAML files. - -### Layout - -By default, the extension's `lightdash download` writes: - -```text -/lightdash/ -├── charts/ -│ └── .yml -└── dashboards/ - └── .yml -``` - -The base path is configurable via the `dj.lightdash.dashboardsAsCodePath` extension setting. The slug in the filename is also the slug used by the Lightdash CLI's `-c` / `-d` flags. - -### When to edit these files - -These YAML files are **inputs to `lightdash upload`**. Edit them when the user wants to tweak a chart's filters, axis labels, dashboard tiles, etc. without clicking through the Lightdash UI. Typical workflow: - -1. User runs the Download tab (entire project or specific charts/dashboards). -2. You edit `/lightdash/charts/.yml` or `…/dashboards/.yml`. -3. User runs the Upload tab (selection-driven by default — only edited files get pushed). - -### YAML shape - -Both file types are validated by official Lightdash JSON schemas (the extension auto-registers them with the Red Hat YAML extension). Top-level keys: - -- **Chart** (`charts/*.yml`): `version`, `slug`, `name`, `description?`, `chartConfig`, `tableConfig`, `metricQuery` (`exploreName`, `dimensions`, `metrics`, `filters`, `sorts`, `limit`, `tableCalculations`, `additionalMetrics`), `dashboardSlug?`, `spaceSlug?`, `tags?`. -- **Dashboard** (`dashboards/*.yml`): `version`, `slug`, `name`, `description?`, `tabs?`, `tiles[]` (each tile has `type`, `properties`, layout `x/y/w/h`), `filters?`, `spaceSlug?`, `tags?`. - -Read the file's `# yaml-language-server: $schema=…` header (or the auto-installed `yaml.schemas` binding) for the authoritative shape — do not invent fields. - -### Editing rules - -- **Do not change `slug`.** The slug is the primary key the upload uses to match local files to remote charts/dashboards. Renaming the slug creates a new resource on upload. -- **Do not change `version`.** It pins the schema; bumping it by hand will break the upload. -- **Preserve unfamiliar keys.** The schemas evolve; keep any field you do not recognize as-is so downloads stay round-trippable. -- **Reference existing dbt models, not raw tables.** `metricQuery.exploreName` and `metrics`/`dimensions` reference the model's Lightdash `table` / dimension / metric names — confirm they exist by reading the model's `.model.json` `lightdash` block. If a metric the user wants does not exist on the model, add it to the `.model.json` (regenerates the dbt YAML) before referencing it from the chart YAML. -- **Dashboard tiles must reference real chart slugs.** A dashboard tile's `properties.savedChartSlug` (or equivalent) must match a chart slug that exists either locally under `charts/` or already on Lightdash. -- **Use the extension's webview to invoke the CLI.** Do not shell out to `lightdash download` / `lightdash upload` directly — the webview handles auth, working directory, and YAML schema sync. - ---- - -## Custom Meta (Free-form) - -Both `.model.json` and `.source.json` accept **free-form user-defined keys** on their `meta` blocks. Use this to attach arbitrary metadata (ownership, compliance tags, process info, SLAs, etc.) that you want to surface in the generated `.yml` and consume downstream (dbt docs, Lightdash, custom tooling). - -Schemas: `model.meta.schema.json`, `column.meta.schema.json`, `source.meta.schema.json`, `source.table.meta.schema.json`. - -### Model-level meta - -Root `meta` block on any model type: - -```jsonc -{ - "type": "mart_select_model", - "group": "finance", - "topic": "billing", - "name": "accounts_daily", - "from": { "model": "int__finance__billing__accounts_daily" }, - "select": [...], - "meta": { - "owner": "finops-team", - "owner_slack": "#finops-team", - "freshness_sla": "daily by 06:00 UTC", - "pii": false, - }, -} -``` - -- Free-form keys flow through to the emitted `.yml` verbatim. -- **No automatic inheritance**: each model declares its own model-level meta (model-level meta is not inherited from upstream models). - -### Column-level meta - -Any select item on `.model.json` accepts a `meta` object: - -```jsonc -{ - "name": "email", - "type": "dim", - "meta": { "pii": true, "compliance": ["gdpr", "ccpa"] }, -} -``` - -- **Inheritance**: Column-level free-form meta IS inherited through **clean passthrough selects** (plain string selects and named-column selects without `expr`). `expr`-based selects (including `expr`-based renames) do **not** inherit meta. -- Downstream per-key overrides work as expected: a downstream column meta key overwrites the inherited key; keys the downstream doesn't declare stay inherited. - -### Framework-reserved keys under `meta` - -A small set of keys are owned by the framework — it writes them into the emitted YAML's `meta` block from structured sibling fields. Authoring any of these under `meta` directly is allowed by the schema but will be silently overwritten at emit time, and the extension surfaces a **Warning-severity diagnostic** in the Problems tab pointing to the canonical field. - -| Scope | Key | Canonical authoring location | -| ------ | ------------------------------ | -------------------------------------------------------- | -| model | `metrics` | `lightdash.metrics` on the model | -| model | `portal_partition_columns` | framework-derived; do not author | -| model | `local_tags` | `tags: [{ "type": "local", "tag": "..." }]` on the model | -| model | `case_sensitive` | `lightdash.case_sensitive` on the model | -| model | (any key on `lightdash.table`) | `lightdash.table.` on the model | -| column | `type` | `type` on the select item | -| column | `dimension` | `lightdash.dimension` on the select item | -| column | `metrics` | `lightdash.metrics` on the select item | -| column | `case_sensitive` | `lightdash.case_sensitive` on the select item | -| column | `origin` | framework-derived from upstream lookup; do not author | - ---- - -## Materialization & Incremental Strategies - -| Layer | Default | Common Override | -| ------ | --------- | --------------------------------------------------------- | -| `stg` | ephemeral | `"materialization": "incremental"` for large sources | -| `int` | ephemeral | `"materialization": "incremental"` for large aggregations | -| `mart` | view | Not configurable — marts are always views | - -**Materialization types**: `ephemeral` (CTE, no table), `incremental` (processes new data only) - -### Materialization (Preferred) - -Use the `materialization` field instead of the legacy `materialized` + `incremental_strategy` + `partitioned_by` combination. It accepts a string shorthand or a structured object. - -**String shorthand** (equivalent to legacy `materialized`): - -```jsonc -{ - "materialization": "incremental", // or "ephemeral" -} -``` - -**Structured form** (full control): - -```jsonc -{ - "materialization": { - "type": "incremental", - "format": "iceberg", // optional: "delta_lake", "hive", or "iceberg" - "partitions": ["portal_partition_daily"], // optional: columns to partition by - "bucket": { "column": "tenant_name", "count": 32 }, // optional: { column, count } or an array of them - "sorted_by": ["tenant_name", "product_area"], // optional: columns to sort by within each file/bucket - "strategy": { "type": "delete+insert" }, // optional: see "Incremental strategies" below - "database": "custom_database", // optional: override target database - }, -} -``` - -- **`format`**: Controls storage format. Defaults to the project's `storage_type` variable in `dbt_project.yml`. Iceberg uses `partitioning` keyword; Delta Lake/Hive uses `partitioned_by`. -- **`bucket`**: Hash-bucket the table by one or more columns. On **Iceberg** each entry becomes a `bucket(column, count)` transform inside `partitioning` (per-column counts allowed). On **Hive/Glue** it emits `bucketed_by` + a single shared `bucket_count` (all entries must use the same `count`). **Not supported on Delta Lake.** The bucket column must be one of the model's `select` columns. -- **`sorted_by`**: Columns to sort data by within each written file. On **Iceberg** it is a standalone sort order; on **Hive/Glue** it sorts within buckets and **requires `bucket`**. **Not supported on Delta Lake.** Columns sort ascending. -- **`strategy`**: See "Incremental strategies" below. If omitted, the extension default applies (configurable via `dj.materialization.defaultIncrementalStrategy`, defaults to `overwrite_existing_partitions`). - -#### Incremental strategies (dbt-trino) - -| Strategy | Shape | When to use | Caveat | -| -------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `append` | `{ "type": "append" }` | Fast insert-only; no de-dup | Upstream must guarantee no duplicates in the new slice | -| `delete+insert` | `{ "type": "delete+insert", "unique_key": "..." }` | Partition-safe upsert (**safe default**) | `unique_key` is auto-derived from partitions when omitted | -| `merge` | `{ "type": "merge", "unique_key": "id", "merge_update_columns": [...], "merge_exclude_columns": [...] }` | Row-level upsert on a primary key | **dbt-trino requires Iceberg format.** Set `materialization.format: "iceberg"` or the project var `storage_type: iceberg` | -| `overwrite_existing_partitions` | `{ "type": "overwrite_existing_partitions" }` | Drop & rewrite only partitions present in the new slice | **Requires a custom dbt macro in your project** (e.g. `get_incremental_overwrite_existing_partitions_sql`). The DJ (Data JSON) Framework does NOT ship this macro and dbt-trino does NOT provide it natively. `unique_key` is **not applicable** for this strategy the macro derives partitions from the new slice itself, and the schema rejects `unique_key`. If your project does not define the macro, use `{ "type": "delete+insert" }` instead, behavior is equivalent for partition-aligned daily/monthly incrementals when `unique_key` is the partition column. | -| `dj_iceberg_partition_overwrite` | `{ "type": "dj_iceberg_partition_overwrite" }` | Drop & rewrite only partitions present in the new slice on **Iceberg** tables | **Shipped by DJ.** No consumer macro required, `macros/strategies.sql` is auto-copied to `/macros/_ext_/strategies.sql` on **DJ: Refresh Projects**. The dispatch macro is `get_incremental_dj_iceberg_partition_overwrite_sql`. **Requires Iceberg format**: set `materialization.format: "iceberg"` or project var `storage_type: iceberg`; otherwise DJ flags it in the Problems tab. `unique_key` is **not applicable**, the macro derives partitions from the new slice itself. On Delta Lake / Hive use `{ "type": "delete+insert" }` instead. | - -### Legacy Incremental Configuration - -Still supported but prefer `materialization` above: - -```jsonc -{ - "materialized": "incremental", - "incremental_strategy": { "type": "delete+insert" }, // or "merge" with "unique_key" - "partitioned_by": ["portal_partition_daily"], -} -``` - -**Date filter options**: `"exclude_date_filter": true` (skip all date filtering), `"exclude_daily_filter": true` (skip daily partition filter only) - ---- - -## Portal-Specific Columns - -The DJ (Data JSON) Framework automatically adds these columns: - -### `portal_source_count` - -Auto-generated `count(*)` for row tracking. Exclude with `"exclude_portal_source_count": true`. - -### Partition Columns - -Created from `interval` on datetime columns: - -| Interval | Generated Column | -| --------- | -------------------------------- | -| `"day"` | `portal_partition_daily` | -| `"hour"` | `portal_partition_hourly` | -| `"month"` | `portal_partition_monthly` | -| `"year"` | (none — only truncates datetime) | - -Drop all of them with `"exclude_portal_partition_columns": true`, or drop any -subset with an array, e.g. `"exclude_portal_partition_columns": ["portal_partition_hourly"]` -removes only the listed columns and keeps the rest. An array overrides -`exclude_framework_artifacts` at the same scope (narrowing its all-partitions -exclusion to just the listed columns). - -### Source-Level Configuration - -```jsonc -{ - "meta": { - "portal_source_count": { "exclude": true }, - "portal_partition_columns": { "daily": "custom_date_column" }, - }, -} -``` - ---- - -## Tags - -Tags are used for model categorization, filtering, and Lightdash integration. - -### Default Tags by Layer - -The framework automatically assigns tags based on model layer: - -| Layer | Auto-Assigned Tag | Auto-Excluded Tags | -| ------ | ----------------- | -------------------------------------------------------- | -| `stg` | `staging` | `intermediate`, `lightdash`, `lightdash-explore`, `mart` | -| `int` | (none) | `lightdash`, `lightdash-explore`, `staging`, `mart` | -| `mart` | `mart` | `staging`, `intermediate` | - -This means: - -- **Mart models automatically get the `mart` tag** — you don't need to add it manually -- Staging models won't appear in Lightdash by default (excluded from `lightdash` tag) -- To make a model appear in Lightdash, add the `lightdash` tag explicitly - -### Tag Types - -Tags can be simple strings or objects with a `type`: - -```jsonc -{ - "tags": [ - "my_tag", // simple string — inherited by downstream models - { "tag": "local_only", "type": "local" }, // NOT inherited downstream - { "tag": "staging", "type": "exclude" }, // removes inherited tag - { "tag": "cost_analysis", "type": "ai_hints" }, // auto-adds 'ai' tag to columns with ai_hint - ], -} -``` - -| Type | Behavior | -| ---------- | ------------------------------------------------------------------ | -| (string) | Inherited by all downstream models | -| `inherit` | Explicitly inherit a tag from upstream models | -| `local` | Applied only to this model, not inherited | -| `exclude` | Removes a tag that would otherwise be inherited | -| `ai_hints` | Auto-adds `ai` tag to metrics/dimensions that have `ai_hint` field | - -### Tag Inheritance - -Tags flow downstream through the model DAG: - -```text -stg model (tags: ["my_project"]) - → int model (inherits "my_project", adds "aggregated") - → mart model (inherits "my_project", "aggregated", auto-adds "mart") -``` - -To prevent a tag from flowing downstream, use `"type": "local"`. - -To remove an inherited tag, use `"type": "exclude"`. - ---- - -## AI Hints - -`ai_hint` provides context to Lightdash's AI assistant. Can be placed at `lightdash.table.schema.json`, `lightdash.dimension.schema.json`, or `lightdash.metric.schema.json` level. Value can be a string or array. - -To auto-tag columns with `ai_hint`, use: `"tags": [{ "tag": "cost_analysis", "type": "ai_hints" }]` - ---- - -## Data Tests - -| Test | Use Case | -| -------------------------- | ----------------------------------------------------------------- | -| `equal_row_count` | Joins where row count should stay the same (1-to-1 relationships) | -| `equal_or_lower_row_count` | Joins with filtering that may reduce rows | -| `no_null_aggregates` | Ensure aggregation columns aren't null | -| `not_null` | Required columns that should never be null | -| `unique` | Primary key or unique identifier columns | - -```jsonc -{ - "data_tests": [ - { "type": "equal_row_count", "column_name": "portal_partition_daily" }, - ], -} -``` - ---- - -## Common Patterns - -### Column Renaming - -```jsonc -{ "name": "customer_id", "expr": "account_id", "type": "dim" } -``` - -### Exclude and Redefine - -Use `"type": "all_from_model", "exclude": ["col1", "col2"]` then redefine those columns with new logic. - -### Where/Having Clauses - -```jsonc -"where": "cost > 0" // simple string -"where": { "and": [{ "expr": "cost > 0" }, { "expr": "status = 1" }] } // AND conditions -"where": { "or": [{ "expr": "region = 'us-east-1'" }] } // OR conditions -"having": { "and": [{ "expr": "sum(cost) > 100" }] } // after aggregation -// subquery condition (see "Inline Subqueries" section for full details) -"where": { "and": [{ "subquery": { "operator": "in", "column": "account_id", "select": ["id"], "from": { "model": "..." } } }] } -``` - -### Join ON Conditions - -```jsonc -"on": "dims" // shorthand: join on all shared dimension columns -"on": { "and": ["account_id", "portal_partition_daily"] } // shorthand (same column names) -"on": { "and": [{ "expr": "base.account_id = joined.customer_id" }] } // explicit expression -``` - -### Self-Joins - -Use `"override_alias": "parent"` on the joined model to reference it by alias. - -### Seed Models - -Reference CSV seeds with `"from": { "model": "seed____" }` in `stg_select_model`. - ---- - -## Inline CTEs - -The following model types support a `ctes` array for inline Common Table Expressions: `int_select_model`, `int_join_models`, `int_union_models`, `mart_select_model`, `mart_join_models`. - -CTEs generate SQL `WITH` clauses within the model. Each CTE has a `name` and a `from` source (model, earlier CTE, or union). Optional: `select`, `where`, `group_by`, `having`. - -CTEs must be ordered — a CTE can only reference CTEs defined before it in the array. The parent model references a CTE via `"from": { "cte": "" }`. - -```jsonc -{ - "type": "int_select_model", - "group": "my_group", - "topic": "my_topic", - "name": "filtered_summary", - "ctes": [ - { - "name": "active_accounts", - "from": { "model": "stg__my_group__my_topic__accounts" }, - "select": ["account_id", "region"], - "where": { "and": [{ "expr": "status = 'active'" }] }, - }, - { - "name": "enriched", // can reference earlier CTE - "from": { - "cte": "active_accounts", - "join": [ - { - "model": "int__my_group__my_topic__daily", - "type": "inner", - "on": { "and": ["account_id"] }, - }, - ], - }, - "select": [ - { "cte": "active_accounts", "type": "all_from_cte" }, - { - "model": "int__my_group__my_topic__daily", - "type": "fcts_from_model", - }, - ], - }, - ], - "from": { "cte": "enriched" }, // parent model reads from a CTE - "select": ["account_id", "region", "cost_sum"], -} -``` - -### CTE Bulk Select with Exclude/Include - -CTE bulk selects (`all_from_cte`, `dims_from_cte`, `fcts_from_cte`) support `exclude` and `include` filters: - -```jsonc -{ - "cte": "active_accounts", - "type": "dims_from_cte", - "exclude": ["internal_id"], // remove specific columns -} -``` - -```jsonc -{ - "cte": "active_accounts", - "type": "all_from_cte", - "include": ["account_id", "region"], // select only these columns -} -``` - -### CTE Column Type Inheritance - -When a CTE selects columns as plain strings (e.g., `"select": ["col_a", "col_b"]`), each column inherits its `dim`/`fct` type from the upstream model or CTE. This means `dims_from_cte` and `fcts_from_cte` will correctly filter by column type in CTE-to-CTE chains without needing to redeclare column types. - -### CTE `group_by` - -Use `"group_by": "dims"` or `"group_by": [{ "type": "dims" }]` inside CTEs. Avoid bare string aliases for computed columns — if a CTE select item has an `expr` (e.g., `{ "name": "month", "expr": "DATE_TRUNC('MONTH', event_date)" }`), using `"group_by": ["month"]` will fail at Trino runtime because the string alias is not a valid SQL GROUP BY target. Use `[{ "type": "dims" }]` instead, which automatically resolves computed expressions. - -### CTE authoring rules - -- **Lightdash metrics belong on the main-model `select`.** `lightdash.metrics` / `lightdash.metrics_merge` on a CTE `select` item is rejected (only the main-model select feeds Lightdash metric generation). Keep the pre-aggregated column in the CTE and re-declare it on the main-model `select` with the metric block. `lightdash.dimension` on CTE selects is still supported. -- **`portal_source_count` auto-injects in CTEs whose `from` is `{ model }` or `{ cte }`.** It's aggregated with `count` when the CTE has a `group_by`; otherwise it passes through. Don't add it manually. Set `override_suffix_agg: true` on the CTE select item only when you need a differently-aggregated variant alongside the audit column. -- **`datetime` and `portal_partition_*` auto-inject in CTEs whose `from` is `{ model }` or `{ cte }`.** Mirrors the main-model behavior: if the upstream (manifest schema for `{ model }`, the in-memory registry for `{ cte }`) has them and the CTE's select did not include them (even through a narrow `dims_from_model.include` list), they're appended automatically. `datetime` emits as a bare passthrough unless the CTE sets `{ "name": "datetime", "interval": "..." }`; in that case the interval drives partition exclusion (`day` drops hourly, `month` drops hourly+daily, `year` drops all three). Auto-inject is still skipped for source and union shapes. Opt out via `"exclude_portal_partition_columns": true` (drop all) or an array such as `["portal_partition_hourly"]` (drop only those) on the CTE or the model (see flag inheritance below). -- **CTE-level exclude/include flags mirror the main-model flags and inherit from the model.** A CTE accepts `exclude_date_filter`, `exclude_daily_filter`, `exclude_datetime`, `exclude_framework_artifacts`, `exclude_portal_partition_columns`, `exclude_portal_source_count`, and `include_full_month` with the same semantics as the corresponding main-model flags. Resolution is uniform: CTE override > model value > false. Set the flag on the model to apply it to every CTE, on a single CTE to override only that CTE, or set `false` on a CTE to opt back in when the model excluded. `exclude_portal_partition_columns` additionally accepts an array (e.g. `["portal_partition_hourly"]`) to drop only the named partition columns; a CTE-level array overrides an inherited `true` or `exclude_framework_artifacts`. `exclude_datetime` and `exclude_portal_partition_columns` are orthogonal — set both for pure-dimension/lookup shapes. `exclude_datetime` is mutually exclusive with `from.rollup` at the same scope (model OR CTE) and the validator errors when both are set together. -- **CTEs may declare `from.rollup` to re-aggregate their source to a coarser grain.** Supported on `from: { model }` and `from: { cte }` (not on `from: { source }` or `from: { union }`, both schema-rejected). The framework rewrites the CTE's `datetime` to `date_trunc(, datetime)`, drops finer-grain `portal_partition_*` columns, wraps fct columns with their suffix-agg (so `revenue_sum` becomes `sum(revenue_sum) as revenue_sum`), and synthesizes a `GROUP BY` from all dim columns when `group_by` is not authored. Chained rollups (CTE A → month, CTE B → year off A) work end-to-end. A rolled-up CTE that sources from another CTE which excludes datetime is rejected with a clear error. -- **Framework columns flow through CTE chains by default.** Once a CTE pulls `datetime` / `portal_partition_*` / `portal_source_count` from its upstream, every downstream `from: { cte }` hop (and a main model with `from: { cte }`) inherits them from the registry. List them in `select` only when you want a transformed alias; opt out with the standard exclude flags on the CTE or the model. When the main model materializes via `incremental` with a partition-overwrite strategy, the auto-flowed `portal_partition_*` typically satisfies the partition-column requirement; if you intentionally exclude them through a chain, set `materialization.partitions: ["datetime"]` on the main model (the partition-strategy warning fires when neither is present). Wrapper SELECTs that reference an already-rolled-up `datetime` do not redundantly re-emit `date_trunc(, datetime)`. -- **`exclude_framework_artifacts` is the combined-flag shortcut.** A single string-enum (`"all"` | `"columns"`) on the model or CTE that bundles multiple individual excludes. `"columns"` implies `exclude_datetime` + `exclude_portal_partition_columns` + `exclude_portal_source_count` (auto WHERE date filters still fire); `"all"` additionally implies `exclude_date_filter`. Individual flags at the same scope override per-column (e.g. `"exclude_framework_artifacts": "all"` paired with `"exclude_portal_source_count": false` keeps that one column). Full resolution chain: CTE individual > CTE combined > model individual > model combined > false. Mutually exclusive with `from.rollup` when the resolved value implies excluding `datetime`. -- **Every `fct` column in the main-model `select` must be aggregated when the main model has a `group_by`.** Set `agg` / `aggs`, wrap an aggregate in `expr` (e.g. `sum(x)`, `avg(x)`, `any_value(x)`, `merge(cast(x as hyperloglog))`, `cast(tdigest_agg(x) as varbinary)`), or set `exclude_from_group_by: true`. This is enforced for scalar selects, CTE scalar refs, and bulk `all_from_cte` / `fcts_from_cte` carriers. -- **Avoid dead outer layers.** A main `select` that's a single `all_from_cte` / `dims_from_cte` passthrough of one CTE with identical `group_by` and no extra filter / limit / projection is flagged as a no-op warning — drop the wrapper (move the CTE's select into the main model) or add new work to the outer layer. - -See `docs/models/CTE_PATTERNS.md` for the full CTE authoring guide. - -### CTE Unions - -CTE unions use the same pattern as model unions: - -```jsonc -{ - "name": "combined", - "from": { "cte": "cte_a", "union": { "ctes": ["cte_b", "cte_c"] } }, -} -``` - ---- - -## Inline Subqueries - -Subqueries can appear in `where`, `having`, and join `on` conditions via the `subquery` key. - -**Structure**: `operator`, `column` (required except for `exists`/`not_exists`), `select`, `from` (model, source, or CTE), optional inner `where`. - -**Operators**: `in`, `not_in`, `exists`, `not_exists`, `eq`, `neq`, `gt`, `gte`, `lt`, `lte` - -```jsonc -// WHERE with subquery -"where": { - "and": [ - { "expr": "cost > 0" }, - { - "subquery": { - "operator": "in", - "column": "account_id", - "select": ["account_id"], - "from": { "model": "int__my_group__my_topic__active_accounts" }, - "where": { "and": [{ "expr": "status = 'active'" }] }, - }, - }, - ], -} - -// JOIN ON with subquery -"on": { - "and": [ - "account_id", - { - "subquery": { - "operator": "exists", - "select": ["1"], - "from": { "cte": "valid_records" }, - "where": { "and": [{ "expr": "a.id = valid_records.id" }] }, - }, - }, - ], -} -``` - ---- - -## Common Pitfalls - -| Problem | Solution | -| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Column Not Found** | Verify columns exist in upstream `select`. Check `exclude` filters in `all_from_model` or `all_from_cte`. | -| **Row Multiplication** | Add `equal_row_count` test. Verify join conditions. Aggregate "many" side before joining. | -| **Duplicate Column Names** | Use `exclude` on one model or rename with `expr`. | -| **Aggregation Without Group By** | Always add `"group_by": "dims"` (or `[{ "type": "dims" }]`) when using `agg`. | -| **Un-aggregated `fct` + group_by** | Every `fct` in the main `select` must set `agg`/`aggs`, wrap an aggregate in `expr` (e.g. `sum(x)`, `merge(cast(x as hyperloglog))`), or `exclude_from_group_by: true`. Applies to scalar and bulk CTE carriers. | -| **Lightdash metrics on a CTE select** | Not supported; only the main-model `select` feeds Lightdash metric generation. Keep the pre-aggregated column in the CTE and declare `lightdash.metrics` / `lightdash.metrics_merge` on the main-model `select`. `lightdash.dimension` on CTE selects is still supported. | -| **Duplicated `portal_source_count` in CTE** | When a CTE's `from` is `{ model }` or `{ cte }`, `portal_source_count` auto-injects (aggregated with `count` when the CTE has `group_by`). Don't add it manually; set `override_suffix_agg: true` only for a differently-aggregated variant alongside the audit column. | -| **Missing `portal_partition_*` / `datetime` in CTE** | When a CTE's `from` is `{ model }` or `{ cte }`, `datetime` and `portal_partition_*` auto-inject from the upstream even if a narrow `dims_from_model.include` list omitted them — do not add them by hand. Explicit `{ "name": "datetime", "interval": X }` drives partition exclusion (`day` drops hourly, `month` drops hourly+daily, `year` drops all three). Opt out per CTE with `"exclude_portal_partition_columns": true` (mirrors the main-model flag); `datetime` itself has no opt-out. | -| **Invalid Source Reference** | Use format `__.` (double underscore, then dot). | -| **Lightdash Case Sensitivity** | Optionally set `"case_sensitive": true/false` at model or column level to override the Lightdash global default. | -| **CTE group_by with computed cols** | Don't use bare string aliases (e.g., `["month"]`) for columns defined with `expr`. Use `"dims"` or `{ "expr": "..." }`. | -| **materialized vs materialization** | Both work; `materialization` is preferred and supports structured config (format, partitions, strategy). | -| **CTE column type mismatch** | Plain string selects in CTEs inherit dim/fct type from upstream. Verify with `dims_from_cte`/`fcts_from_cte`. | -| **Framework-reserved `meta` keys** | Column `type`/`dimension`/`metrics`/`case_sensitive` and model `metrics`/`local_tags`/`case_sensitive` are framework-owned — see "Custom Meta". | -| **Dead outer-layer warning** | Main `select` is a single `all_from_cte` / `dims_from_cte` passthrough of one CTE with identical `group_by` and no extra filter / limit / projection — drop the wrapper or add work to it. See `docs/models/CTE_PATTERNS.md`. | - ---- - -## Scheduling & ETL - -The DJ (Data JSON) Framework uses an ETL scheduler (via Airflow) that determines **which event dates** need to be processed. This is driven by source configurations: - -1. **Sources with `event_count` ETL type**: The scheduler queries source tables to detect which dates have new or changed rows, then runs only those dates through the downstream model DAG. -2. **Sources with `run_schedule` ETL type**: The scheduler triggers downstream models on a fixed cron schedule. -3. **Models inherit their schedule** from their upstream sources — you don't configure scheduling on individual models. The framework traces the DAG back to the source to determine when to run. - -### How the Schedule Flows - -```text -Source (etl config) → stg model → int model(s) → mart model - ↑ schedule ↓ inherits schedule from source -``` - -When creating a new model: - -- If it reads from an **existing source**, the schedule is already handled. -- If it reads from a **new source**, you need to create a `.source.json` with the `meta.etl` configuration. -- The `backfill_start` date determines from when historical data will be processed. - ---- - -## File Creation Checklist - -When adding a new model to the project: - -1. **Identify the layer**: staging (`stg_*`), intermediate (`int_*`), or mart (`mart_*`) -2. **Choose the model type** based on the data transformation needed -3. **Create only the `.model.json` file** — SQL and YML are auto-generated -4. **Place the file** in the correct directory: `models////` -5. **Name the file**: `______.model.json` -6. If reading from a new external table, **create a `.source.json` file** first -7. **Do NOT** create or edit `.sql` or `.yml` files — they are auto-generated by the DJ (Data JSON) Framework - -When adding a new source: - -1. **Create the `.source.json` file** at `models/sources//__.source.json` -2. Define all tables and their columns with Trino-compatible data types -3. Configure `meta.etl` if the source should be scheduled -4. Configure `meta.partition_date` or `meta.partitions` for partition pruning -5. **Do NOT** create or edit the corresponding `.yml` file — it is auto-generated - ---- - -## Validation Rules - -- `group`: lowercase alphanumeric with underscores, no leading/trailing underscores, no consecutive underscores (`^(?!.*__.*)(?!_)(?!.*_$)([a-z]|[0-9]|_)+$`) -- `topic`: same pattern as group -- `name`: same pattern as group -- `tags`: alphanumeric with underscores and hyphens -- `database`: lowercase alphanumeric with underscores -- `schema`: lowercase alphanumeric with underscores -- Column `name`: lowercase alphanumeric with underscores and dots -- Source references: format `__.` -- Model references: the full model name (e.g., `int__my_group__my_topic__daily_summary`) -- `materialized`: must be `"incremental"` or `"ephemeral"` (legacy; prefer `materialization`) -- `materialization`: string `"incremental"` or `"ephemeral"`, or object with `"type": "incremental"` and optional `format`, `partitions`, `strategy`, `database` -- `format` (in `materialization`): must be `"delta_lake"`, `"hive"`, or `"iceberg"` -- `agg`: must be one of `"sum"`, `"count"`, `"min"`, `"max"`, `"hll"`, `"tdigest"` -- `interval`: must be one of `"day"`, `"hour"`, `"month"`, `"year"` -- Join `type`: must be one of `"left"`, `"inner"`, `"right"`, `"full"`, `"cross"` - ---- - -## Schema Reference (Dynamic Lookup) - -The authoritative JSON Schemas for all model and source types live in the `.dj/schemas/` directory at the workspace root. **When you need exact field definitions, allowed values, or validation rules beyond what is documented above, read the relevant schema file.** - -### Key schema files - -| Schema File | Purpose | -| ------------------------------------------- | --------------------------------------------------------------- | -| `model.schema.json` | Top-level model validator (dispatches to type-specific schemas) | -| `model.type.stg_select_source.schema.json` | Schema for `stg_select_source` models | -| `model.type.stg_select_model.schema.json` | Schema for `stg_select_model` models | -| `model.type.stg_union_sources.schema.json` | Schema for `stg_union_sources` models | -| `model.type.int_select_model.schema.json` | Schema for `int_select_model` models | -| `model.type.int_join_models.schema.json` | Schema for `int_join_models` models | -| `model.type.int_join_column.schema.json` | Schema for `int_join_column` models | -| `model.type.int_union_models.schema.json` | Schema for `int_union_models` models | -| `model.type.int_rollup_model.schema.json` | Schema for `int_rollup_model` models | -| `model.type.int_lookback_model.schema.json` | Schema for `int_lookback_model` models | -| `model.type.mart_select_model.schema.json` | Schema for `mart_select_model` models | -| `model.type.mart_join_models.schema.json` | Schema for `mart_join_models` models | -| `source.schema.json` | Top-level source validator | -| `source.table.schema.json` | Source table structure | -| `source.etl.schema.json` | ETL scheduling configuration | -| `source.partition.schema.json` | Partition filter configuration | -| `source.partition_date.schema.json` | Partition date configuration | -| `model.select.col.schema.json` | Column selection options | -| `model.select.expr.schema.json` | Expression-based column selection | -| `model.select.model.schema.json` | Select columns from another model | -| `model.select.source.schema.json` | Select columns from a source | -| `model.from.join.models.schema.json` | Join configuration | -| `model.from.rollup.schema.json` | Rollup configuration for time-grain re-aggregation | -| `model.sql_hooks.schema.json` | `pre` / `post` SQL for staging and intermediate models | -| `model.materialization.schema.json` | Materialization config (string shorthand or structured object) | -| `model.incremental_strategy.schema.json` | Incremental strategy (`delete+insert` or `merge`) | -| `model.format.schema.json` | Storage format (`delta_lake`, `hive`, `iceberg`) | -| `model.partitions.schema.json` | Partition columns for materialization | -| `model.group_by.schema.json` | Group by config (`"dims"` shorthand or array) | -| `model.subquery.schema.json` | Inline subquery definition (WHERE, HAVING, JOIN ON) | -| `model.cte.schema.json` | Single CTE definition | -| `model.ctes.schema.json` | CTE array configuration | -| `model.select.cte.schema.json` | Select columns from a CTE | -| `column.lightdash.schema.json` | Lightdash BI column configuration | -| `model.lightdash.schema.json` | Lightdash BI model-level configuration | -| `model.meta.schema.json` | Free-form model-level meta (reserved-key notes inside) | -| `column.meta.schema.json` | Free-form column-level meta (reserved-key notes inside) | -| `source.meta.schema.json` | Free-form source-level meta | -| `source.table.meta.schema.json` | Free-form source-table-level meta | - -When creating or editing a model, read the type-specific schema (e.g., `model.type.int_join_models.schema.json`) to confirm all required/optional fields and their exact constraints. Schemas use `$ref` to reference sub-schemas — follow those references as needed. - -You can also look at existing `.model.json` and `.source.json` files in the `models/` directory for real examples from this project. - ---- - -## Important Conventions - -1. **Only create `.model.json` and `.source.json` files.** The `.sql` and `.yml` files are auto-generated. -2. **Use JSONC format** — comments are allowed and should be preserved. -3. **Follow the naming convention** strictly: `______` -4. **Reference models by their full name** (e.g., `int__my_group__my_topic__daily_summary`). -5. **Reference sources** as `__.`. -6. **Materialization defaults** to view-like behavior. Prefer `"materialization": "incremental"` (or the structured object form) over legacy `"materialized"`. Both `materialized` and `materialization` are accepted; when both are present, `materialization` takes precedence. Optional `sql_hooks` (`pre`/`post`) exist on staging and intermediate types only — not on marts. -7. **Scheduling is inherited** from sources — don't try to configure schedules on models. -8. **Tags** can be simple strings or objects with `{ "tag": "name", "type": "exclude" | "inherit" | "local" | "ai_hints" }`. -9. **`group_by`** accepts column name strings, `{ "expr": "..." }` objects, or the shorthand `"dims"` (or `[{ "type": "dims" }]`) to group by all dimension columns automatically. The string `"dims"` is equivalent to `[{ "type": "dims" }]`. Similarly, join `on` accepts `"dims"` to auto-join on all shared dimension columns. -10. **`expr`** on select columns lets you write arbitrary Trino SQL. For declarative aggregation, prefer `agg`/`aggs` on `fct` columns over manual `expr`. Note: `mart_select_model` and `int_union_models` do not support `agg`/`aggs` — use `expr` or pre-aggregate upstream. When using `agg`, always set `group_by`. Never duplicate the same aggregate in both `expr` and `agg`. -11. **Verify upstream columns before selecting them.** When creating or editing a downstream model, always open and read the upstream model's `.sql` file (there will be multiple in joins) or source `.yml` for staging models to confirm which columns are actually available. Determine the effective column name by inspecting each entry in the upstream `select` directive. - - - **`"expr"` key exists** → the column is a computed expression, ensure that any column names referenced existing in one of the parent models. - - **`"name"` key exists (no `"expr"`)** → use the `"name"` value as the column name. - - **Plain string** (e.g., `"account_id"`) → use that string directly as the column name. - - **`"all_from_model"` / `"dims_from_model"` / `"fcts_from_model"`** → the upstream pulls columns from _its own_ upstream model; follow the chain to that model's `select` to discover the actual column names. If `"include"` or `"exclude"` is present, apply those filters. - - **`"all_from_source"` / `"dims_from_source"` / `"fcts_from_source"`** → the upstream pulls columns from a source; open the referenced source `.yml` and inspect the table's `columns` array for the available column names and data types. - - **`"all_from_cte"` / `"dims_from_cte"` / `"fcts_from_cte"`** → the upstream pulls columns from a CTE; trace to that CTE's `select` to discover available columns. - - **Never assume a column exists — always verify it in the upstream definition.** This prevents referencing columns that don't exist. - -12. **To rename or move a model, update its `.model.json` fields — not the filename.** Change the `group`, `topic`, and/or `name` fields inside the JSON file. The DJ (Data JSON) Framework will automatically rename/move the file and regenerate the corresponding `.sql` and `.yml` files to match. Do not manually rename or move model files on disk. -13. **CTE `group_by` must not use bare string aliases for computed columns.** If a CTE select item has `"name": "month", "expr": "DATE_TRUNC('MONTH', event_date)"`, using `"group_by": ["month"]` will pass schema validation but fail at Trino with `COLUMN_NOT_FOUND`. Use `"group_by": "dims"` or `"group_by": [{ "expr": "DATE_TRUNC('MONTH', event_date)" }]` instead. -14. **CTE bulk selects support `exclude`/`include` filters.** `all_from_cte`, `dims_from_cte`, and `fcts_from_cte` accept `exclude` and `include` arrays, matching the behavior of model-level bulk selects. Plain string column selects in CTEs inherit their `dim`/`fct` type from the upstream model or CTE. -15. **Source freshness can be disabled.** Set `"freshness": null` at source level or table level to disable dbt freshness checks. Individual tables can override the source-level `loaded_at_field`. diff --git a/templates/_agents-dj/_AGENTS.md b/templates/_agents-dj/_AGENTS.md new file mode 100644 index 0000000..8981686 --- /dev/null +++ b/templates/_agents-dj/_AGENTS.md @@ -0,0 +1,140 @@ +# AGENTS.md — DJ (Data JSON) Framework Guide + +> Auto-generated by the Workday DJ (Data JSON) Framework VS Code extension and regenerated on sync — do not edit this file. Use it to author `.model.json` and `.source.json` files in this dbt project. + +## Overview + +This project uses the **DJ (Data JSON) Framework** (**DJ**) — a JSON-based abstraction layer on top of dbt. Author `.model.json` and `.source.json` files instead of writing raw SQL and YML by hand; DJ then **auto-generates** the corresponding `.sql` and `.yml` files via a process called "JSON Sync." Never edit the generated `.sql` or `.yml` files — edit only the `.model.json` and `.source.json` files. + +All JSON files use the **JSONC** format (JSON with Comments). Trailing commas are allowed. Preserve any existing comments when editing files. + +--- + +## How this guide is organized + +Follow the **always-relevant** rules below — project structure, naming, structural governance, project/environment resolution, command safety, and the core conventions. Load a `reference/` file on demand only when your task matches it: + +| To do this | Load | +| ----------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | +| Shape any model — the 11 types with examples, the Advanced CTE/rollup/shorthand/subquery map, select-column types, common optional fields | [reference/model-types.md](reference/model-types.md) | +| Define a source (`.source.json`) — ETL config, source partitions, scheduling | [reference/sources.md](reference/sources.md) | +| Set materialization, incremental strategy, storage format, or framework/partition columns | [reference/materialization.md](reference/materialization.md) | +| Write inline CTEs, CTE unions, subqueries, or common SQL patterns | [reference/ctes-and-subqueries.md](reference/ctes-and-subqueries.md) | +| Configure Lightdash (model/column), dashboards-as-code, AI hints, tags, or data tests | [reference/lightdash-tags-tests.md](reference/lightdash-tags-tests.md) | +| Add free-form or governance `meta`, or understand framework-reserved `meta` keys | [reference/meta-and-governance.md](reference/meta-and-governance.md) | +| Troubleshoot, validate, follow the file-creation checklist, or find the exact schema file | [reference/pitfalls-and-validation.md](reference/pitfalls-and-validation.md) | +| Run a dbt command (compile/parse/test, or a warehouse-writing run/build/seed) — venv activation, project dir, command classes | [reference/running-dbt.md](reference/running-dbt.md) | +| Run a read-only Trino query to inspect data or schema — CLI resolution, connection env, invocation | [reference/running-trino.md](reference/running-trino.md) | +| Run the Lightdash CLI (start-preview / download / upload) — connection env, settings, guardrails | [reference/running-lightdash.md](reference/running-lightdash.md) | +| Commit, branch, or stage DJ work in git — what to commit, what to ignore, commit hygiene | [reference/git-workflow.md](reference/git-workflow.md) | + +--- + +## Project Structure + +The dbt project root (where `dbt_project.yml` lives) contains the following structure: + +```text +/ +├── models/ +│ ├── staging/ # stg__ models (stg_select_source, stg_select_model, stg_union_sources) +│ │ └── / +│ │ └── / +│ │ ├── .model.json +│ │ ├── .sql (auto-generated, do NOT edit) +│ │ └── .yml (auto-generated, do NOT edit) +│ ├── intermediate/ # int__ models (int_select_model, int_join_models, int_union_models, etc.) +│ │ └── / +│ │ └── / +│ ├── marts/ # mart__ models (mart_select_model, mart_join_models) +│ │ └── / +│ │ └── / +│ └── sources/ # source definitions +│ └── / +│ ├── __.source.json +│ └── __.yml (auto-generated, do NOT edit) +├── seeds/ +│ └── / +│ └── seed____.csv +├── macros/ +└── dbt_project.yml +``` + +--- + +## Model Naming Convention + +Model names follow the pattern: `______` + +- **layer**: Derived from the model `type` field (`stg`, `int`, or `mart`) +- **group**: Team or project classification (e.g., `finance`, `analytics`, `sales`) +- **topic**: Subject area within the group (e.g., `billing`, `orders`, `customers`) +- **name**: Descriptive name for the model (e.g., `daily_summary`, `account_hierarchy`) + +--- + +## Structural Governance (framework-enforced) + +Model placement and naming are **derived and enforced by the framework** — they are not free choices. Do not create model files in arbitrary folders or under arbitrary names. + +- **Path is derived, never chosen.** The framework computes each model's location from `type` → layer folder (`staging` / `intermediate` / `marts`) + `group` + `topic` + `name`, and writes/relocates the file there. Renaming or moving a model is done by editing the `type` / `group` / `topic` / `name` fields in the `.model.json` — never by moving the file on disk. +- **`group` must be a registered group.** dbt registers groups in any `.yml` file under a top-level `groups:` key — commonly a central `models/_groups.yml` or `models/groups.yml`, or per-folder files like `models//group_*.yml` — and models are assigned via the `group` config (model-level or `dbt_project.yml` `+group:`). There is no single fixed path: scan the project's `.yml` files for `groups:` definitions (and existing sibling models' `group` values) to find the valid set. If the group a user asks for is not registered, **ask them to pick a registered group or to register a new one** (via the `dj-initialize` skill) — do not invent a group or create an unregistered folder. +- **`topic` and `name`** follow the lowercase/underscore pattern (no leading/trailing `_`, no `__`). Mirror the conventions of existing sibling models. + +## Project & Environment Resolution + +When the workspace holds more than one dbt project, or a target environment/catalog/database is ambiguous, **ask the user which one to use — do not silently pick a default.** + +- **dbt project.** The `.model.json` / `.source.json` paths are relative to the dbt project root (`dbt_project.yml`), which may be nested and differ from the DJ/workspace root (where `.dj/schemas/` lives). If multiple dbt projects exist, confirm the target — prefer the one named in `dj.dbtProjectNames` only after confirming it is the intended one. +- **Environment / warehouse target.** For any operation that reads from or writes to a warehouse (Trino catalog/schema, Python model output database, Lightdash project), lay out the choices and let the user pick. Never assume prod. For Lightdash uploads specifically, projects listed in `dj.lightdash.restrictedProjects` are guarded (`block` refuses the upload; `warn` requires confirmation) — treat those as production and confirm before uploading. + +## Command & Query Execution Safety + +Treat the warehouse as **read-only by default**. Data is written by the framework and dbt through generated SQL — not by ad-hoc commands. + +- **Read-only queries only, unless the user confirms otherwise.** `SELECT` / `SHOW` / `DESCRIBE` / `EXPLAIN` are safe to run for inspection. Any **DDL** (`CREATE`, `DROP`, `ALTER`, `TRUNCATE`, `RENAME`, `GRANT`, `REVOKE`) or **DML** (`INSERT`, `UPDATE`, `DELETE`, `MERGE`), and any warehouse-writing dbt command (`dbt run`, `build`, `seed`, `snapshot`, `run-operation`), require **explicit per-command user confirmation** before you run them. +- **Never write to production.** Do not run a DDL/DML statement or a warehouse-writing dbt command against a production target, even with confirmation. If you cannot confirm a target is non-production, stop and ask. +- **Confirm the connection target before any query.** The user may not have permission on every catalog/schema/table. Before running even a `SELECT`, confirm the catalog/schema/profile/environment to use — ask once; do not assume a default. +- **Prefer framework facilities over ad-hoc CLI.** To inspect columns, sources, or lineage, read `.source.json` / `.model.json` / `target/manifest.json` / `.dj/schemas/`, or use DJ's **Create Source** flow (which browses Trino catalogs / schemas / tables / columns), before shelling out to the `trino` CLI. Run CLI SQL only after the user confirms the connection and that it is read-only. +- **The DJ commands are not dbt runs.** `DJ: Sync to SQL and YML` regenerates the `.sql` / `.yml` and reparses the manifest on demand — it runs `dbt parse` only when a synced model is missing or the manifest is stale. `DJ: Refresh Projects` re-reads each `dbt_project.yml` and reloads the on-disk `target/manifest.json` (and rewrites extension-managed files); it does **not** run `dbt parse` / `compile`. Of the dbt CLI commands, `parse` / `compile` / `ls` / `deps` / `docs generate` don't write to the warehouse; `run` / `build` / `seed` / `snapshot` / `run-operation` do, and `source freshness` issues queries. If a manifest must be built from scratch, ask the user to run `dbt parse`. +- **Keep inspection queries cheap.** Always add a `LIMIT` to ad-hoc `SELECT`s and constrain by partition — never trigger a full-history or unpartitioned scan just to check a shape or a few values. + +For the mechanics of running each tool under this policy, load [reference/running-dbt.md](reference/running-dbt.md) (venv activation, project dir, command classes), [reference/running-trino.md](reference/running-trino.md) (CLI resolution, connection env, invocation), [reference/running-lightdash.md](reference/running-lightdash.md) (preview / download / upload), or [reference/git-workflow.md](reference/git-workflow.md) (what to commit and ignore). + +--- + +## Important Conventions + +1. **Only create `.model.json` and `.source.json` files.** The `.sql` and `.yml` files are auto-generated. +2. **Use JSONC format** — comments are allowed and should be preserved. +3. **Follow the naming convention** strictly: `______` +4. **Reference models by their full name** (e.g., `int__my_group__my_topic__daily_summary`). +5. **Reference sources** as `__.`. +6. **Materialization defaults** to view-like behavior. Prefer `"materialization": "incremental"` (or the structured object form) over legacy `"materialized"`. Both `materialized` and `materialization` are accepted; when both are present, `materialization` takes precedence. Optional `sql_hooks` (`pre`/`post`) exist on staging and intermediate types only — not on marts. +7. **Scheduling is inherited** from sources — don't try to configure schedules on models. +8. **Tags** can be simple strings or objects with `{ "tag": "name", "type": "exclude" | "inherit" | "local" | "ai_hints" }`. +9. **`group_by`** accepts column name strings, `{ "expr": "..." }` objects, or the shorthand `"dims"` (or `[{ "type": "dims" }]`) to group by all dimension columns automatically. The string `"dims"` is equivalent to `[{ "type": "dims" }]`. Similarly, join `on` accepts `"dims"` to auto-join on all shared dimension columns. +10. **`expr`** on select columns lets you write arbitrary Trino SQL. For declarative aggregation, prefer `agg`/`aggs` on `fct` columns over manual `expr`. Note: `mart_select_model` and `int_union_models` do not support `agg`/`aggs` — use `expr` or pre-aggregate upstream. When using `agg`, always set `group_by`. Never duplicate the same aggregate in both `expr` and `agg`. +11. **Verify upstream columns before selecting them.** When creating or editing a downstream model, always open and read the upstream model's `.sql` file (there will be multiple in joins) or source `.yml` for staging models to confirm which columns are actually available. Determine the effective column name by inspecting each entry in the upstream `select` directive. + + - **`"expr"` key exists** → the column is a computed expression, ensure that any column names referenced existing in one of the parent models. + - **`"name"` key exists (no `"expr"`)** → use the `"name"` value as the column name. + - **Plain string** (e.g., `"account_id"`) → use that string directly as the column name. + - **`"all_from_model"` / `"dims_from_model"` / `"fcts_from_model"`** → the upstream pulls columns from _its own_ upstream model; follow the chain to that model's `select` to discover the actual column names. If `"include"` or `"exclude"` is present, apply those filters. + - **`"all_from_source"` / `"dims_from_source"` / `"fcts_from_source"`** → the upstream pulls columns from a source; open the referenced source `.yml` and inspect the table's `columns` array for the available column names and data types. + - **`"all_from_cte"` / `"dims_from_cte"` / `"fcts_from_cte"`** → the upstream pulls columns from a CTE; trace to that CTE's `select` to discover available columns. + + **Never assume a column exists — always verify it in the upstream definition.** + +12. **To rename or move a model, update its `.model.json` fields — not the filename.** Change the `group`, `topic`, and/or `name` fields inside the JSON file. DJ automatically renames/moves the file and regenerates the corresponding `.sql` and `.yml` files to match. Do not manually rename or move model files on disk. +13. **CTE `group_by` must not use bare string aliases for computed columns.** If a CTE select item has `"name": "month", "expr": "DATE_TRUNC('MONTH', event_date)"`, using `"group_by": ["month"]` will pass schema validation but fail at Trino with `COLUMN_NOT_FOUND`. Use `"group_by": "dims"` or `"group_by": [{ "expr": "DATE_TRUNC('MONTH', event_date)" }]` instead. +14. **CTE bulk selects support `exclude`/`include` filters.** `all_from_cte`, `dims_from_cte`, and `fcts_from_cte` accept `exclude` and `include` arrays, matching the behavior of model-level bulk selects. Plain string column selects in CTEs inherit their `dim`/`fct` type from the upstream model or CTE. +15. **Source freshness can be disabled.** Set `"freshness": null` at source level or table level to disable dbt freshness checks. Individual tables can override the source-level `loaded_at_field`. + +--- + +## Schema Reference + +The authoritative JSON Schemas for all model and source types live in the `.dj/schemas/` directory at the workspace root. **When you need exact field definitions, allowed values, or validation rules beyond what these guides document, read the relevant schema file** — schemas use `$ref` to reference sub-schemas, so follow those references as needed. A full index of the key schema files is in [reference/pitfalls-and-validation.md](reference/pitfalls-and-validation.md). + +Read existing `.model.json` and `.source.json` files in the `models/` directory for real examples from this project. diff --git a/templates/_agents-dj/reference/ctes-and-subqueries.md b/templates/_agents-dj/reference/ctes-and-subqueries.md new file mode 100644 index 0000000..0625cbf --- /dev/null +++ b/templates/_agents-dj/reference/ctes-and-subqueries.md @@ -0,0 +1,185 @@ +# Inline CTEs, Subqueries & Common SQL Patterns + +Load this when a model needs CTEs (`ctes` array), inline subqueries in `where` / `having` / join `on`, or the recurring SQL patterns (renaming, where/having, join conditions, self-joins, seeds). CTE and subquery support is limited to `int_select_model`, `int_join_models`, `int_union_models`, `mart_select_model`, `mart_join_models`. + +## Common Patterns + +### Column Renaming + +```jsonc +{ "name": "customer_id", "expr": "account_id", "type": "dim" } +``` + +### Exclude and Redefine + +Use `"type": "all_from_model", "exclude": ["col1", "col2"]` then redefine those columns with new logic. + +### Where/Having Clauses + +```jsonc +"where": "cost > 0" // simple string +"where": { "and": [{ "expr": "cost > 0" }, { "expr": "status = 1" }] } // AND conditions +"where": { "or": [{ "expr": "region = 'us-east-1'" }] } // OR conditions +"having": { "and": [{ "expr": "sum(cost) > 100" }] } // after aggregation +// subquery condition (see "Inline Subqueries" section for full details) +"where": { "and": [{ "subquery": { "operator": "in", "column": "account_id", "select": ["id"], "from": { "model": "..." } } }] } +``` + +### Join ON Conditions + +```jsonc +"on": "dims" // shorthand: join on all shared dimension columns +"on": { "and": ["account_id", "portal_partition_daily"] } // shorthand (same column names) +"on": { "and": [{ "expr": "base.account_id = joined.customer_id" }] } // explicit expression +``` + +### Self-Joins + +Use `"override_alias": "parent"` on the joined model to reference it by alias. + +### Seed Models + +Reference CSV seeds with `"from": { "model": "seed____" }` in `stg_select_model`. + +--- + +## Inline CTEs + +The following model types support a `ctes` array for inline Common Table Expressions: `int_select_model`, `int_join_models`, `int_union_models`, `mart_select_model`, `mart_join_models`. + +CTEs generate SQL `WITH` clauses within the model. Each CTE has a `name` and a `from` source (model, earlier CTE, or union). Optional: `select`, `where`, `group_by`, `having`. + +CTEs must be ordered — a CTE can only reference CTEs defined before it in the array. The parent model references a CTE via `"from": { "cte": "" }`. + +```jsonc +{ + "type": "int_select_model", + "group": "my_group", + "topic": "my_topic", + "name": "filtered_summary", + "ctes": [ + { + "name": "active_accounts", + "from": { "model": "stg__my_group__my_topic__accounts" }, + "select": ["account_id", "region"], + "where": { "and": [{ "expr": "status = 'active'" }] }, + }, + { + "name": "enriched", // can reference earlier CTE + "from": { + "cte": "active_accounts", + "join": [ + { + "model": "int__my_group__my_topic__daily", + "type": "inner", + "on": { "and": ["account_id"] }, + }, + ], + }, + "select": [ + { "cte": "active_accounts", "type": "all_from_cte" }, + { + "model": "int__my_group__my_topic__daily", + "type": "fcts_from_model", + }, + ], + }, + ], + "from": { "cte": "enriched" }, // parent model reads from a CTE + "select": ["account_id", "region", "cost_sum"], +} +``` + +### CTE Bulk Select with Exclude/Include + +CTE bulk selects (`all_from_cte`, `dims_from_cte`, `fcts_from_cte`) support `exclude` and `include` filters: + +```jsonc +{ + "cte": "active_accounts", + "type": "dims_from_cte", + "exclude": ["internal_id"], // remove specific columns +} +``` + +```jsonc +{ + "cte": "active_accounts", + "type": "all_from_cte", + "include": ["account_id", "region"], // select only these columns +} +``` + +### CTE Column Type Inheritance + +When a CTE selects columns as plain strings (e.g., `"select": ["col_a", "col_b"]`), each column inherits its `dim`/`fct` type from the upstream model or CTE. This means `dims_from_cte` and `fcts_from_cte` will correctly filter by column type in CTE-to-CTE chains without needing to redeclare column types. + +### CTE `group_by` + +Use `"group_by": "dims"` or `"group_by": [{ "type": "dims" }]` inside CTEs. Avoid bare string aliases for computed columns — if a CTE select item has an `expr` (e.g., `{ "name": "month", "expr": "DATE_TRUNC('MONTH', event_date)" }`), using `"group_by": ["month"]` will fail at Trino runtime because the string alias is not a valid SQL GROUP BY target. Use `[{ "type": "dims" }]` instead, which automatically resolves computed expressions. + +### CTE authoring rules + +- **Lightdash metrics belong on the main-model `select`.** `lightdash.metrics` / `lightdash.metrics_merge` on a CTE `select` item is rejected (only the main-model select feeds Lightdash metric generation). Keep the pre-aggregated column in the CTE and re-declare it on the main-model `select` with the metric block. `lightdash.dimension` on CTE selects is still supported. +- **`portal_source_count` auto-injects in CTEs whose `from` is `{ model }` or `{ cte }`.** It's aggregated with `count` when the CTE has a `group_by`; otherwise it passes through. Don't add it manually. Set `override_suffix_agg: true` on the CTE select item only when you need a differently-aggregated variant alongside the audit column. +- **`datetime` and `portal_partition_*` auto-inject in CTEs whose `from` is `{ model }` or `{ cte }`.** Mirrors the main-model behavior: if the upstream (manifest schema for `{ model }`, the in-memory registry for `{ cte }`) has them and the CTE's select did not include them (even through a narrow `dims_from_model.include` list), they're appended automatically. `datetime` emits as a bare passthrough unless the CTE sets `{ "name": "datetime", "interval": "..." }`; in that case the interval drives partition exclusion (`day` drops hourly, `month` drops hourly+daily, `year` drops all three). Auto-inject is still skipped for source and union shapes. Opt out via `"exclude_portal_partition_columns": true` (drop all) or an array such as `["portal_partition_hourly"]` (drop only those) on the CTE or the model (see flag inheritance below). +- **CTE-level exclude/include flags mirror the main-model flags and inherit from the model.** A CTE accepts `exclude_date_filter`, `exclude_daily_filter`, `exclude_datetime`, `exclude_framework_artifacts`, `exclude_portal_partition_columns`, `exclude_portal_source_count`, and `include_full_month` with the same semantics as the corresponding main-model flags. Resolution is uniform: CTE override > model value > false. Set the flag on the model to apply it to every CTE, on a single CTE to override only that CTE, or set `false` on a CTE to opt back in when the model excluded. `exclude_portal_partition_columns` additionally accepts an array (e.g. `["portal_partition_hourly"]`) to drop only the named partition columns; a CTE-level array overrides an inherited `true` or `exclude_framework_artifacts`. `exclude_datetime` and `exclude_portal_partition_columns` are orthogonal — set both for pure-dimension/lookup shapes. `exclude_datetime` is mutually exclusive with `from.rollup` at the same scope (model OR CTE) and the validator errors when both are set together. +- **CTEs may declare `from.rollup` to re-aggregate their source to a coarser grain.** Supported on `from: { model }` and `from: { cte }` (not on `from: { source }` or `from: { union }`, both schema-rejected). The framework rewrites the CTE's `datetime` to `date_trunc(, datetime)`, drops finer-grain `portal_partition_*` columns, wraps fct columns with their suffix-agg (so `revenue_sum` becomes `sum(revenue_sum) as revenue_sum`), and synthesizes a `GROUP BY` from all dim columns when `group_by` is not authored. Chained rollups (CTE A → month, CTE B → year off A) work end-to-end. A rolled-up CTE that sources from another CTE which excludes datetime is rejected with a clear error. +- **Framework columns flow through CTE chains by default.** Once a CTE pulls `datetime` / `portal_partition_*` / `portal_source_count` from its upstream, every downstream `from: { cte }` hop (and a main model with `from: { cte }`) inherits them from the registry. List them in `select` only when you want a transformed alias; opt out with the standard exclude flags on the CTE or the model. When the main model materializes via `incremental` with a partition-overwrite strategy, the auto-flowed `portal_partition_*` typically satisfies the partition-column requirement; if you intentionally exclude them through a chain, set `materialization.partitions: ["datetime"]` on the main model (the partition-strategy warning fires when neither is present). Wrapper SELECTs that reference an already-rolled-up `datetime` do not redundantly re-emit `date_trunc(, datetime)`. +- **`exclude_framework_artifacts` is the combined-flag shortcut.** A single string-enum (`"all"` | `"columns"`) on the model or CTE that bundles multiple individual excludes. `"columns"` implies `exclude_datetime` + `exclude_portal_partition_columns` + `exclude_portal_source_count` (auto WHERE date filters still fire); `"all"` additionally implies `exclude_date_filter`. Individual flags at the same scope override per-column (e.g. `"exclude_framework_artifacts": "all"` paired with `"exclude_portal_source_count": false` keeps that one column). Full resolution chain: CTE individual > CTE combined > model individual > model combined > false. Mutually exclusive with `from.rollup` when the resolved value implies excluding `datetime`. +- **Every `fct` column in the main-model `select` must be aggregated when the main model has a `group_by`.** Set `agg` / `aggs`, wrap an aggregate in `expr` (e.g. `sum(x)`, `avg(x)`, `any_value(x)`, `merge(cast(x as hyperloglog))`, `cast(tdigest_agg(x) as varbinary)`), or set `exclude_from_group_by: true`. This is enforced for scalar selects, CTE scalar refs, and bulk `all_from_cte` / `fcts_from_cte` carriers. +- **Avoid dead outer layers.** A main `select` that's a single `all_from_cte` / `dims_from_cte` passthrough of one CTE with identical `group_by` and no extra filter / limit / projection is flagged as a no-op warning — drop the wrapper (move the CTE's select into the main model) or add new work to the outer layer. + +### CTE Unions + +CTE unions use the same pattern as model unions: + +```jsonc +{ + "name": "combined", + "from": { "cte": "cte_a", "union": { "ctes": ["cte_b", "cte_c"] } }, +} +``` + +--- + +## Inline Subqueries + +Subqueries can appear in `where`, `having`, and join `on` conditions via the `subquery` key. + +**Structure**: `operator`, `column` (required except for `exists`/`not_exists`), `select`, `from` (model, source, or CTE), optional inner `where`. + +**Operators**: `in`, `not_in`, `exists`, `not_exists`, `eq`, `neq`, `gt`, `gte`, `lt`, `lte` + +```jsonc +// WHERE with subquery +"where": { + "and": [ + { "expr": "cost > 0" }, + { + "subquery": { + "operator": "in", + "column": "account_id", + "select": ["account_id"], + "from": { "model": "int__my_group__my_topic__active_accounts" }, + "where": { "and": [{ "expr": "status = 'active'" }] }, + }, + }, + ], +} + +// JOIN ON with subquery +"on": { + "and": [ + "account_id", + { + "subquery": { + "operator": "exists", + "select": ["1"], + "from": { "cte": "valid_records" }, + "where": { "and": [{ "expr": "a.id = valid_records.id" }] }, + }, + }, + ], +} +``` diff --git a/templates/_agents-dj/reference/git-workflow.md b/templates/_agents-dj/reference/git-workflow.md new file mode 100644 index 0000000..a946024 --- /dev/null +++ b/templates/_agents-dj/reference/git-workflow.md @@ -0,0 +1,36 @@ +# Git workflow + +Load this when committing, staging, or branching DJ work in this dbt project. For merge conflicts between a `.model.json` and its generated `.sql` / `.yml`, use the `dj-resolve-merge-conflicts` skill instead. + +## Commit the JSON source and its generated output together + +A DJ model is two coupled artifacts: the hand-authored source (`.model.json` / `.source.json`) and the framework-generated siblings (`.sql` / `.yml`, plus `.python.py` for Python models). Commit them **together** so the repository stays consistent for dbt runs and CI. + +- **Sync before you commit.** After editing a `.model.json`, ask the user to run **`DJ: Sync to SQL and YML`** so the generated files reflect the source. Never commit a JSON change while its `.sql` / `.yml` are stale. +- **Never hand-edit the generated files** to make a diff look right — change the `.model.json` and re-sync. + +## Do not commit DJ internal state + +- `.dj/` holds DJ's local caches, diagnostics, and state; DJ adds it to `.gitignore`. Never stage it. +- The dashboards-as-code path (`lightdash/` by default) may be intentionally gitignored via marker blocks depending on the project's setup — respect the existing `.gitignore` rather than force-adding those files. + +## Branch, stage, commit + +Standard git: `git checkout -b `, `git add `, `git commit`. Follow the **project's own** commit-message style — scan `git log --oneline` to match it. (The DJ extension repository uses `type(scope): description`, but a downstream dbt project may have its own convention; do not impose DJ's scopes on it.) + +## Find what changed + +To see which models a change touched, mirror what DJ's own tooling does: + +```bash +git --no-pager diff --name-only origin/master.. +git status --porcelain +``` + +Filter for `.model.json` / `.source.json` to find edited sources, and confirm their generated `.sql` / `.yml` are staged alongside them. + +## Guardrails + +- **Ask before pushing.** Do not `git push` without the user's go-ahead, and never `git push --force` to a shared branch. Stop at the commit by default — pushing, opening a PR, or any `gh` operation needs the user to ask for it first. +- **Do not discard work.** Avoid `git reset --hard`, `git checkout -- .`, or deleting untracked files that may be in-progress work — ask first. +- **Don't commit secrets, and follow `.gitignore`.** API tokens and hard-coded warehouse credentials must never be staged; honor the repo's `.gitignore` rather than force-adding an ignored file. A `profiles.yml` is safe to commit only when it reads its secrets from environment variables (e.g. `{{ env_var('...') }}`) instead of embedding them. If you're unsure whether a file contains a secret, stop and confirm with the user before staging it. diff --git a/templates/_agents-dj/reference/lightdash-tags-tests.md b/templates/_agents-dj/reference/lightdash-tags-tests.md new file mode 100644 index 0000000..c1dd882 --- /dev/null +++ b/templates/_agents-dj/reference/lightdash-tags-tests.md @@ -0,0 +1,191 @@ +# Lightdash, Tags & Data Tests + +Load this when configuring Lightdash (model or column BI config), editing dashboards-as-code YAML, adding AI hints, tagging models, or declaring data tests. + +For authoring or editing the exported dashboards-as-code YAML (chart / dashboard files under the dashboards-as-code path — `lightdash/` by default, configurable via `dj.lightdash.dashboardsAsCodePath`), use the `dj-create-lightdash-yaml` and `dj-edit-lightdash-yaml` skills — they cover the YAML shapes, field-ID rules, and upload gotchas this reference does not. + +## Lightdash Configuration + +### Model-level Lightdash + +```jsonc +{ + "lightdash": { + "table": { + "label": "Daily Cost Summary", + "group_label": "Cost Analytics", + "ai_hint": "Daily aggregated costs by account and region", + "sql_filter": "cost_sum > 0", + "required_filters": ["portal_partition_daily"], + }, + "metrics": [ + { + "name": "total_cost", + "type": "sum", + "label": "Total Cost", + "group_label": "Cost Metrics", + "sql": "${TABLE}.cost_sum", + "round": 0, + "format": "usd", + }, + ], + }, +} +``` + +### Column-level Lightdash + +```jsonc +{ + "name": "cost", + "type": "fct", + "lightdash": { + "dimension": { "hidden": true, "label": "Raw Cost", "group_label": "Cost" }, + "metrics": [ + { + "name": "total_cost", + "type": "sum", + "label": "Total Cost", + "format": "usd", + "round": 0, + }, + ], + "metrics_merge": { + "format": "usd", + "round": 0, + "group_label": "Cost Metrics", + }, + }, +} +``` + +## Lightdash Dashboards as Code + +The DJ extension also exposes Lightdash's [Dashboards as Code](https://docs.lightdash.com/guides/developer/dashboards-as-code) workflow via the `DJ: Lightdash — Dashboards as Code` command. This is **separate from** the model-level `lightdash` config above (which generates `meta.dimensions` / `meta.metrics` blocks in the dbt-managed YAML). Dashboards-as-Code lets you author the actual saved charts and dashboards (the things visible in the Lightdash UI) as version-control-friendly YAML files. + +### Layout + +By default, the extension's `lightdash download` writes: + +```text +/lightdash/ +├── charts/ +│ └── .yml +└── dashboards/ + └── .yml +``` + +The base path is configurable via the `dj.lightdash.dashboardsAsCodePath` extension setting. The slug in the filename is also the slug used by the Lightdash CLI's `-c` / `-d` flags. + +### When to edit these files + +These YAML files are **inputs to `lightdash upload`**. Edit them when the user wants to tweak a chart's filters, axis labels, dashboard tiles, etc. without clicking through the Lightdash UI. Typical workflow: + +1. User runs the Download tab (entire project or specific charts/dashboards). +2. You edit `/lightdash/charts/.yml` or `…/dashboards/.yml`. +3. User runs the Upload tab (selection-driven by default — only edited files get pushed). + +### YAML shape + +Both file types are validated by official Lightdash JSON schemas (the extension auto-registers them with the Red Hat YAML extension). Top-level keys: + +- **Chart** (`charts/*.yml`): `version`, `slug`, `name`, `description?`, `chartConfig`, `tableConfig`, `metricQuery` (`exploreName`, `dimensions`, `metrics`, `filters`, `sorts`, `limit`, `tableCalculations`, `additionalMetrics`), `dashboardSlug?`, `spaceSlug?`, `tags?`. +- **Dashboard** (`dashboards/*.yml`): `version`, `slug`, `name`, `description?`, `tabs?`, `tiles[]` (each tile has `type`, `properties`, layout `x/y/w/h`), `filters?`, `spaceSlug?`, `tags?`. + +Read the file's `# yaml-language-server: $schema=…` header (or the auto-installed `yaml.schemas` binding) for the authoritative shape — do not invent fields. + +### Editing rules + +- **Do not change `slug`.** The slug is the primary key the upload uses to match local files to remote charts/dashboards. Renaming the slug creates a new resource on upload. +- **Do not change `version`.** It pins the schema; bumping it by hand will break the upload. +- **Preserve unfamiliar keys.** The schemas evolve; keep any field you do not recognize as-is so downloads stay round-trippable. +- **Reference existing dbt models, not raw tables.** `metricQuery.exploreName` and `metrics`/`dimensions` reference the model's Lightdash `table` / dimension / metric names — confirm they exist by reading the model's `.model.json` `lightdash` block. If a metric the user wants does not exist on the model, add it to the `.model.json` (regenerates the dbt YAML) before referencing it from the chart YAML. +- **Dashboard tiles must reference real chart slugs.** A dashboard tile's `properties.savedChartSlug` (or equivalent) must match a chart slug that exists either locally under `charts/` or already on Lightdash. +- **Use the extension's webview to invoke the CLI.** Do not shell out to `lightdash download` / `lightdash upload` directly — the webview handles auth, working directory, and YAML schema sync. + +--- + +## AI Hints + +`ai_hint` provides context to Lightdash's AI assistant. Can be placed at `lightdash.table.schema.json`, `lightdash.dimension.schema.json`, or `lightdash.metric.schema.json` level. Value can be a string or array. + +To auto-tag columns with `ai_hint`, use: `"tags": [{ "tag": "cost_analysis", "type": "ai_hints" }]` + +--- + +## Tags + +Tags are used for model categorization, filtering, and Lightdash integration. + +### Default Tags by Layer + +The framework automatically assigns tags based on model layer: + +| Layer | Auto-Assigned Tag | Auto-Excluded Tags | +| ------ | ----------------- | -------------------------------------------------------- | +| `stg` | `staging` | `intermediate`, `lightdash`, `lightdash-explore`, `mart` | +| `int` | (none) | `lightdash`, `lightdash-explore`, `staging`, `mart` | +| `mart` | `mart` | `staging`, `intermediate` | + +This means: + +- **Mart models automatically get the `mart` tag** — you don't need to add it manually +- Staging models won't appear in Lightdash by default (excluded from `lightdash` tag) +- To make a model appear in Lightdash, add the `lightdash` tag explicitly + +### Tag Types + +Tags can be simple strings or objects with a `type`: + +```jsonc +{ + "tags": [ + "my_tag", // simple string — inherited by downstream models + { "tag": "local_only", "type": "local" }, // NOT inherited downstream + { "tag": "staging", "type": "exclude" }, // removes inherited tag + { "tag": "cost_analysis", "type": "ai_hints" }, // auto-adds 'ai' tag to columns with ai_hint + ], +} +``` + +| Type | Behavior | +| ---------- | ------------------------------------------------------------------ | +| (string) | Inherited by all downstream models | +| `inherit` | Explicitly inherit a tag from upstream models | +| `local` | Applied only to this model, not inherited | +| `exclude` | Removes a tag that would otherwise be inherited | +| `ai_hints` | Auto-adds `ai` tag to metrics/dimensions that have `ai_hint` field | + +### Tag Inheritance + +Tags flow downstream through the model DAG: + +```text +stg model (tags: ["my_project"]) + → int model (inherits "my_project", adds "aggregated") + → mart model (inherits "my_project", "aggregated", auto-adds "mart") +``` + +To prevent a tag from flowing downstream, use `"type": "local"`. + +To remove an inherited tag, use `"type": "exclude"`. + +--- + +## Data Tests + +| Test | Use Case | +| -------------------------- | ----------------------------------------------------------------- | +| `equal_row_count` | Joins where row count should stay the same (1-to-1 relationships) | +| `equal_or_lower_row_count` | Joins with filtering that may reduce rows | +| `no_null_aggregates` | Ensure aggregation columns aren't null | +| `not_null` | Required columns that should never be null | +| `unique` | Primary key or unique identifier columns | + +```jsonc +{ + "data_tests": [ + { "type": "equal_row_count", "column_name": "portal_partition_daily" }, + ], +} +``` diff --git a/templates/_agents-dj/reference/materialization.md b/templates/_agents-dj/reference/materialization.md new file mode 100644 index 0000000..0ec7648 --- /dev/null +++ b/templates/_agents-dj/reference/materialization.md @@ -0,0 +1,108 @@ +# Materialization, Incremental Strategies & Framework Columns + +Load this when setting how a model is stored (view / ephemeral / incremental), choosing an incremental strategy or storage format, or working with the framework-injected partition and audit columns. + +## Materialization & Incremental Strategies + +| Layer | Default | Common Override | +| ------ | --------- | --------------------------------------------------------- | +| `stg` | ephemeral | `"materialization": "incremental"` for large sources | +| `int` | ephemeral | `"materialization": "incremental"` for large aggregations | +| `mart` | view | Not configurable — marts are always views | + +**Materialization types**: `ephemeral` (CTE, no table), `incremental` (processes new data only) + +### Materialization (Preferred) + +Use the `materialization` field instead of the legacy `materialized` + `incremental_strategy` + `partitioned_by` combination. It accepts a string shorthand or a structured object. + +**String shorthand** (equivalent to legacy `materialized`): + +```jsonc +{ + "materialization": "incremental", // or "ephemeral" +} +``` + +**Structured form** (full control): + +```jsonc +{ + "materialization": { + "type": "incremental", + "format": "iceberg", // optional: "delta_lake", "hive", or "iceberg" + "partitions": ["portal_partition_daily"], // optional: columns to partition by + "bucket": { "column": "tenant_name", "count": 32 }, // optional: { column, count } or an array of them + "sorted_by": ["tenant_name", "product_area"], // optional: columns to sort by within each file/bucket + "strategy": { "type": "delete+insert" }, // optional: see "Incremental strategies" below + "database": "custom_database", // optional: override target database + }, +} +``` + +- **`format`**: Controls storage format. Defaults to the project's `storage_type` variable in `dbt_project.yml`. Iceberg uses `partitioning` keyword; Delta Lake/Hive uses `partitioned_by`. When a per-model `format` is not set, the project-level `dbt_project.yml` vars `storage_type`, `etl_schema`, and `project_catalog` drive storage-specific SQL generation. +- **`bucket`**: Hash-bucket the table by one or more columns. On **Iceberg** each entry becomes a `bucket(column, count)` transform inside `partitioning` (per-column counts allowed). On **Hive/Glue** it emits `bucketed_by` + a single shared `bucket_count` (all entries must use the same `count`). **Not supported on Delta Lake.** The bucket column must be one of the model's `select` columns. +- **`sorted_by`**: Columns to sort data by within each written file. On **Iceberg** it is a standalone sort order; on **Hive/Glue** it sorts within buckets and **requires `bucket`**. **Not supported on Delta Lake.** Columns sort ascending. +- **`strategy`**: See "Incremental strategies" below. If omitted, the extension default applies (configurable via `dj.materialization.defaultIncrementalStrategy`, defaults to `overwrite_existing_partitions`). + +#### Incremental strategies (dbt-trino) + +| Strategy | Shape | When to use | Caveat | +| -------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `append` | `{ "type": "append" }` | Fast insert-only; no de-dup | Upstream must guarantee no duplicates in the new slice | +| `delete+insert` | `{ "type": "delete+insert", "unique_key": "..." }` | Partition-safe upsert (**safe default**) | `unique_key` is auto-derived from partitions when omitted | +| `merge` | `{ "type": "merge", "unique_key": "id", "merge_update_columns": [...], "merge_exclude_columns": [...] }` | Row-level upsert on a primary key | **dbt-trino requires Iceberg format.** Set `materialization.format: "iceberg"` or the project var `storage_type: iceberg` | +| `overwrite_existing_partitions` | `{ "type": "overwrite_existing_partitions" }` | Drop & rewrite only partitions present in the new slice | **Requires a custom dbt macro in your project** (e.g. `get_incremental_overwrite_existing_partitions_sql`). DJ does NOT ship this macro and dbt-trino does NOT provide it natively. `unique_key` is **not applicable** for this strategy the macro derives partitions from the new slice itself, and the schema rejects `unique_key`. If your project does not define the macro, use `{ "type": "delete+insert" }` instead, behavior is equivalent for partition-aligned daily/monthly incrementals when `unique_key` is the partition column. | +| `dj_iceberg_partition_overwrite` | `{ "type": "dj_iceberg_partition_overwrite" }` | Drop & rewrite only partitions present in the new slice on **Iceberg** tables | **Shipped by DJ.** No consumer macro required, `macros/strategies.sql` is auto-copied to `/macros/_ext_/strategies.sql` on **DJ: Refresh Projects**. The dispatch macro is `get_incremental_dj_iceberg_partition_overwrite_sql`. **Requires Iceberg format**: set `materialization.format: "iceberg"` or project var `storage_type: iceberg`; otherwise DJ flags it in the Problems tab. `unique_key` is **not applicable**, the macro derives partitions from the new slice itself. On Delta Lake / Hive use `{ "type": "delete+insert" }` instead. | + +### Legacy Incremental Configuration + +Still supported but prefer `materialization` above: + +```jsonc +{ + "materialized": "incremental", + "incremental_strategy": { "type": "delete+insert" }, // or "merge" with "unique_key" + "partitioned_by": ["portal_partition_daily"], +} +``` + +**Date filter options**: `"exclude_date_filter": true` (skip all date filtering), `"exclude_daily_filter": true` (skip daily partition filter only) + +--- + +## Portal-Specific Columns + +DJ automatically adds these columns: + +### `portal_source_count` + +Auto-generated `count(*)` for row tracking. Exclude with `"exclude_portal_source_count": true`. + +### Partition Columns + +Created from `interval` on datetime columns: + +| Interval | Generated Column | +| --------- | -------------------------------- | +| `"day"` | `portal_partition_daily` | +| `"hour"` | `portal_partition_hourly` | +| `"month"` | `portal_partition_monthly` | +| `"year"` | (none — only truncates datetime) | + +Drop all of them with `"exclude_portal_partition_columns": true`, or drop any +subset with an array, e.g. `"exclude_portal_partition_columns": ["portal_partition_hourly"]` +removes only the listed columns and keeps the rest. An array overrides +`exclude_framework_artifacts` at the same scope (narrowing its all-partitions +exclusion to just the listed columns). + +### Source-Level Configuration + +```jsonc +{ + "meta": { + "portal_source_count": { "exclude": true }, + "portal_partition_columns": { "daily": "custom_date_column" }, + }, +} +``` diff --git a/templates/_agents-dj/reference/meta-and-governance.md b/templates/_agents-dj/reference/meta-and-governance.md new file mode 100644 index 0000000..ad6c8a1 --- /dev/null +++ b/templates/_agents-dj/reference/meta-and-governance.md @@ -0,0 +1,80 @@ +# Custom Meta & Governance Metadata + +Load this when attaching free-form `meta` (ownership, compliance, SLAs), applying the optional governance vocabulary, or checking which `meta` keys the framework owns. + +## Custom Meta (Free-form) + +Both `.model.json` and `.source.json` accept **free-form user-defined keys** on their `meta` blocks. Use this to attach arbitrary metadata (ownership, compliance tags, process info, SLAs, etc.) that you want to surface in the generated `.yml` and consume downstream (dbt docs, Lightdash, custom tooling). + +Schemas: `model.meta.schema.json`, `column.meta.schema.json`, `source.meta.schema.json`, `source.table.meta.schema.json`. + +### Governance metadata conventions (optional) + +These keys are **not required and not enforced** by the framework — they are a shared vocabulary so that projects that choose to track governance metadata do so consistently. Offer them when authoring; if the user skips, omit them entirely (do not write placeholders). Teams that want them mandatory enforce that themselves (CI, review, custom validation). Before offering, mirror the keys the project already uses by scanning sibling models. + +| Key | Scope | Meaning | +| ---------------- | -------------- | --------------------------------------------------------------------------- | +| `owner` | model / source | Owning team or individual (e.g., `finops-team`) | +| `owner_slack` | model / source | Contact channel (e.g., `#finops-team`) | +| `pii` | model / column | Whether the model/column carries personally identifiable info | +| `classification` | model / column | Sensitivity tier (e.g., `public`, `internal`, `confidential`, `restricted`) | +| `compliance` | model / column | Applicable regimes (e.g., `["gdpr", "ccpa"]`) | +| `freshness_sla` | model / source | Expected freshness (e.g., `daily by 06:00 UTC`) | + +Column-level `pii` / `classification` / `compliance` inherit through clean passthrough selects (see below), so tagging a source or staging column once can propagate downstream. + +### Model-level meta + +Root `meta` block on any model type: + +```jsonc +{ + "type": "mart_select_model", + "group": "finance", + "topic": "billing", + "name": "accounts_daily", + "from": { "model": "int__finance__billing__accounts_daily" }, + "select": [...], + "meta": { + "owner": "finops-team", + "owner_slack": "#finops-team", + "freshness_sla": "daily by 06:00 UTC", + "pii": false, + }, +} +``` + +- Free-form keys flow through to the emitted `.yml` verbatim. +- **No automatic inheritance**: each model declares its own model-level meta (model-level meta is not inherited from upstream models). + +### Column-level meta + +Any select item on `.model.json` accepts a `meta` object: + +```jsonc +{ + "name": "email", + "type": "dim", + "meta": { "pii": true, "compliance": ["gdpr", "ccpa"] }, +} +``` + +- **Inheritance**: Column-level free-form meta IS inherited through **clean passthrough selects** (plain string selects and named-column selects without `expr`). `expr`-based selects (including `expr`-based renames) do **not** inherit meta. +- Downstream per-key overrides work as expected: a downstream column meta key overwrites the inherited key; keys the downstream doesn't declare stay inherited. + +### Framework-reserved keys under `meta` + +A small set of keys are owned by the framework — it writes them into the emitted YAML's `meta` block from structured sibling fields. Authoring any of these under `meta` directly is allowed by the schema but will be silently overwritten at emit time, and the extension surfaces a **Warning-severity diagnostic** in the Problems tab pointing to the canonical field. + +| Scope | Key | Canonical authoring location | +| ------ | ------------------------------ | -------------------------------------------------------- | +| model | `metrics` | `lightdash.metrics` on the model | +| model | `portal_partition_columns` | framework-derived; do not author | +| model | `local_tags` | `tags: [{ "type": "local", "tag": "..." }]` on the model | +| model | `case_sensitive` | `lightdash.case_sensitive` on the model | +| model | (any key on `lightdash.table`) | `lightdash.table.` on the model | +| column | `type` | `type` on the select item | +| column | `dimension` | `lightdash.dimension` on the select item | +| column | `metrics` | `lightdash.metrics` on the select item | +| column | `case_sensitive` | `lightdash.case_sensitive` on the select item | +| column | `origin` | framework-derived from upstream lookup; do not author | diff --git a/templates/_agents-dj/reference/model-types.md b/templates/_agents-dj/reference/model-types.md new file mode 100644 index 0000000..f52d979 --- /dev/null +++ b/templates/_agents-dj/reference/model-types.md @@ -0,0 +1,441 @@ +# Model Types & Column Selection + +Load this when shaping a model. Covers the 11 model types with worked examples, the Advanced map (CTEs, rollup, shorthands, subqueries), the select-column vocabulary, and common optional fields. **Exact shapes and required keys live in `.dj/schemas/model.type..schema.json`** (follow `$ref`s) — read the type schema before writing JSON. + +## Model Types + +### 1. `stg_select_source` — Staging: Select from a Source + +Selects columns from a raw data source table. + +```jsonc +{ + "type": "stg_select_source", + "group": "my_group", + "topic": "my_topic", + "name": "raw_data_conformed", + "materialized": "incremental", // optional: "incremental" or "ephemeral" + "from": { + "source": "my_database__my_schema.my_table", // format: __.
+ }, + "select": [ + "account_id", // simple column reference (string) + "region", + { + "name": "cost", // column with additional config + "type": "fct", // "dim" (dimension) or "fct" (fact/measure) + "expr": "CAST(cost AS double)", // optional SQL expression override + }, + { + "name": "event_date", + "type": "dim", + "data_type": "date", // optional Trino data type + }, + ], + "where": { + // optional filter + "and": [{ "expr": "cost > 0" }], + }, +} +``` + +**Required fields**: `type`, `group`, `topic`, `name`, `from.source`, `select` + +### 2. `stg_select_model` — Staging: Select from Another Model + +Selects from another model (commonly used for seeds). + +```jsonc +{ + "type": "stg_select_model", + "group": "my_group", + "topic": "my_topic", + "name": "lookup_mapping", + "from": { + "model": "seed__my_topic__lookup_mapping", + }, + "select": ["key_column", "value_column"], +} +``` + +**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `select` + +### 3. `stg_union_sources` — Staging: Union Multiple Sources + +Unions multiple source tables. + +```jsonc +{ + "type": "stg_union_sources", + "group": "my_group", + "topic": "my_topic", + "name": "combined_accounts", + "from": { + "source": "my_database__my_schema.accounts_us", + "union": { + "sources": [ + "my_database__my_schema.accounts_eu", + "my_database__my_schema.accounts_apac", + ], + }, + }, + "select": ["account_id", "account_name"], +} +``` + +**Required fields**: `type`, `group`, `topic`, `name`, `from.source`, `from.union.sources` + +### 4. `int_select_model` — Intermediate: Select from a Model + +Transforms data from a single upstream model. Supports optional `from.rollup` for time-grain re-aggregation (provides `int_rollup_model` functionality with more control over columns). + +```jsonc +{ + "type": "int_select_model", + "group": "my_group", + "topic": "my_topic", + "name": "daily_summary", + "materialized": "incremental", + "from": { + "model": "stg__my_group__my_topic__raw_data_conformed", + // optional: re-aggregate to coarser time grain + "rollup": { + "interval": "day", // "day", "hour", "month", "year" + }, + }, + "select": [ + "account_id", + { + "name": "cost", + "type": "fct", + "agg": "sum", // auto-creates aggregation columns: sum, count, min, max, hll, tdigest + }, + { + "name": "datetime", + "interval": "day", // interval column: "day", "hour", "month", "year" + }, + ], + "group_by": [ + { "type": "dims" }, // group by all dimension columns + ], + "where": "cost > 0", // simple string where clause +} +``` + +**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `select` + +### 5. `int_join_models` — Intermediate: Join Multiple Models + +Joins a primary model with one or more additional models. Supports optional `from.rollup` for time-grain re-aggregation alongside joins. + +```jsonc +{ + "type": "int_join_models", + "group": "my_group", + "topic": "my_topic", + "name": "enriched_daily", + "materialized": "incremental", + "from": { + "model": "int__my_group__my_topic__daily_summary", + // optional: re-aggregate to coarser time grain + "rollup": { + "interval": "day", // "day", "hour", "month", "year" + }, + "join": [ + { + "model": "int__my_group__other_topic__dimension_table", + "type": "inner", // "left", "inner", "right", "full", "cross" + "on": { + "and": [ + "account_id", // shorthand: join on same column name + "event_date", + { "expr": "a.region = b.region" }, // or explicit SQL expression + ], + }, + }, + ], + }, + "select": [ + { + "model": "int__my_group__my_topic__daily_summary", + "type": "dims_from_model", // "all_from_model", "dims_from_model", "fcts_from_model" + "include": ["account_id", "region"], // optional: filter which columns + }, + { + "model": "int__my_group__other_topic__dimension_table", + "type": "dims_from_model", + }, + { + "name": "allocated_cost", + "type": "fct", + "expr": "sum(a.cost * b.ratio)", + }, + ], + "group_by": [{ "type": "dims" }], +} +``` + +**Required fields**: `type`, `group`, `name`, `from.model`, `from.join`, `select` + +### 6. `int_union_models` — Intermediate: Union Multiple Models + +```jsonc +{ + "type": "int_union_models", + "group": "my_group", + "topic": "my_topic", + "name": "all_providers_daily", + "from": { + "model": "int__my_group__provider_a__daily", + "union": { + "models": [ + "int__my_group__provider_b__daily", + "int__my_group__provider_c__daily", + ], + }, + }, +} +``` + +**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `from.union.models` + +### 7. `int_rollup_model` — Intermediate: Time-based Rollup + +Aggregates data to a coarser time interval. + +```jsonc +{ + "type": "int_rollup_model", + "group": "my_group", + "topic": "my_topic", + "name": "daily_from_hourly", + "materialized": "incremental", + "from": { + "model": "int__my_group__my_topic__hourly_summary", + "rollup": { + "interval": "day", // "day", "hour", "month", "year" + }, + }, +} +``` + +**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `from.rollup.interval` + +### 8. `int_lookback_model` — Intermediate: Trailing Window Aggregation + +Aggregates over a trailing number of days. + +```jsonc +{ + "type": "int_lookback_model", + "group": "my_group", + "topic": "my_topic", + "name": "trailing_30d", + "materialized": "incremental", + "from": { + "model": "int__my_group__my_topic__daily_summary", + "lookback": { + "days": 30, + "exclude_event_date": false, // optional + }, + }, + "select": ["account_id", { "name": "cost", "type": "fct", "agg": "sum" }], + "group_by": [{ "type": "dims" }], +} +``` + +**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `from.lookback.days`, `select` + +### 9. `int_join_column` — Intermediate: Cross Join on Unnested Column + +Cross joins a model with an unnested array column. + +**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `from.join.column`, `select` + +### 10. `mart_select_model` — Mart: Select from a Model + +Final business-ready model selecting from an intermediate model. + +```jsonc +{ + "type": "mart_select_model", + "group": "my_group", + "topic": "my_topic", + "name": "accounts_daily", + "from": { + "model": "int__my_group__my_topic__daily_summary", + }, + "select": ["account_id", "cost_sum"], +} +``` + +**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `select` + +### 11. `mart_join_models` — Mart: Join Multiple Models + +Final business-ready model that joins multiple intermediate models. Same join syntax as `int_join_models`. + +**Required fields**: `type`, `group`, `topic`, `name`, `from.model`, `from.join`, `select` + +### Advanced: CTEs, rollup, shorthands, subqueries + +For **`int_select_model`**, **`int_join_models`**, **`int_union_models`**, **`mart_select_model`**, **`mart_join_models`** (not staging). **Shapes and required keys** live in **`.dj/schemas/model.type..schema.json`** and **`$ref`** targets — read those first; this section is a map, not a full spec. + +- **CTEs**: Optional ordered **`ctes`**. **`model.ctes.schema.json`**, **`model.cte.schema.json`**. Authoring rules and gotchas: [ctes-and-subqueries.md](ctes-and-subqueries.md). +- **`from`**: Each type’s **`from`** **`anyOf`** lists legal combinations (**`model`**, **`cte`**, **`join`**, optional **`rollup`** on **`int_*` select/join only** — not on marts). +- **Rollup on select/join**: Optional **`rollup`** on **`from.model`** for **`int_select_model`** and **`int_join_models`** only (not marts). Requires the upstream to expose a select column with an **`interval`** field (e.g. **`{ "name": "datetime", "interval": "day" }`**). Keeps a normal **`select`** / join; coarser **`interval`** triggers **re-aggregation** of declarative **`agg`/`aggs`**. **`model.from.rollup.schema.json`**. For **`group_by` / `agg` / `expr`** rules see `.agents/dj/AGENTS.md` **Important Conventions** (#9–#10). +- **Rollup inside a CTE**: Optional **`rollup`** on a CTE's **`from.model`** or **`from.cte`** (not **`from.source`**, not **`from.union`**). Re-aggregates that CTE's source to a coarser grain — same DATE_TRUNC + suffix-agg + GROUP BY behavior as the model-level rollup, but scoped to one stage of the pipeline. Available on every CTE-supporting model type. **`exclude_datetime`** / **`exclude_framework_artifacts`** at the same scope is rejected as a conflict; chained rollups (e.g. month CTE feeding a year CTE) work end-to-end. +- **Shorthands & CTE columns**: **`dims_from_*`**, **`fcts_from_*`**, **`all_from_*`** and explicit CTE column objects — **`model.select.model.schema.json`**, **`model.select.cte.schema.json`**, related **`model.select.*`**. CTE bulk selects support **`exclude`/`include`** filters and **inherit dim/fct types** from upstream. +- **`where` / `having`**: Nested **`subquery`** — **`model.subquery.schema.json`**. See [ctes-and-subqueries.md](ctes-and-subqueries.md). +- **`"dims"` shorthand**: **`group_by: "dims"`** equivalent to **`[{ "type": "dims" }]`**; join **`on: "dims"`** auto-joins on all shared dimension columns — **`model.group_by.schema.json`**. +- **Materialization**: String **`"incremental"`** / **`"ephemeral"`** or structured object with **`type`**, **`format`**, **`partitions`**, **`strategy`**, **`database`** — **`model.materialization.schema.json`**. See [materialization.md](materialization.md). + +--- + +## Select Column Types + +### Simple String Reference + +```jsonc +"column_name" +``` + +Selects a column by name with default dimension type. + +### Named Column (`dim` or `fct`) + +```jsonc +{ + "name": "column_name", + "type": "dim", // "dim" (dimension) or "fct" (fact/measure) — default is "dim" + "data_type": "varchar", // optional: Trino data type + "description": "Description", // optional + "expr": "CAST(col AS varchar)", // optional: SQL expression override +} +``` + +### Aggregated Column + +```jsonc +{ + "name": "cost", + "type": "fct", + "agg": "sum", // "sum", "count", "min", "max", "hll", "tdigest" +} +``` + +This auto-creates an aggregation column named `_` (e.g., `cost_sum`). + +### Multi-Aggregations + +```jsonc +{ + "name": "cost", + "type": "fct", + "aggs": ["sum", "count", "min", "max"], +} +``` + +### From Another Model (in join/union models) + +```jsonc +{ + "model": "int__my_group__my_topic__daily_summary", + "type": "dims_from_model", // "all_from_model", "dims_from_model", "fcts_from_model" +} +``` + +With optional include/exclude: + +```jsonc +{ + "model": "int__my_group__my_topic__daily_summary", + "type": "dims_from_model", + "include": ["account_id", "region"], + "exclude": ["internal_id"], +} +``` + +### Named Column from Specific Model + +```jsonc +{ + "model": "int__my_group__my_topic__daily_summary", + "name": "cost", + "type": "fct", +} +``` + +### From Source (in staging models) + +```jsonc +{ + "source": "my_database__my_schema.my_table", + "type": "all_from_source", +} +``` + +### From CTE (in models with `ctes`) + +```jsonc +{ + "cte": "my_cte_name", + "type": "all_from_cte", // "all_from_cte", "dims_from_cte", "fcts_from_cte" +} +``` + +With optional include/exclude: + +```jsonc +{ + "cte": "my_cte_name", + "type": "dims_from_cte", + "include": ["account_id", "region"], +} +``` + +Named column from a CTE: + +```jsonc +{ + "cte": "my_cte_name", + "name": "cost", + "type": "fct", +} +``` + +### Interval (Datetime) + +```jsonc +{ + "name": "datetime", + "interval": "day", // "day", "hour", "month", "year" +} +``` + +--- + +## Common Optional Model Fields + +| Field | Type | Description | +| ---------------------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `description` | string | Model description | +| `tags` | array | Tags for categorization, e.g. `["my_tag", "my_group"]` | +| `materialized` | string | Legacy: `"incremental"` or `"ephemeral"` (default is view-like). Prefer `materialization` instead. | +| `materialization` | string/object | Preferred. String `"incremental"` or `"ephemeral"`, or object `{ "type": "incremental", "format"?, "partitions"?, "strategy"?, "database"? }`. See [materialization.md](materialization.md). | +| `incremental_strategy` | object | Legacy: `{ "type": "delete+insert" }` or `{ "type": "merge", "unique_key": "id" }`. Prefer `materialization.strategy`. | +| `sql_hooks` | object | `{ "pre": "SET ...", "post": "..." }` — SQL to run before/after (staging and intermediate only) | +| `partitioned_by` | array | Legacy: Column(s) to partition by. Prefer `materialization.partitions`. | +| `group_by` | string/array | `"dims"` or `[{ "type": "dims" }]` or `["col1", "col2"]` or `[{ "expr": "..." }]` | +| `where` | string/object | Filter clause — simple string or `{ "and": [...], "or": [...] }` | +| `having` | object | HAVING clause (same shape as `where`) | +| `order_by` | array | ORDER BY columns | +| `limit` | integer | LIMIT clause | +| `offset` | integer | OFFSET clause | +| `exclude_date_filter` | boolean | Skip auto date filtering | +| `exclude_daily_filter` | boolean | Skip daily partition filter | +| `exclude_portal_partition_columns` | boolean/array | Drop portal partition columns. `true` drops all; an array (e.g. `["portal_partition_hourly"]`) drops only the named ones | +| `exclude_portal_source_count` | boolean | Don't add portal source count | +| `data_tests` | array | dbt test configurations | +| `lightdash` | object | Lightdash BI tool configuration | +| `meta` | object | Free-form user-defined metadata (see [meta-and-governance.md](meta-and-governance.md)) | diff --git a/templates/_agents-dj/reference/pitfalls-and-validation.md b/templates/_agents-dj/reference/pitfalls-and-validation.md new file mode 100644 index 0000000..aa834e7 --- /dev/null +++ b/templates/_agents-dj/reference/pitfalls-and-validation.md @@ -0,0 +1,119 @@ +# Pitfalls, Validation & Schema Index + +Load this to troubleshoot a generation error, validate a `.model.json` / `.source.json`, follow the file-creation checklist, or locate the exact schema file for a field. + +## File Creation Checklist + +When adding a new model to the project: + +1. **Identify the layer**: staging (`stg_*`), intermediate (`int_*`), or mart (`mart_*`) +2. **Choose the model type** based on the data transformation needed +3. **Create only the `.model.json` file** — SQL and YML are auto-generated +4. **Place the file** in the correct directory: `models////` +5. **Name the file**: `______.model.json` +6. If reading from a new external table, **create a `.source.json` file** first +7. **Do NOT** create or edit `.sql` or `.yml` files — they are auto-generated by DJ + +When adding a new source: + +1. **Create the `.source.json` file** at `models/sources//__.source.json` +2. Define all tables and their columns with Trino-compatible data types +3. Configure `meta.etl` if the source should be scheduled +4. Configure `meta.partition_date` or `meta.partitions` for partition pruning +5. **Do NOT** create or edit the corresponding `.yml` file — it is auto-generated + +--- + +## Common Pitfalls + +| Problem | Solution | +| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Column Not Found** | Verify columns exist in upstream `select`. Check `exclude` filters in `all_from_model` or `all_from_cte`. | +| **Row Multiplication** | Add `equal_row_count` test. Verify join conditions. Aggregate "many" side before joining. | +| **Duplicate Column Names** | Use `exclude` on one model or rename with `expr`. | +| **Aggregation Without Group By** | Always add `"group_by": "dims"` (or `[{ "type": "dims" }]`) when using `agg`. | +| **Un-aggregated `fct` + group_by** | Every `fct` in the main `select` must set `agg`/`aggs`, wrap an aggregate in `expr` (e.g. `sum(x)`, `merge(cast(x as hyperloglog))`), or `exclude_from_group_by: true`. Applies to scalar and bulk CTE carriers. | +| **Lightdash metrics on a CTE select** | Not supported; only the main-model `select` feeds Lightdash metric generation. Keep the pre-aggregated column in the CTE and declare `lightdash.metrics` / `lightdash.metrics_merge` on the main-model `select`. `lightdash.dimension` on CTE selects is still supported. | +| **Duplicated `portal_source_count` in CTE** | When a CTE's `from` is `{ model }` or `{ cte }`, `portal_source_count` auto-injects (aggregated with `count` when the CTE has `group_by`). Don't add it manually; set `override_suffix_agg: true` only for a differently-aggregated variant alongside the audit column. | +| **Missing `portal_partition_*` / `datetime` in CTE** | When a CTE's `from` is `{ model }` or `{ cte }`, `datetime` and `portal_partition_*` auto-inject from the upstream even if a narrow `dims_from_model.include` list omitted them — do not add them by hand. Explicit `{ "name": "datetime", "interval": X }` drives partition exclusion (`day` drops hourly, `month` drops hourly+daily, `year` drops all three). Opt out per CTE with `"exclude_portal_partition_columns": true` (mirrors the main-model flag); `datetime` itself has no opt-out. | +| **Invalid Source Reference** | Use format `__.` (double underscore, then dot). | +| **Lightdash Case Sensitivity** | Optionally set `"case_sensitive": true/false` at model or column level to override the Lightdash global default. | +| **CTE group_by with computed cols** | Don't use bare string aliases (e.g., `["month"]`) for columns defined with `expr`. Use `"dims"` or `{ "expr": "..." }`. | +| **materialized vs materialization** | Both work; `materialization` is preferred and supports structured config (format, partitions, strategy). | +| **CTE column type mismatch** | Plain string selects in CTEs inherit dim/fct type from upstream. Verify with `dims_from_cte`/`fcts_from_cte`. | +| **Framework-reserved `meta` keys** | Column `type`/`dimension`/`metrics`/`case_sensitive` and model `metrics`/`local_tags`/`case_sensitive` are framework-owned — see [meta-and-governance.md](meta-and-governance.md). | +| **Dead outer-layer warning** | Main `select` is a single `all_from_cte` / `dims_from_cte` passthrough of one CTE with identical `group_by` and no extra filter / limit / projection — drop the wrapper or add work to it. See [ctes-and-subqueries.md](ctes-and-subqueries.md). | + +--- + +## Validation Rules + +- `group`: lowercase alphanumeric with underscores, no leading/trailing underscores, no consecutive underscores (`^(?!.*__.*)(?!_)(?!.*_$)([a-z]|[0-9]|_)+$`) +- `topic`: same pattern as group +- `name`: same pattern as group +- `tags`: alphanumeric with underscores and hyphens +- `database`: lowercase alphanumeric with underscores +- `schema`: lowercase alphanumeric with underscores +- Column `name`: lowercase alphanumeric with underscores and dots +- Source references: format `__.` +- Model references: the full model name (e.g., `int__my_group__my_topic__daily_summary`) +- `materialized`: must be `"incremental"` or `"ephemeral"` (legacy; prefer `materialization`) +- `materialization`: string `"incremental"` or `"ephemeral"`, or object with `"type": "incremental"` and optional `format`, `partitions`, `strategy`, `database` +- `format` (in `materialization`): must be `"delta_lake"`, `"hive"`, or `"iceberg"` +- `agg`: must be one of `"sum"`, `"count"`, `"min"`, `"max"`, `"hll"`, `"tdigest"` +- `interval`: must be one of `"day"`, `"hour"`, `"month"`, `"year"` +- Join `type`: must be one of `"left"`, `"inner"`, `"right"`, `"full"`, `"cross"` + +--- + +## Schema Reference (Dynamic Lookup) + +The authoritative JSON Schemas for all model and source types live in the `.dj/schemas/` directory at the workspace root. **When you need exact field definitions, allowed values, or validation rules beyond what is documented in these guides, read the relevant schema file.** + +### Key schema files + +| Schema File | Purpose | +| ------------------------------------------- | --------------------------------------------------------------- | +| `model.schema.json` | Top-level model validator (dispatches to type-specific schemas) | +| `model.type.stg_select_source.schema.json` | Schema for `stg_select_source` models | +| `model.type.stg_select_model.schema.json` | Schema for `stg_select_model` models | +| `model.type.stg_union_sources.schema.json` | Schema for `stg_union_sources` models | +| `model.type.int_select_model.schema.json` | Schema for `int_select_model` models | +| `model.type.int_join_models.schema.json` | Schema for `int_join_models` models | +| `model.type.int_join_column.schema.json` | Schema for `int_join_column` models | +| `model.type.int_union_models.schema.json` | Schema for `int_union_models` models | +| `model.type.int_rollup_model.schema.json` | Schema for `int_rollup_model` models | +| `model.type.int_lookback_model.schema.json` | Schema for `int_lookback_model` models | +| `model.type.mart_select_model.schema.json` | Schema for `mart_select_model` models | +| `model.type.mart_join_models.schema.json` | Schema for `mart_join_models` models | +| `source.schema.json` | Top-level source validator | +| `source.table.schema.json` | Source table structure | +| `source.etl.schema.json` | ETL scheduling configuration | +| `source.partition.schema.json` | Partition filter configuration | +| `source.partition_date.schema.json` | Partition date configuration | +| `model.select.col.schema.json` | Column selection options | +| `model.select.expr.schema.json` | Expression-based column selection | +| `model.select.model.schema.json` | Select columns from another model | +| `model.select.source.schema.json` | Select columns from a source | +| `model.from.join.models.schema.json` | Join configuration | +| `model.from.rollup.schema.json` | Rollup configuration for time-grain re-aggregation | +| `model.sql_hooks.schema.json` | `pre` / `post` SQL for staging and intermediate models | +| `model.materialization.schema.json` | Materialization config (string shorthand or structured object) | +| `model.incremental_strategy.schema.json` | Incremental strategy (`delete+insert` or `merge`) | +| `model.format.schema.json` | Storage format (`delta_lake`, `hive`, `iceberg`) | +| `model.partitions.schema.json` | Partition columns for materialization | +| `model.group_by.schema.json` | Group by config (`"dims"` shorthand or array) | +| `model.subquery.schema.json` | Inline subquery definition (WHERE, HAVING, JOIN ON) | +| `model.cte.schema.json` | Single CTE definition | +| `model.ctes.schema.json` | CTE array configuration | +| `model.select.cte.schema.json` | Select columns from a CTE | +| `column.lightdash.schema.json` | Lightdash BI column configuration | +| `model.lightdash.schema.json` | Lightdash BI model-level configuration | +| `model.meta.schema.json` | Free-form model-level meta (reserved-key notes inside) | +| `column.meta.schema.json` | Free-form column-level meta (reserved-key notes inside) | +| `source.meta.schema.json` | Free-form source-level meta | +| `source.table.meta.schema.json` | Free-form source-table-level meta | + +When creating or editing a model, read the type-specific schema (e.g., `model.type.int_join_models.schema.json`) to confirm all required/optional fields and their exact constraints. Schemas use `$ref` to reference sub-schemas — follow those references as needed. + +You can also look at existing `.model.json` and `.source.json` files in the `models/` directory for real examples from this project. diff --git a/templates/_agents-dj/reference/running-dbt.md b/templates/_agents-dj/reference/running-dbt.md new file mode 100644 index 0000000..54f1106 --- /dev/null +++ b/templates/_agents-dj/reference/running-dbt.md @@ -0,0 +1,39 @@ +# Running dbt + +Load this when you need to run a dbt command from a terminal in this project — compile, parse, list, test, or a warehouse-writing run/build/seed. First read **Command & Query Execution Safety** in `.agents/dj/AGENTS.md`; the rules below are the mechanics, that section is the policy. + +## Activate the Python environment first + +dbt runs inside the project's Python virtual environment. A terminal you open does **not** inherit that environment automatically — activate it yourself before invoking `dbt`. + +1. **Find the venv.** Read `dj.pythonVenvPath` from `.vscode/settings.json` (workspace settings, then user settings). A relative value is resolved against the workspace root; an absolute path is used as-is. +2. **Fall back to the conventional venv.** If `dj.pythonVenvPath` is unset, look for `.venv/` at the workspace or project root (check for `.venv/bin/activate`). If dbt is already on `PATH`, you may skip activation. +3. **Activate and verify.** Run `source /bin/activate` (macOS/Linux) or `\Scripts\activate.bat` (Windows), then confirm with `dbt --version`. +4. **If none resolves,** ask the user to set `dj.pythonVenvPath` or to tell you how dbt is installed — do not guess an interpreter. + +## Run from the dbt project directory + +Run `dbt` from the directory that contains `dbt_project.yml`, not necessarily the workspace root — the dbt project may be nested. If more than one dbt project exists, confirm which one the model belongs to before running (see **Project & Environment Resolution** in `.agents/dj/AGENTS.md`). dbt reads its warehouse credentials from `~/.dbt/profiles.yml` by default; DJ does not override the profiles directory. + +## Command classes + +| Class | Commands | Notes | +| ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| **Read-only** (safe to run) | `dbt parse`, `dbt compile`, `dbt ls`, `dbt deps`, `dbt docs generate`, `dbt test`, `dbt source freshness` | `test` and `source freshness` issue `SELECT`s but write nothing. Keep them scoped with `--select`. | +| **Warehouse-writing** (per-command confirmation, never prod) | `dbt run`, `dbt build`, `dbt seed`, `dbt snapshot`, `dbt run-operation` | Confirm the target is non-production before each run. `--full-refresh` rebuilds tables from scratch — treat as destructive and confirm. | + +Never run a warehouse-writing command against a production target, even with confirmation. If you cannot confirm the target is non-production, stop and ask. + +## Common invocations + +- **Compile one model's generated SQL:** `dbt compile --select ` +- **List models:** `dbt ls --select ` +- **Test a model:** `dbt test --select ` +- **Refresh dependencies:** `dbt deps` +- **Build the manifest from scratch:** `dbt parse` (ask first if the manifest is missing) + +Selection syntax: `--select` / `-s` picks nodes, `--exclude` removes them, graph operators expand the set (`+model` = model and its ancestors, `model+` = model and its descendants). Prefer the narrowest selector that covers the task. + +## dbt reads generated SQL — sync first + +dbt compiles and runs the **generated `.sql`** files, not the `.model.json` sources. After editing a `.model.json`, ask the user to run **`DJ: Sync to SQL and YML`** (which regenerates the `.sql` / `.yml` and reparses the manifest when needed) before you compile or run that model — otherwise dbt sees stale SQL. You cannot run VS Code commands yourself; run `dbt parse` in the terminal only when a manifest must be built from scratch and the user has approved it. diff --git a/templates/_agents-dj/reference/running-lightdash.md b/templates/_agents-dj/reference/running-lightdash.md new file mode 100644 index 0000000..df6192e --- /dev/null +++ b/templates/_agents-dj/reference/running-lightdash.md @@ -0,0 +1,43 @@ +# Running the Lightdash CLI + +Load this when you need to run the Lightdash CLI (or reason about its connection) for a DJ project — starting a preview, downloading, or uploading dashboards-as-code. For authoring or editing the chart / dashboard YAML itself, use the `dj-create-lightdash-yaml` and `dj-edit-lightdash-yaml` skills; this reference is only the CLI, connection, and guardrail basics they share. + +## Executable and connection + +The `lightdash` CLI must be on `PATH` (commonly inside the same Python venv as dbt — see `.agents/dj/reference/running-dbt.md` for venv activation). Connection and target come from environment variables, the same names DJ and the CLI use: + +- `LIGHTDASH_URL` — instance base URL +- `LIGHTDASH_API_KEY` — API token +- `LIGHTDASH_PROJECT` — default project UUID +- `LIGHTDASH_PREVIEW_NAME` — preview name (defaults to `DJ Preview`) +- `LIGHTDASH_TRINO_HOST` — Trino host override (e.g. `host.docker.internal` under Docker) + +If the URL / API key / project are not set in your terminal, ask the user for them rather than guessing. + +## Relevant settings (`.vscode/settings.json`) + +- `dj.lightdash.dashboardsAsCodePath` — where chart/dashboard YAML lives (default `lightdash/`). +- `dj.lightdashProjectPath` / `dj.lightdashProfilesPath` — dbt project and profiles dirs for previews. +- `dj.lightdash.restrictedProjects` — array of `{ uuid, mode: "block" | "warn", label }`. `block` refuses the upload from the DJ **Upload tab**; `warn` requires confirmation. Treat any listed project as production. + +## Commands + +- **Start a preview:** `lightdash start-preview --name "" --profiles-dir ~/.dbt --project-dir -s -y`. Do not pass `--defer` (it routes to `dbt ls`, which rejects it); `-y` skips the credential prompt. Capture the printed preview project UUID (`.../projects//tables`). +- **Stop a preview:** `lightdash stop-preview --name ""` (DJ runs it with `CI=true` in the environment). Tears down the named preview project. +- **Download:** `lightdash download -p --project ` (add `-c ` / `-d ` to scope). +- **Upload:** `lightdash upload --project -c -d [--force] [--include-charts] [--validate]`. Net-new files require `--force`; always `--validate` (the only check that catches bad field IDs / missing slugs). + +## Preview registry (`.dj/lightdash/previews.json`) + +The Lightdash CLI has no "list previews" command, so DJ keeps its own registry at `.dj/lightdash/previews.json` — this is exactly what the `DJ: Lightdash Preview` webview lists. It holds 2-space JSON of the form `{ "previews": [ { "name", "url", "createdAt", "models": [...], "status": "active" | "inactive" } ] }`; DJ writes an `active` entry (deduped by `name`) when a preview starts and removes it when one stops. + +- **To enumerate current previews** (e.g. to choose one to stop), read this file — there is no CLI list to fall back on. +- **Keep it in sync when you run the CLI directly.** If you start a preview from the terminal instead of the webview, add or replace its entry (dedup by `name`) so the UI shows it; if you stop one, remove its entry. Match the shape exactly — `url` is the printed `.../projects//tables`, `createdAt` is an ISO-8601 timestamp, `models` is the `-s` selection, `status` is `"active"`. +- It lives under gitignored `.dj/` — never commit it. +- **Prefer the webview.** `DJ: Lightdash Preview` starts/stops previews _and_ maintains this registry and captures the URL for you; edit `previews.json` by hand only when you ran the CLI yourself. + +## Guardrails + +- **Confirm the target project UUID before any upload**, and pass `--project ` explicitly rather than relying on the `LIGHTDASH_PROJECT` default. Deliberately target a **preview**, not prod. +- `dj.lightdash.restrictedProjects` guards only the DJ **Upload tab** — it does **not** stop a direct `lightdash upload` (Lightdash allows it if you have permission). So never upload to a restricted/prod project from the CLI without explicit confirmation. +- **Prefer the DJ webview.** The `DJ: Lightdash - Dashboards as Code` tabs handle auth, working directory, and schema binding — offer to run the CLI directly only after confirming the target, and never instruct the user to run uploads blindly. diff --git a/templates/_agents-dj/reference/running-trino.md b/templates/_agents-dj/reference/running-trino.md new file mode 100644 index 0000000..7f2661c --- /dev/null +++ b/templates/_agents-dj/reference/running-trino.md @@ -0,0 +1,45 @@ +# Running Trino queries + +Load this when you need to run a Trino query from a terminal to inspect warehouse data or schema. First read **Command & Query Execution Safety** in `.agents/dj/AGENTS.md`; the rules below are the mechanics, that section is the policy. + +## Resolve the CLI executable + +DJ resolves the Trino CLI from the `dj.trinoPath` setting (in `.vscode/settings.json`), defaulting to `trino-cli` on `PATH`: + +- **Command name** (no path separators, e.g. `trino-cli` or `trino`) → used as-is, resolved from `PATH`. +- **Full path** ending in `trino` or `trino-cli` → used directly. +- **Directory path** → `${dir}/trino-cli` is tried first, then `${dir}/trino`. +- **Unset** → `trino-cli` on `PATH`. + +If neither `trino-cli` nor `trino` resolves, ask the user to install the Trino CLI or set `dj.trinoPath` — do not guess a path. + +## Connection comes from the environment + +DJ invokes the CLI with only the query and output format — it does **not** pass `--server` / `--catalog` / `--schema` / `--user` flags. The connection is read from the process environment, so the CLI (or the site's `trino-cli` wrapper) resolves it from these variables: + +`TRINO_HOST`, `TRINO_PORT`, `TRINO_USERNAME`, `TRINO_CATALOG`, `TRINO_SCHEMA` + +These usually live in the user's shell profile. If they are not set in your terminal: + +- Ask the user for the cluster host/port, catalog, and schema, and **confirm it is the intended, non-production cluster** before running anything. +- When using the open-source `trino` CLI (not a wrapper), pass the values as flags: `--server : --user --catalog --schema `. + +There is no password flag — authentication comes from the environment or the user's profile. + +## Invocation + +Mirror how DJ runs the CLI: + +```bash +trino-cli --execute "SHOW COLUMNS FROM \"\".\"\".\"
\"" --output-format=CSV_HEADER +``` + +- `--execute ""` runs one statement; `--file ` runs a file. +- `--output-format=CSV_HEADER` returns RFC-4180 CSV with a header row and handles complex Trino types (arrays, maps, rows) that the CLI's JSON format cannot serialize. +- **Quote every identifier** with double quotes (`"".""."
"`) so mixed-case or reserved names resolve. + +## Read-only by default + +Run only `SELECT` / `SHOW` / `DESCRIBE` / `EXPLAIN`. Add a `LIMIT` and constrain by partition on every ad-hoc `SELECT` — never trigger a full-history or unpartitioned scan just to check a shape. Any DDL/DML requires explicit per-command confirmation and must never target production. + +Before shelling out, prefer framework facilities: read `.source.json` / `.model.json` / `target/manifest.json` / `.dj/schemas/`, or use DJ's **Create Source** flow to browse catalogs, schemas, tables, and columns. Run the CLI only after the user confirms the connection and that the statement is read-only. diff --git a/templates/_agents-dj/reference/sources.md b/templates/_agents-dj/reference/sources.md new file mode 100644 index 0000000..20b4270 --- /dev/null +++ b/templates/_agents-dj/reference/sources.md @@ -0,0 +1,132 @@ +# Source Files & Scheduling + +Load this when defining an external table (`.source.json`) or reasoning about how the ETL schedule flows to models. + +## Source Files (`.source.json`) + +Source files define external database tables that staging models read from. They are placed at: +`models/sources//__.source.json` + +### Source Structure + +```jsonc +{ + "database": "my_database", // catalog/database name + "schema": "my_schema", // schema name + "tables": [ + { + "name": "my_table", // table name + "columns": [ + { + "name": "account_id", + "data_type": "varchar", // Trino data type + }, + { + "name": "cost", + "data_type": "double", + "description": "The raw cost amount", // optional + }, + ], + }, + ], +} +``` + +**Required fields**: `database`, `schema`, `tables` +**Required per table**: `name`, `columns` +**Required per column**: `name`, `data_type` + +### Source Naming + +The source name is derived as: `__` + +When referenced in a model's `from.source`, use: `__.` + +### Source ETL Configuration + +Sources can include ETL metadata in the `meta` field (at either schema or table level) to control scheduling: + +```jsonc +{ + "database": "my_database", + "schema": "my_schema", + "meta": { + "etl": { + "active": true, // whether ETL monitors this source + "backfill_start": "2024-01-01", // date to start backfilling from (YYYY-MM-DD) + "type": "event_count" // "event_count" (default) or "run_schedule" + }, + "event_datetime": { + "expr": "event_timestamp" // expression to extract event datetime + }, + "partition_date": { + "expr": "dt", // partition date expression + "interval": "day" // "day" or "month" + } + }, + "tables": [...] +} +``` + +#### ETL Types + +- **`event_count`** (default): The scheduler queries this source to detect which event dates have new or changed data, then runs downstream models only for those dates. Requires `backfill_start`. +- **`run_schedule`**: The scheduler runs downstream models on a fixed schedule regardless of data changes. Does not require `backfill_start`. + +### Source Partitions + +Sources can define partition filters to enable efficient querying: + +```jsonc +{ + "meta": { + "partitions": [ + { + "type": "event_dates", // filter by project event dates + "expr": "dt", // partition column expression + }, + { + "type": "gte", // comparison: "eq", "gt", "gte", "lt", "lte", "neq" + "expr": "created_date", + "value": "2024-01-01", + }, + ], + }, +} +``` + +### Optional Source Fields + +- `description`: Description of the source +- `freshness`: dbt freshness configuration object, or `null` to disable freshness checks for the entire source +- `loaded_at_field`: Column indicating data freshness +- `meta.portal_partition_columns`: Custom partition columns for the framework +- `meta.portal_source_count`: Custom source count configuration +- `meta.table_function`: Table function configuration +- `meta.where`: Static where clause applied whenever the source is queried +- Per-table `meta`: Table-level overrides for the same meta fields above +- Per-table `freshness`: Table-level freshness config or `null` to disable for a specific table +- Per-table `loaded_at_field`: Table-level override for the timestamp field used in freshness checks + +--- + +## Scheduling & ETL + +DJ uses an ETL scheduler (via Airflow) that determines **which event dates** need to be processed. This is driven by source configurations: + +1. **Sources with `event_count` ETL type**: The scheduler queries source tables to detect which dates have new or changed rows, then runs only those dates through the downstream model DAG. +2. **Sources with `run_schedule` ETL type**: The scheduler triggers downstream models on a fixed cron schedule. +3. **Models inherit their schedule** from their upstream sources — you don't configure scheduling on individual models. The framework traces the DAG back to the source to determine when to run. + +### How the Schedule Flows + +```text +Source (etl config) → stg model → int model(s) → mart model + ↑ schedule ↓ inherits schedule from source +``` + +When creating a new model: + +- If it reads from an **existing source**, the schedule is already handled. +- If it reads from a **new source**, you need to create a `.source.json` with the `meta.etl` configuration. +- The `backfill_start` date determines from when historical data will be processed. diff --git a/templates/skills/dj-convert-sql-to-model/_SKILL.md b/templates/skills/dj-convert-sql-to-model/_SKILL.md index d8f7882..64d1fce 100644 --- a/templates/skills/dj-convert-sql-to-model/_SKILL.md +++ b/templates/skills/dj-convert-sql-to-model/_SKILL.md @@ -3,7 +3,9 @@ name: dj-convert-sql-to-model description: >- Convert an existing SQL query into a DJ .model.json file. Use when the user has a working SQL query (often from a .draft.sql file) and wants to formalize - it as a DJ/dbt model. + it as a DJ/dbt model. Not for authoring a model from requirements (-> + dj-create-new-model), Python ETL (-> dj-create-python-model), or registering a + raw table as a source (-> dj-create-source). compatibility: DJ (Data JSON) Framework extension workspace with .dj/schemas/ and .agents/dj/AGENTS.md metadata: dj-skill: '1.0' @@ -15,7 +17,7 @@ Convert a raw SQL query into a **new** `.model.json` file following the DJ (Data **CRITICAL: This skill ONLY creates new `.model.json` files. NEVER modify, update, or overwrite existing `.model.json`, `.source.json`, `.sql`, or `.yml` files. If an upstream model or source already exists, reference it by name — do not edit it.** -**Reading order:** `.dj/schemas/model.type..schema.json` (follow `$ref`s) → `.agents/dj/AGENTS.md` **Model Types** section → this skill's SQL mapping rules. Always read the schema **before** writing any JSON. +**Reading order:** `.dj/schemas/model.type..schema.json` (follow `$ref`s) → `.agents/dj/reference/model-types.md` → this skill's SQL mapping rules. Always read the schema **before** writing any JSON. ## Output structure (mandatory) @@ -76,6 +78,8 @@ When the SQL has `GROUP BY`, add `"group_by": "dims"` to the model. Every `fct` - **Raw tables** → `"from": { "source": "__.
" }` (double underscore `__` between catalog and schema, dot `.` between schema and table) - **Existing dbt models** → `"from": { "model": "______" }` +**The raw table must be a registered source.** A `"from": { "source": … }` only resolves if a matching `.source.json` exists and has been synced into the manifest. If it doesn't, see **Missing source?** below — the source has to be created first (with exact Trino data types), not invented. + ## CTE handling If the SQL has `WITH` clauses, convert to the `ctes` array. CTEs must be ordered: a CTE can only reference CTEs defined before it. The main query becomes the model's primary `from` and `select`. Read `model.cte.schema.json` for the exact shape. @@ -91,18 +95,29 @@ Before writing CTEs, search for existing `.model.json` files in the project that 1. **Read the SQL query** provided by the user 2. **Always ask the user** for the new model's naming before creating anything: - - `group` — must be one of the groups defined in your project (e.g., `analytics`, `finops`, `marketing`, `engineering`, `sales`, `platform`) + - `group` — must be a **registered** group. dbt registers groups in any `.yml` under a top-level `groups:` key (commonly `models/_groups.yml` or `models/groups.yml`, or per-folder `group_*.yml`) and assigns models via the `group` config or `dbt_project.yml` `+group:` — there is no single fixed path, so scan the project's `.yml` files for `groups:` definitions (and sibling models' `group` values) for the valid set. If the requested group is not registered, ask the user to pick a registered one or register a new one via `dj-initialize` — do not invent a group. - `topic` (e.g., aws_cur, billing, salesforce) - `name` (e.g., daily_summary, accounts) Do NOT infer or reuse names from the SQL query — the user must confirm the name. + - If the workspace has more than one dbt project, also ask which project to target — do not silently pick a default. 3. **Determine the model type** from the SQL pattern table above 4. **Read the schema** at `.dj/schemas/model.type..schema.json` — follow all `$ref` links -5. **Read `.agents/dj/AGENTS.md`** Model Types section for the selected type's example +5. **Read `.agents/dj/reference/model-types.md`** for the selected type's example 6. **Scan existing `.model.json` files** in the project's `models/` directory — especially models of the same type — to learn naming conventions, CTE patterns, `select` structure, `group_by` usage, and other structural patterns. Use these as reference when creating the new model. Pay particular attention to models that use `ctes`, `join`, `where`, and `group_by` to understand how the project applies these features -7. **Verify upstream sources/models exist** by reading their JSON files (read-only — do NOT modify them) +7. **Verify upstream sources/models exist** by reading their JSON files (read-only — do NOT modify them). For a `stg_select_source` / `stg_union_sources` raw table, confirm a matching `.source.json` is registered; if it isn't, see **Missing source?** below before generating the model — do not reference an unregistered table 8. **Confirm the target file path does not already exist** — if it does, ask the user for a different name 9. **Create a new `.model.json`** file at the correct path — never overwrite an existing file -10. **Validate** the output against the schema +10. **Offer governance metadata (optional)** — offer to tag `owner` / `owner_slack` / `pii` / `classification` / `compliance` in `meta` (see `.agents/dj/reference/meta-and-governance.md`), matching keys sibling models already use. If the user skips, write nothing and do not re-ask. +11. **Validate** the output against the schema + +Layer folder placement is derived and enforced by DJ from `type` + `group` + `topic` + `name` — do not hand-pick folders (see AGENTS.md **Structural Governance**). + +## Missing source? Register it first + +A `stg_select_source` / `stg_union_sources` reads a raw `catalog.schema.table`, which resolves only if a `.source.json` registers it and the manifest has been synced. Before converting such a query, confirm the source exists by reading its `.source.json`. + +- **If the source is missing, do not invent its columns or data types.** Offer to register it first via the **`dj-create-source`** skill — it introspects the exact Trino types with `SHOW COLUMNS` — and confirm with the user before creating anything. The user can instead create it with the **`DJ: Create Source`** command (webview) if they prefer to do it by hand. +- **Then refresh the manifest.** A newly created `.source.json` isn't resolvable until DJ syncs it. Ask the user to run **`DJ: Sync to SQL and YML`**, then convert the SQL against the registered source. The agent cannot run VS Code commands itself, so this is a user action. ## File path convention @@ -207,13 +222,10 @@ GROUP BY customer_id, customer_name - **Never edit** generated `.sql` or `.yml` files — only create new `.model.json` - **Never modify upstream models or sources** — read them for column/reference info only - Use **JSONC format**: trailing commas allowed, preserve comments -- Source references use `__.
` format (double underscore, then dot) - Column types are `dim` or `fct`, default is `dim` — do NOT use `"dimension"` or `"measure"` - `expr` holds the SQL expression — do NOT use `"expression"` -- When using aggregates, always include `"group_by": "dims"` - Verify upstream columns exist before referencing them -- Prefer `"materialization": "incremental"` over legacy `"materialized": "incremental"` -- For all conventions, gotchas, and advanced features (CTEs, rollup, subqueries, Lightdash), follow the `dj-create-new-model` skill — it is the authoritative reference +- For all other conventions, gotchas, and advanced features (naming, source-reference format, `group_by`, materialization, CTEs, rollup, subqueries, Lightdash), follow the `dj-create-new-model` skill and `.agents/dj/AGENTS.md` **Important Conventions** — the authoritative references ## Gotchas diff --git a/templates/skills/dj-create-lightdash-yaml/_SKILL.md b/templates/skills/dj-create-lightdash-yaml/_SKILL.md index be5ec1e..cf4015e 100644 --- a/templates/skills/dj-create-lightdash-yaml/_SKILL.md +++ b/templates/skills/dj-create-lightdash-yaml/_SKILL.md @@ -22,7 +22,8 @@ The hard part is not the YAML shape (the bound schema guides that) -- it is usin the **exact** field IDs, honoring the model's **required filters**, and knowing that the **schema cannot validate references**. The detailed, copy-pasteable recipes live in `references/lightdash-as-code-authoring.md`; load it before -writing. +writing. For the Lightdash CLI executable, connection env vars, and the +restricted-projects guardrail, see `.agents/dj/reference/running-lightdash.md`. ## Prerequisites (the explore must already exist) diff --git a/templates/skills/dj-create-new-model/_SKILL.md b/templates/skills/dj-create-new-model/_SKILL.md index 54aa6bb..520d1ec 100644 --- a/templates/skills/dj-create-new-model/_SKILL.md +++ b/templates/skills/dj-create-new-model/_SKILL.md @@ -3,7 +3,10 @@ name: dj-create-new-model description: >- Create a DJ .model.json file for a new dbt model. Use when the user wants to create, add, or scaffold a dbt model -- staging, intermediate, or mart -- - including joins, CTEs, rollup, subqueries, or aggregations. + including joins, CTEs, rollup, subqueries, or aggregations. Not for Python ETL + (-> dj-create-python-model), converting existing SQL text (-> + dj-convert-sql-to-model), or registering a raw table as a source (-> + dj-create-source). compatibility: DJ (Data JSON) Framework extension workspace with .dj/schemas/ and .agents/dj/AGENTS.md metadata: dj-skill: '1.0' @@ -11,9 +14,23 @@ metadata: # Create DJ model -**Create** new **`.model.json`** files (and **`.source.json`** when adding sources). **Never** hand-edit auto-generated **`.sql`** / **`.yml`** — only the JSON sources of truth. +**Create** new **`.model.json`** files. **Never** hand-edit auto-generated **`.sql`** / **`.yml`** — only the JSON sources of truth. (Registering a raw table as a **`.source.json`** → **`dj-create-source`**; this skill reads sources but delegates their creation.) -**Reading order:** **`.dj/schemas/`** (type schema + **`$ref`s**) for exact shapes → **`.agents/dj/AGENTS.md`** **Model Types** (examples) → **Advanced** (short map: CTEs, rollup, shorthands, subqueries, materialization, `"dims"` — still defer to schemas) → **Important Conventions** **#6**–**#15**. +**Execution safety:** authoring is file-only — this skill does not run SQL or dbt. If you must inspect data to author a model, follow **Command & Query Execution Safety** in **`.agents/dj/AGENTS.md`**: read-only `SELECT` only, confirm the catalog/schema first, never touch production. + +**Reading order:** **`.dj/schemas/`** (type schema + `$ref`s) for exact field shapes → **`.agents/dj/reference/model-types.md`**: the **Model Types** worked example for your type, then its **Advanced** map (CTEs, rollup, shorthands, subqueries, materialization, `"dims"`) → this skill's **Important Conventions** + **Gotchas** for the framework rules the schema can't express. + +## When this skill applies + +Author **SQL** `.model.json` files (staging / intermediate / mart) that read from existing models and sources. + +**Out of scope** — delegate to a sibling skill: + +- Python ETL / ingestion (`.python.json`, `.python.py`) → `dj-create-python-model`. **This skill never writes Python models.** +- Formalizing an existing SQL query into a model → `dj-convert-sql-to-model`. +- Registering a raw Trino table as a source (`.source.json`) → `dj-create-source`. +- Reviewing / modernizing / refactoring an existing `.model.json` → `dj-review-and-refactor-model`. +- Authoring or editing Lightdash chart/dashboard YAML → `dj-create-lightdash-yaml` / `dj-edit-lightdash-yaml`. ## Model `type` (infer — do not ask the user) @@ -23,51 +40,39 @@ metadata: | **int** | one model → `int_select_model`; joins → `int_join_models`; unnest → `int_join_column`; lookback → `int_lookback_model`; union → `int_union_models` | | **mart** | one model → `mart_select_model`; joins → `mart_join_models` | -**Rollup:** optional **`from.model.rollup`** on **`int_select_model`** / **`int_join_models`** — coarser time **`interval`**, **`agg`/`aggs`** re-aggregated for the new grain (**`model.from.rollup.schema.json`**; **AGENTS** **Advanced**, **#9**–**#10**). +**Rollup:** optional **`from.rollup`** (a sibling of `from.model` — not nested under it) on **`int_select_model`** / **`int_join_models`** — a coarser time **`interval`** with **`agg`/`aggs`** re-aggregated for the new grain. See `model.from.rollup.schema.json` and `.agents/dj/reference/model-types.md` (**Advanced**). One clarifying question if source vs existing model is unclear. -## Inputs, path, workflow - -**Fields:** `type`, `group`, `name`; `topic` on all types in schema **except** `int_join_models` is not in **`required`** (still set in practice). Ask for missing names; mirror project patterns. +**BI / dashboard intent → mart.** When the user wants a model for BI, a Lightdash explore/dashboard, metrics, or "something to chart", default to a **mart** (`mart_select_model` for one upstream, `mart_join_models` for several) and follow [references/mart-lightdash-recipes.md](references/mart-lightdash-recipes.md). Marts are the layer that surfaces to Lightdash — don't expose staging/intermediate directly. -**Path:** `models////______.model.json` (`stg_*`→`staging`, etc.). +## Inputs & placement -**Locate the project first:** the dbt project may be nested, not the workspace root — find its `dbt_project.yml` and treat `models/...` paths as relative to that directory. `.dj/schemas/` lives at the DJ/workspace root, which can differ from the dbt project dir. If more than one dbt project exists, use the one named in `dj.dbtProjectNames`. +**Fields:** `type`, `group`, `topic`, `name`. All four are `required` in every type schema **except** `topic`, which `int_join_models` omits from `required` (still set it in practice). Ask for any the user didn't give; mirror the naming of existing models in the project. -**Checklist:** +- **`group`** must be a **registered** group. dbt registers groups in any `.yml` under a top-level `groups:` key (commonly `models/_groups.yml` or `models/groups.yml`, or per-folder `group_*.yml`) and assigns models via the `group` config or `dbt_project.yml` `+group:` — there is no single fixed path, so scan the project's `.yml` files for `groups:` definitions (and sibling models' `group` values) for the valid set. If the requested group isn't registered, ask the user to pick a registered one or register a new one (via `dj-initialize`); never invent a group or create an unregistered folder. +- **`topic`** and **`name`** are free identifiers (e.g. `topic: aws_cur`, `name: accounts_billing_daily`) — follow sibling models' conventions. -- [ ] `type` from table; read **`.dj/schemas/model.type..schema.json`**; if CTE / subquery / **`from.model.rollup`** / hooks / **`agg`** / materialization, also **`model.cte`**, **`model.subquery`**, **`model.from.rollup`**, **`model.sql_hooks`**, **`model.materialization`**, **`model.select.*.with.agg`** as needed -- [ ] **`.agents/dj/AGENTS.md`**: **Model Types** example; **Advanced** if CTE / rollup / shorthand / subquery -- [ ] Upstream columns from **`.model.json`** / **`.source.json`** (trace **`ctes`** if any) -- [ ] Write **JSONC**; validate against schema +**Path (framework-derived — never chosen):** `models////______.model.json`, where `` comes from the `type` prefix (`stg_*`→`staging`, `int_*`→`intermediate`, `mart_*`→`mart`). DJ writes and relocates the file from `type` + `group` + `topic` + `name`; do not place files in arbitrary folders or rename on disk (AGENTS.md **Structural Governance**). Rename a model by editing those JSON fields, never the filename. -## Conventions & gotchas +**Locate the project first:** the dbt project may be nested, not the workspace root — find its `dbt_project.yml` and treat `models/...` paths as relative to that directory (`.dj/schemas/` lives at the DJ/workspace root, which can differ). **If more than one dbt project exists, ask which to target — do not silently pick a default.** Prefer a project named in `dj.dbtProjectNames` only after confirming it is the one they mean. -- **type**: Type of the model - mart, staging, intermediate, source, etc. Determined from the decision tree above. -- **group**: Must be one of the groups defined in your project (e.g., `analytics`, `finops`, `marketing`, `engineering`, `sales`, `platform`). -- **topic**: Topic of the model - aws_cur, gcp_billing, salesforce, etc. -- **name**: Name of the model - accounts_billing_daily, opportunities_facts, etc. - -If the user hasn't provided group/topic/name, ask for them. Look at existing models in the project for naming conventions. +## Workflow -## File Naming and Path +1. **Determine `type`** from the decision table above (one clarifying question only if source-vs-model is ambiguous). +2. **Gather inputs** — `group`, `topic`, `name`, and type-specific fields. +3. **Read the schema** at `.dj/schemas/model.type..schema.json` for required/optional fields, following `$ref`s. For CTEs / subqueries / `from.rollup` / hooks / `agg` / materialization also read `model.cte`, `model.subquery`, `model.from.rollup`, `model.sql_hooks`, `model.materialization`, `model.select.*.with.agg` as needed. +4. **Read `.agents/dj/reference/model-types.md`** — the **Model Types** example for your `type`, plus its **Advanced** map if using CTEs / rollup / shorthands / subqueries. +5. **Verify upstream columns** by reading each `from` reference's `.model.json` / `.source.json` (trace `ctes` too). If a reference doesn't exist yet, see **Missing upstream?** below — do not invent columns. +6. **Write the `.model.json`** at the derived path in **JSONC** (comments and trailing commas allowed; preserve existing comments). +7. **Verify** via the editor's bound schema and DJ's on-save regeneration/diagnostics (Problems tab) — do not assume standalone validators (`jsonschema`, `pyyaml`, `pip`) are installed (see [references/mart-lightdash-recipes.md](references/mart-lightdash-recipes.md) §4). -- **Model name**: `______` (e.g., `int__analytics__billing__daily_summary`) -- **File name**: `.model.json` -- **Directory**: `models////` where layer is `staging`, `intermediate`, or `mart` +## Missing upstream? Build the chain first -The layer directory is derived from the type prefix: `stg_*` -> `staging`, `int_*` -> `intermediate`, `mart_*` -> `mart`. +A mart reads from intermediate/staging models; those read from staging/sources. Before authoring, confirm every `from` reference already exists by reading its `.model.json` / `.source.json` (or checking the manifest). -## Workflow - -- [ ] Step 1: Determine the model type from the user's request -- [ ] Step 2: Gather required inputs (group, topic, name, type-specific fields) -- [ ] Step 3: Read the JSON schema at `.dj/schemas/model.type..schema.json` to understand required and optional fields. Follow `$ref` links to sub-schemas as needed -- [ ] Step 4: Refer to the AGENTS.md "Model Types" section for the example structure of the selected type -- [ ] Step 5: Read upstream model/source files to verify available columns before writing `select` -- [ ] Step 6: Create the `.model.json` file at the correct path using JSONC format (comments and trailing commas allowed) -- [ ] Step 7: Verify validity via the editor's bound schema + DJ's on-save regeneration/diagnostics — do not assume standalone validators (`jsonschema`, `pyyaml`, `pip`) are installed (see [references/mart-lightdash-recipes.md](references/mart-lightdash-recipes.md) §4) +- **If an upstream layer is missing, do not invent its columns.** Offer to build the missing layers **upstream-first** — source → staging → intermediate → mart — and confirm scope with the user before creating anything. Build only the layers the requested model actually needs; skip a layer that adds no transformation (a mart can read a staging model directly when no intermediate logic is required). **A missing raw source (`.source.json`) is created via the `dj-create-source` skill** (or the `DJ: Create Source` webview) — it introspects the exact Trino data types with `SHOW COLUMNS`; never hand-author a source's `data_type`s. +- **Refresh the manifest before building the downstream.** A newly created `.source.json` or upstream `.model.json` is not resolvable by a downstream model until the dbt manifest registers it. After creating an upstream, ask the user to run **`DJ: Sync to SQL and YML`** — it regenerates the `.sql` / `.yml` and reparses the manifest on demand (running `dbt parse` only when a synced model is missing or the manifest is stale) — then author the downstream against it. `DJ: Refresh Projects` only re-reads project config and reloads the on-disk manifest; it does not run `dbt parse`. The agent cannot run VS Code commands itself, so this is a user action. ## Important Conventions @@ -77,42 +82,27 @@ The layer directory is derived from the type prefix: `stg_*` -> `staging`, `int_ - Column types are `dim` (dimension) or `fct` (fact/measure), default is `dim` - When using `agg`, always include `"group_by": "dims"` (or `[{ "type": "dims" }]`) - `"dims"` shorthand: `group_by: "dims"` groups by all dimension columns; join `on: "dims"` auto-joins on all shared dimension columns -- For joins, verify upstream columns exist by reading the upstream model's `.model.json` or source `.source.json` -- Rename models by changing JSON fields (type/group/topic/name), never by renaming the file on disk -- Prefer `"materialization": "incremental"` over legacy `"materialized": "incremental"`. For full control, use the structured form: `{ "type": "incremental", "format"?: "iceberg"|"delta_lake"|"hive", "partitions"?: [...], "strategy"?: {...} }`. See `model.materialization.schema.json` -- **Incremental strategies** (`materialization.strategy.type`): `append` (insert-only, no dedup), `delete+insert` (partition-safe upsert; `unique_key` auto-derived from partitions), `merge` (row-level upsert on `unique_key` **requires Iceberg format in dbt-trino**), `overwrite_existing_partitions` (**requires a custom dbt macro in the consumer project**; if not available, use `delete+insert` instead), `dj_iceberg_partition_overwrite` (**shipped by DJ** via `macros/_ext_/strategies.sql`; **requires Iceberg format** on Delta Lake / Hive use `delete+insert` instead). If omitted, the extension default applies (`dj.materialization.defaultIncrementalStrategy`). See `model.incremental_strategy.schema.json` -- `int_select_model` and `int_join_models` support `from.rollup` for time-grain re-aggregation without needing a separate `int_rollup_model`. See AGENTS.md "Model Types" and `model.from.rollup.schema.json` -- Use the `ctes` array for inline CTEs on `int_select_model`, `int_join_models`, `int_union_models`, `mart_select_model`, `mart_join_models`. CTE bulk selects support `exclude`/`include` filters. See AGENTS.md "Inline CTEs" and `model.cte.schema.json` -- Inside a CTE, `from.rollup` is supported on `from.model` and `from.cte` (not `from.source`, not `from.union`). The framework rewrites the CTE's `datetime`, drops finer-grain partitions, wraps fct columns with their suffix-agg, and synthesizes `GROUP BY `. See `docs/models/CTE_PATTERNS.md` and `model.from.rollup.schema.json` -- WHERE, HAVING, and JOIN ON conditions support inline subqueries via the `subquery` key. See AGENTS.md "Inline Subqueries" and `model.subquery.schema.json` +- **Materialization & incremental strategies** — string `"incremental"` / `"ephemeral"` or the structured form; five incremental strategies with storage-format constraints; the Iceberg-vs-Delta/Hive partitioning-keyword switch. See `.agents/dj/reference/materialization.md`, `model.materialization.schema.json`, and `model.incremental_strategy.schema.json` +- **Inline CTEs** — use the `ctes` array on `int_select_model`, `int_join_models`, `int_union_models`, `mart_select_model`, `mart_join_models` (bulk selects support `exclude`/`include`). For the CTE-vs-new-model decision see [references/cte-authoring.md](references/cte-authoring.md); for the mechanics, gotchas, and `datetime` / `portal_partition_*` / `portal_source_count` / `from.rollup` behavior inside CTEs see `.agents/dj/reference/ctes-and-subqueries.md` and `model.cte.schema.json` +- `int_select_model` and `int_join_models` support `from.rollup` for time-grain re-aggregation without a separate `int_rollup_model`. See `.agents/dj/reference/model-types.md` (**Advanced**) and `model.from.rollup.schema.json` +- WHERE, HAVING, and JOIN ON conditions support inline subqueries via the `subquery` key. See `.agents/dj/reference/ctes-and-subqueries.md` and `model.subquery.schema.json` - Source freshness can be disabled with `"freshness": null` at source or table level -- Free-form `meta` keys are allowed at both model and column level on `.model.json` (e.g., `owner`, `pii`, `compliance`). See AGENTS.md "Custom Meta" section, `model.meta.schema.json`, `column.meta.schema.json` +- Free-form `meta` keys are allowed at both model and column level on `.model.json` (e.g., `owner`, `pii`, `compliance`). See `.agents/dj/reference/meta-and-governance.md`, `model.meta.schema.json`, `column.meta.schema.json` +- **Governance metadata is optional — offer it, never require it.** After the model shape is settled, offer to tag governance keys (`owner`, `owner_slack`, `pii`, `classification`, `compliance`, `freshness_sla`) per `.agents/dj/reference/meta-and-governance.md` (**Governance metadata conventions**). First scan sibling models and offer the keys the project actually uses. If the user skips, write nothing (no placeholders) and do not re-ask. Teams that want these mandatory enforce it themselves. - For Lightdash column config, author `select[i].lightdash.dimension`, `.metrics`, `.metrics_merge`, `.case_sensitive` — not `meta.dimension` etc. The framework surfaces a Warning-severity diagnostic in the Problems tab if authored under `meta` - **Marts that back a Lightdash explore/dashboard** — for a default time window (`lightdash.table.required_filters`), a summable metric on a `mart_select_model` passthrough, the right framework-column exclude flag, and how validation works, see [references/mart-lightdash-recipes.md](references/mart-lightdash-recipes.md) ## Gotchas +CTE / framework-column / partition / `from.rollup` gotchas are in `.agents/dj/reference/ctes-and-subqueries.md`; materialization / storage gotchas are in `.agents/dj/reference/materialization.md`. The high-frequency ones: + - Subquery `column` is required for all operators except `exists`/`not_exists` - CTEs must be ordered: a CTE can only reference CTEs defined **before** it in the `ctes` array -- **CTE `group_by` with computed columns**: bare string aliases (e.g., `["month"]`) for columns defined with `expr` (e.g., `DATE_TRUNC(...)`) pass schema validation but fail at Trino with `COLUMN_NOT_FOUND`. Use `"group_by": "dims"` or `[{ "expr": "..." }]` instead -- **CTE column type inheritance**: plain string selects in CTEs inherit `dim`/`fct` type from the upstream model or CTE -- no need to redeclare column types. This means `dims_from_cte` and `fcts_from_cte` correctly filter by type in CTE-to-CTE chains -- **CTE bulk select filtering**: `all_from_cte`, `dims_from_cte`, `fcts_from_cte` support `exclude` and `include` arrays to filter columns -- **`lightdash.metrics` / `lightdash.metrics_merge` on a CTE `select` item is an error** — declare those on the main-model `select` only. Keep the pre-aggregated column in the CTE and re-aggregate it in the main model (`agg` / `aggs` / aggregate `expr`). `lightdash.dimension` on CTE selects still propagates. - **Un-aggregated `fct` + main-model `group_by` is an error** — every `fct` in the main `select` must set `agg` / `aggs`, wrap an aggregate in `expr` (`sum(x)`, `avg(x)`, `merge(cast(x as hyperloglog))`, `cast(tdigest_agg(x) as varbinary)`, `any_value(x)`, …), or `exclude_from_group_by: true`. Applies to scalar selects, CTE scalar refs, and bulk `all_from_cte` / `fcts_from_cte` carriers. -- **`portal_source_count` auto-injects in CTEs whose `from` is a model or another CTE** — don't duplicate it in the CTE `select`; it's appended automatically from the upstream (aggregated with `count` when the CTE has a `group_by`). Set `override_suffix_agg: true` only when you need a differently-aggregated variant alongside the audit column. -- **`datetime` and `portal_partition_*` auto-inject in CTEs whose `from` is a model or another CTE** — mirrors the main-model behavior. If the upstream (manifest schema for `{ model }`, the in-memory registry for `{ cte }`) has them and the CTE's select (or `dims_from_model.include`) did not list them, they're appended automatically. An explicit `{ "name": "datetime", "interval": X }` drives partition exclusion: `day` drops hourly, `month` drops hourly+daily, `year` drops all three. Auto-inject is still skipped for source and union shapes. -- **CTE exclude/include flags mirror the main-model flags and inherit from the model** — a CTE accepts `exclude_date_filter`, `exclude_daily_filter`, `exclude_datetime`, `exclude_framework_artifacts`, `exclude_portal_partition_columns`, `exclude_portal_source_count`, and `include_full_month` with the same semantics as their main-model counterparts. Resolution is uniform: **CTE override > model value > false**. Set a flag on the model to apply it to every CTE, on a single CTE to override only that CTE, or set `false` on a CTE to opt back in when the model excluded. `exclude_datetime` and `exclude_portal_partition_columns` are orthogonal — set both for pure-dim/lookup shapes; `exclude_datetime` is mutually exclusive with `from.rollup` at the same scope (model OR CTE) and the validator errors when both are set together. -- **Rolling up a CTE that sources from another CTE that excludes datetime is rejected** — the upstream must produce a datetime column for the rollup to truncate. Either drop `exclude_datetime` on the upstream CTE, or have the upstream itself declare `from.rollup`. -- **Framework columns flow through CTE chains by default** — `datetime`, `portal_partition_*`, and `portal_source_count` propagate through every `from: { cte }` hop (and into a main model with `from: { cte }`) by inheriting from the upstream registry. List them in `select` only for a transformed alias, or opt out per CTE / per model with the standard exclude flags. When the main model uses an `incremental` partition-overwrite strategy, the auto-flowed `portal_partition_*` typically satisfies the partition-column requirement; if you intentionally exclude them through a chain, set `materialization.partitions: ["datetime"]` on the main model. Wrapper SELECTs that reference an already-rolled-up `datetime` do not redundantly re-emit `date_trunc(, datetime)`. -- **`exclude_framework_artifacts` is the combined-flag shortcut** — a single string-enum (`"all"` | `"columns"`) on the model or CTE that bundles `exclude_datetime` + `exclude_portal_partition_columns` + `exclude_portal_source_count` (`"columns"`), with `"all"` additionally implying `exclude_date_filter`. Individual flags at the same scope override per-column (e.g. `"exclude_framework_artifacts": "all"` + `"exclude_portal_source_count": false` keeps that one column). Resolution chain: CTE individual > CTE combined > model individual > model combined > false. Mutually exclusive with `from.rollup` when the resolved value implies excluding `datetime`. -- **Dead outer-layer warning** — if the main `select` is a single `all_from_cte` / `dims_from_cte` passthrough of one CTE with identical `group_by` and no extra filter / limit / projection, drop the wrapper or add new work to it. See `docs/models/CTE_PATTERNS.md`. -- `from.rollup` requires the upstream model to have a select column with an `"interval"` field (e.g., `{ "name": "datetime", "interval": "day" }`) - Cross joins have no `on` property -- do not include `on: {}` or `on: null` - Subquery `from` can reference a model, source, or CTE -- use `{ "cte": "name" }` for CTEs defined in the same model - `topic` is not in `required` for `int_join_models` (it is for all other types) -- still set it in practice - `mart_select_model` and `int_union_models` do not support `agg`/`aggs` in select items -- use only passthrough or expression columns -- `materialization` structured form allows `"format": "iceberg"` for Iceberg storage -- partitioning keyword changes automatically based on format -- Both `materialized` (legacy) and `materialization` (preferred) are accepted; when both are present, `materialization` takes precedence - **`meta` is free-form but has a few reserved keys**. Column `type`, `dimension`, `metrics`, `case_sensitive`, `origin` and model `metrics`, `local_tags`, `case_sensitive`, and any key on `lightdash.table` are framework-owned — author via the structured sibling field (`type`, `lightdash.*`, `tags: [{ type: "local", tag }]`, etc.). Collisions trigger Warning diagnostics in the Problems tab ## References diff --git a/templates/skills/dj-create-new-model/references/cte-authoring.md b/templates/skills/dj-create-new-model/references/cte-authoring.md new file mode 100644 index 0000000..af23738 --- /dev/null +++ b/templates/skills/dj-create-new-model/references/cte-authoring.md @@ -0,0 +1,29 @@ +# CTE authoring + +Load on demand when a model uses the `ctes` array (`int_select_model`, +`int_join_models`, `int_union_models`, `mart_select_model`, `mart_join_models`) +and you need to decide whether a CTE is the right tool. For the CTE mechanics, +authoring rules, and gotchas — ordering, bulk selects, column-type inheritance, +`group_by` on computed columns, framework-column auto-injection, and `from.rollup` +inside CTEs — see `.agents/dj/reference/ctes-and-subqueries.md`. + +## CTE or a new model? + +A CTE is **non-materialized** — it is a transient, in-memory query stage that +exists only inside the one model that declares it and is recomputed every time +that model runs. A new model is a **named, reusable** node in the DAG (a view or +an incremental/ephemeral table) that other downstream models can select from. + +**Reach for a CTE when:** + +- Pre-aggregating an upstream model before a join so the join key space shrinks. +- Normalizing column shapes (types, names, grouping) across several upstreams before a union. +- Factoring a repeated sub-expression out of a complex `select` list. + +The work is local to this model and nothing else needs to reuse it. + +**Prefer a new model when:** + +- The intermediate result should be reusable by other downstream models — a CTE would force each consumer to recompute it. +- The work is heavy (window functions, wide cross-joins, multi-CTE chains, unpartitioned full-history scans) and should materialize once rather than per consumer query. Use an `int_select_model` / `int_rollup_model`, or an `incremental` materialization. +- You just want an additional aggregation on top of another model's output — that belongs in a downstream model, not an inline CTE. diff --git a/templates/skills/dj-create-python-model/_SKILL.md b/templates/skills/dj-create-python-model/_SKILL.md index 3351c50..b8d86a9 100644 --- a/templates/skills/dj-create-python-model/_SKILL.md +++ b/templates/skills/dj-create-python-model/_SKILL.md @@ -3,7 +3,9 @@ name: dj-create-python-model description: >- Create a DJ .python.json file for a new Python ETL model. Use when the user wants to create a Python model, ETL pipeline, data ingestion, API fetch, - CSV import, or any pre-dbt Python data processing task. + CSV import, or any pre-dbt Python data processing task. Not for dbt SQL models + (-> dj-create-new-model) or registering a raw table as a source (-> + dj-create-source). compatibility: DJ (Data JSON) Framework extension workspace with .dj/schemas/ metadata: dj-skill: '1.0' @@ -24,7 +26,7 @@ Use this skill when the user mentions: python model, ETL, data ingestion, API fe **Out of scope** — delegate to sibling skills: - SQL `.model.json` files (staging/intermediate/mart) → `dj-create-new-model` -- `.source.json` files → `dj-create-new-model` +- `.source.json` files (registering a raw table as a source) → `dj-create-source` - Lightdash YAML → `dj-edit-lightdash-yaml` - Refactoring existing models → `dj-review-and-refactor-model` @@ -36,12 +38,12 @@ Ask the user the following questions in order. Batch related questions together Ask for: -| Field | Rule | Example | -|-------|------|---------| -| **name** | `^[a-z][a-z0-9_]*$` | `backstage_catalogs` | -| **group** | One of: `ml`, `etl`, `analytics`, `others` (or project-configured groups) | `etl` | -| **topic** | `^[a-z][a-z0-9_]*$` | `api_data` | -| **description** | Free text (optional) | `Fetches catalog data from Backstage API` | +| Field | Rule | Example | +| --------------- | ------------------------------------------------------------------------- | ----------------------------------------- | +| **name** | `^[a-z][a-z0-9_]*$` | `backstage_catalogs` | +| **group** | One of: `ml`, `etl`, `analytics`, `others` (or project-configured groups) | `etl` | +| **topic** | `^[a-z][a-z0-9_]*$` | `api_data` | +| **description** | Free text (optional) | `Fetches catalog data from Backstage API` | ### Step 2: DAG assignment @@ -51,13 +53,13 @@ Ask which Airflow DAG(s) to attach the model to. Look at existing DAGs in the pr Ask what kind of data source the model will extract from: -| Source type | Typical packages | Extract pattern | -|-------------|-----------------|-----------------| -| **REST API** | `requests` | HTTP GET/POST with pagination | -| **Database / Trino** | `trino` | SQL query via Trino client | -| **CSV / file** | `pandas` | `pd.read_csv()` from local or S3 | -| **S3 objects** | `boto3` | List/download from S3 bucket | -| **Custom** | User-specified | User provides extract logic | +| Source type | Typical packages | Extract pattern | +| -------------------- | ---------------- | -------------------------------- | +| **REST API** | `requests` | HTTP GET/POST with pagination | +| **Database / Trino** | `trino` | SQL query via Trino client | +| **CSV / file** | `pandas` | `pd.read_csv()` from local or S3 | +| **S3 objects** | `boto3` | List/download from S3 bucket | +| **Custom** | User-specified | User provides extract logic | ### Step 4: Transformation needs — SQL-first decision @@ -65,17 +67,17 @@ Ask what transformations are needed. Then apply this decision tree: **Can the transformation be expressed as SQL?** -| If YES (use Trino SQL) | If NO (use pandas DataFrame) | -|------------------------|------------------------------| -| Filtering / WHERE clauses | Nested JSON flattening (dicts/lists) | -| Column renaming / aliasing | API response parsing / pagination | -| Type casting (CAST) | ML preprocessing (scikit-learn, etc.) | -| Deduplication (ROW_NUMBER) | Complex string parsing not in SQL | -| Aggregation (GROUP BY) | External service calls mid-transform | -| Joins with other Trino tables | Binary/image data processing | -| Date partitioning | | -| Window functions | | -| CASE expressions | | +| If YES (use Trino SQL) | If NO (use pandas DataFrame) | +| ----------------------------- | ------------------------------------- | +| Filtering / WHERE clauses | Nested JSON flattening (dicts/lists) | +| Column renaming / aliasing | API response parsing / pagination | +| Type casting (CAST) | ML preprocessing (scikit-learn, etc.) | +| Deduplication (ROW_NUMBER) | Complex string parsing not in SQL | +| Aggregation (GROUP BY) | External service calls mid-transform | +| Joins with other Trino tables | Binary/image data processing | +| Date partitioning | | +| Window functions | | +| CASE expressions | | **Always ask:** "Can this transformation be done in Trino SQL?" before defaulting to pandas. SQL transformations are preferred because: @@ -88,14 +90,16 @@ Ask what transformations are needed. Then apply this decision tree: Present the defaults and ask if the user wants to override any: -| Field | Default | Override? | -|-------|---------|-----------| -| `output.database` | `glue_development` | Yes | -| `output.schema` | `opus_python_source` | Yes | -| `output_type` | `iceberg` | Yes (alternative: `s3`) | -| `output.write_mode` | `overwrite_partitions` | Yes | -| `output.partition_by` | `["portal_partition_daily"]` | Yes | -| `namespace` | `python` | Yes | +| Field | Default | Override? | +| --------------------- | ---------------------------- | ----------------------- | +| `output.database` | `glue_development` | Yes | +| `output.schema` | `opus_python_source` | Yes | +| `output_type` | `iceberg` | Yes (alternative: `s3`) | +| `output.write_mode` | `overwrite_partitions` | Yes | +| `output.partition_by` | `["portal_partition_daily"]` | Yes | +| `namespace` | `python` | Yes | + +**Confirm the write target before proceeding.** The output database/schema is where this model _writes_ data, and `overwrite_partitions` is destructive (it replaces existing partitions). Do not assume an environment: if the workspace has more than one candidate database/catalog, or the user's request implies anything other than the `glue_development` default, **state the resolved `output.database` / `output.schema` / `write_mode` and get explicit confirmation** before writing the `.python.json`. Never point a new model at a production database without the user confirming it. ### Step 6: Dependencies & optional fields @@ -107,6 +111,8 @@ Ask about: - **owner** — team or individual - **enable_notebook** — generate companion `.python.ipynb` (default: `true`) +**Governance metadata is optional — offer, never require.** Alongside `owner`, offer to tag `owner_slack`, `pii`, `classification`, and `compliance` in `meta` (see `.agents/dj/reference/meta-and-governance.md`), matching the keys sibling models in the project already use. If the user skips, write nothing and do not re-ask. + ## Optimization guidance **Proactively suggest these optimizations** to the user while building the model. Do not wait for the user to ask — surface relevant advice based on the model's source type, transformation needs, and data volume. @@ -129,11 +135,11 @@ Ask about: ### Write mode selection -| Write mode | Use when | -|------------|----------| +| Write mode | Use when | +| ---------------------- | ---------------------------------------------------------------------- | | `overwrite_partitions` | Idempotent daily loads — rerun-safe, replaces only affected partitions | -| `append` | Event streams or append-only logs — never overwrites existing data | -| `overwrite` | Full table refresh — replaces entire table on each run | +| `append` | Event streams or append-only logs — never overwrites existing data | +| `overwrite` | Full table refresh — replaces entire table on each run | ### Resource management @@ -188,17 +194,17 @@ Use this when data comes from an external source (API, CSV, S3) but all transfor The extension auto-generates `dags/python_models/_trino_io.py` with these functions. **Do not edit this file** — it is always overwritten by the extension. -| Function | Use when | -|----------|----------| -| `execute_trino(sql)` | DDL/DML with no return value (CREATE, DROP, INSERT, etc.) | -| `execute_trino(sql, return_result=True)` | Scalar reads (SELECT COUNT(*), SELECT MAX(...), etc.) — returns first column of first row | -| `append(table_fqn, insert_sql)` | Append-only INSERT — runs the user's INSERT SQL as-is | -| `overwrite_partition(table_fqn, partition_col, partition_value, *, insert_sql=...)` | Idempotent daily loads — DELETE one partition, then INSERT (pass a full `INSERT INTO ...` statement) | +| Function | Use when | +| ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | +| `execute_trino(sql)` | DDL/DML with no return value (CREATE, DROP, INSERT, etc.) | +| `execute_trino(sql, return_result=True)` | Scalar reads (SELECT COUNT(\*), SELECT MAX(...), etc.) — returns first column of first row | +| `append(table_fqn, insert_sql)` | Append-only INSERT — runs the user's INSERT SQL as-is | +| `overwrite_partition(table_fqn, partition_col, partition_value, *, insert_sql=...)` | Idempotent daily loads — DELETE one partition, then INSERT (pass a full `INSERT INTO ...` statement) | | `overwrite_partition(table_fqn, partition_col, partition_value, source_query, *, columns=...)` | Same as above, but DJ builds `INSERT INTO ... (cols) ` from a SELECT / WITH ... SELECT | -| `overwrite(table_fqn, overwrite_filter, *, insert_sql=...)` | Overwrite rows matching any SQL filter (use `"true"` for full-table refresh) | -| `merge(table_fqn, merge_sql)` | Upsert — runs the user's full MERGE INTO ... USING ... SQL | -| `delete(table_fqn, where_clause)` | Standalone conditional DELETE | -| `update(table_fqn, set_clause, where_clause)` | Standalone conditional UPDATE | +| `overwrite(table_fqn, overwrite_filter, *, insert_sql=...)` | Overwrite rows matching any SQL filter (use `"true"` for full-table refresh) | +| `merge(table_fqn, merge_sql)` | Upsert — runs the user's full MERGE INTO ... USING ... SQL | +| `delete(table_fqn, where_clause)` | Standalone conditional DELETE | +| `update(table_fqn, set_clause, where_clause)` | Standalone conditional UPDATE | `insert_sql` is keyword-only. Prefer `insert_sql=...` when you already have a full INSERT; use `source_query=` (+ optional `columns=`) when you want the helper to assemble the INSERT. @@ -288,6 +294,8 @@ glue_development__opus_python_source. (Double underscore separates catalog from schema, dot separates schema from table.) +The output table must be **registered as a `.source.json`** before a downstream model can read it — use the **`dj-create-source`** skill (it introspects the Iceberg table's columns with `SHOW COLUMNS`) or the `DJ: Create Source` webview, then sync. + ## Conventions and gotchas - **SQL-first** — prefer Trino SQL for all transformations that SQL can express. Use DataFrames only for external data ingestion and Python-only logic diff --git a/templates/skills/dj-create-source/_SKILL.md b/templates/skills/dj-create-source/_SKILL.md new file mode 100644 index 0000000..e5b87c8 --- /dev/null +++ b/templates/skills/dj-create-source/_SKILL.md @@ -0,0 +1,128 @@ +--- +name: dj-create-source +description: >- + Register a raw Trino table as a DJ .source.json so models can read it via + from.source. Use when a model needs a raw catalog.schema.table that is not + defined as a source yet, or to add a table/columns to an existing source. Not + for authoring models (-> dj-create-new-model), converting SQL text (-> + dj-convert-sql-to-model), or Python ETL (-> dj-create-python-model). +compatibility: DJ (Data JSON) Framework extension workspace with .dj/schemas/ and .agents/dj/AGENTS.md +metadata: + dj-skill: '1.0' +--- + +# Create DJ source + +Register a raw Trino table as a DJ **`.source.json`** so `.model.json` files can read it +via `"from": { "source": "__.
" }`. Use this when a model needs a raw +`catalog.schema.table` that has no source definition yet, or to add a new table (or new columns) +to an existing source file. + +**The one hard rule: never guess column data types.** A source's `columns[].data_type` must be the +**exact** Trino type (`varchar`, `timestamp(3)`, `decimal(38,9)`, `array(varchar)`, `row(...)`, …). +Get them from a live `SHOW COLUMNS` introspection — do not infer them from the SQL, the column name, +or a sample value. Everything else in a source file is simple; this is the part that must be exact. + +**Never** hand-edit the generated **`__.yml`** — DJ regenerates it from the +`.source.json` on sync. You author only the `.source.json`. + +**Reading order:** `.dj/schemas/source.schema.json` (follow `$ref`s — `source.table`, `column.name`, +`column.type`) for the exact shape → an existing `*.source.json` in the project (best example of local +conventions) → this skill. + +## Workflow + +1. **Resolve the raw table identity.** Determine the Trino **catalog**, **schema**, and **table** to + register — do not blindly pick or invent one: + + - **Prefer context.** When this skill is triggered from a SQL query (`FROM catalog.schema.table`) + or the user already names the table, use that directly — don't re-ask what you already know. + - **Browse only when it's unknown or ambiguous.** Discover it the same way the webview does, with + read-only introspection — `SHOW CATALOGS` → `SHOW SCHEMAS FROM ""` → + `SHOW TABLES FROM "".""` — and let the **user pick**. Never guess a name. + - **Confirm before writing** whenever there's any doubt about which table the user means. + + Also locate the dbt project first — it may be nested, not the workspace root: find its + `dbt_project.yml` and treat `models/...` as relative to that directory. **If more than one dbt + project exists, ask which to target — do not silently pick a default.** + +2. **Introspect columns from Trino (mandatory, read-only).** Run: + + ```sql + SHOW COLUMNS FROM "".""."
" + ``` + + through the Trino access described in `.agents/dj/AGENTS.md` **Command & Query Execution Safety** + (read-only metadata query; confirm the catalog/schema first; never target production writes) — see + `.agents/dj/reference/running-trino.md` for resolving the CLI and connection. Use + the returned **Column** and **Type** values verbatim for `name` and `data_type`. If the table does + not exist or you cannot reach Trino, stop and tell the user — do not fabricate a source. + +3. **Derive the file path (never chosen).** The source **name** is `__` where + `database` is the Trino catalog and `schema` is the Trino schema. The file lives at: + + ```text + models/sources//__.source.json + ``` + + `database` and `schema` must match `^([a-z]|[0-9]|_)+$` (lowercase alphanumeric + underscore). + +4. **Merge or create.** + + - **File exists** → read it as **JSONC** (preserve comments), append the new table to `tables[]`, + and keep `tables` sorted by `name`. **If a table with that `name` already exists, do not + duplicate it** — the source already covers it; add only missing columns if that was the intent. + - **File does not exist** → create it with top-level `database`, `schema`, and `tables[]`. + +5. **Write the table.** Each table is `{ "name": "
", "columns": [ … ] }`; each column is + `{ "name": "", "data_type": "", "description": "" }`. Set `"type": "dim"` + or `"type": "fct"` only when the role is known (default is `dim` at model level; sources usually + omit it). Do not add fields that are not in `source.schema.json`. + +6. **Refresh the manifest.** A new `.source.json` is not resolvable by a downstream model until the + dbt manifest registers it. After writing, ask the user to run **`DJ: Sync to SQL and YML`** (it + regenerates the `.yml` and reparses the manifest on demand) before any model references the source. + The agent cannot run VS Code commands itself, so this is a user action. + +## Manual alternative (the extension does this deterministically) + +The DJ extension ships the same flow as a GUI. The user can run the **`DJ: Create Source`** command, +pick **project → catalog → schema → table**, and the extension introspects the columns and writes the +exact same `.source.json` (merging into an existing file, sorted, dup-safe). Offer this path when the +Trino CLI is not available in the terminal or the user prefers to do it by hand — the result is +identical, so a model authored against it is safe either way. + +## Source file shape + +```jsonc +{ + "database": "gsheets_opus", // Trino catalog (lowercase, matches folder + name prefix) + "schema": "default", // Trino schema + // "freshness": null, // optional — null disables dbt freshness checks + "tables": [ + { + "name": "savings_tracker", + "columns": [ + { "name": "fiscal_year", "data_type": "varchar", "description": "" }, + { "name": "actual_amount", "data_type": "double", "description": "" }, + ], + }, + ], +} +``` + +- **Required:** `database`, `schema`, `tables[]`; each table needs `name` + `columns[]`; each column + needs `name` + `data_type`. +- **Optional:** source/table `description`, `freshness` (or `null` to disable), `loaded_at_field`, + `meta`; column `type` (`dim`/`fct`), `lightdash`, `meta`. + +## Gotchas + +- **`data_type` is never guessed** — it comes from `SHOW COLUMNS`. A wrong type surfaces later as a + cast/compile error in the downstream model, far from the cause. +- **One source file per `__`** — every table from the same catalog+schema lives in + the same `.source.json`; add tables to it rather than creating parallel files. +- **Do not duplicate an existing table** — merge into `tables[]`; a repeated `name` is invalid. +- **Never edit the generated `__.yml`** — it is regenerated from the JSON on sync. +- **A model can't reference the source until it's synced** — remind the user to run + `DJ: Sync to SQL and YML` before authoring the downstream model. diff --git a/templates/skills/dj-edit-lightdash-yaml/_SKILL.md b/templates/skills/dj-edit-lightdash-yaml/_SKILL.md index 4476b2c..648dd88 100644 --- a/templates/skills/dj-edit-lightdash-yaml/_SKILL.md +++ b/templates/skills/dj-edit-lightdash-yaml/_SKILL.md @@ -19,7 +19,8 @@ download step, or previously authored) so they can be re-uploaded. Don't tell th user to run the upload CLI directly: either point them to the `DJ: Lightdash - Dashboards as Code` webview (which keeps auth and YAML schema bindings in sync), or **offer to run the command yourself** after confirming the -target project. +target project. For the CLI executable, connection env vars, and the +restricted-projects guardrail, see `.agents/dj/reference/running-lightdash.md`. ## When this skill applies @@ -71,22 +72,27 @@ target project. directly.** Either point them to the extension's webview (so auth, working directory, and YAML schema bindings stay in sync), or offer to run the command yourself after confirming the target project. +- **Confirm the target project before uploading; never assume prod.** State the + project UUID/name you will upload to and get explicit confirmation. If it is a + production project or listed in `dj.lightdash.restrictedProjects`, call that out + and confirm again — the restriction only guards the webview Upload tab, not a + direct `lightdash upload`. Prefer a preview project unless the user chose prod. - **Never edit `.sql` or `.yml` files under `models/`** as part of this skill — those belong to DJ's JSON-sync flow, not Dashboards-as-Code. ## Common edits -| Intent | Where in the YAML | -| --- | --- | -| Change a chart's row limit | `metricQuery.limit` | -| Change a date window (e.g. last 30 -> 90 days) | `metricQuery.filters.dimensions.and[].values` (keep the rule `id` UUID) | -| Add/remove a chart filter | `metricQuery.filters` (preserve `id` UUIDs on existing filter rules) | -| Re-order chart sorts | `metricQuery.sorts` (each entry has `fieldId` and `descending`) | -| Add a custom table calc | `metricQuery.tableCalculations` | -| Toggle column visibility | `tableConfig.columnOrder` and the chart's `chartConfig` | -| Add a tile to a dashboard | append to `tiles`, set `type`, `properties`, and a non-overlapping `x/y/w/h` | -| Add a dashboard-level filter | `filters.dimensions` / `filters.metrics` / `filters.tableCalculations` | -| Rename what's shown in the UI | `name`, `description`, axis `label` fields, dimension `label` overrides — never `slug` | +| Intent | Where in the YAML | +| ---------------------------------------------- | -------------------------------------------------------------------------------------- | +| Change a chart's row limit | `metricQuery.limit` | +| Change a date window (e.g. last 30 -> 90 days) | `metricQuery.filters.dimensions.and[].values` (keep the rule `id` UUID) | +| Add/remove a chart filter | `metricQuery.filters` (preserve `id` UUIDs on existing filter rules) | +| Re-order chart sorts | `metricQuery.sorts` (each entry has `fieldId` and `descending`) | +| Add a custom table calc | `metricQuery.tableCalculations` | +| Toggle column visibility | `tableConfig.columnOrder` and the chart's `chartConfig` | +| Add a tile to a dashboard | append to `tiles`, set `type`, `properties`, and a non-overlapping `x/y/w/h` | +| Add a dashboard-level filter | `filters.dimensions` / `filters.metrics` / `filters.tableCalculations` | +| Rename what's shown in the UI | `name`, `description`, axis `label` fields, dimension `label` overrides — never `slug` | ## Gotchas diff --git a/templates/skills/dj-git-workflow/_SKILL.md b/templates/skills/dj-git-workflow/_SKILL.md new file mode 100644 index 0000000..28b42aa --- /dev/null +++ b/templates/skills/dj-git-workflow/_SKILL.md @@ -0,0 +1,24 @@ +--- +name: dj-git-workflow +description: >- + Git hygiene for a DJ (Data JSON) Framework dbt project — what to commit (the + .model.json / .source.json sources together with their generated .sql / .yml), + what to ignore (.dj/), branching, and commit conventions. Use when the user + asks to commit, stage, branch, or check in their DJ models, or asks what should + be committed. For merge conflicts between JSON and generated files, use + dj-resolve-merge-conflicts instead. +compatibility: DJ (Data JSON) Framework workspace under git with .agents/dj/AGENTS.md +metadata: + dj-skill: '1.0' +--- + +# DJ git workflow + +Commit DJ work consistently. The full guidance — coupling JSON sources with generated output, ignoring DJ state, finding changed models, and the guardrails — lives in `.agents/dj/reference/git-workflow.md`. Read it before staging. + +Essentials: + +1. **Commit the pair together.** A `.model.json` / `.source.json` and its generated `.sql` / `.yml` are one unit. Ask the user to run `DJ: Sync to SQL and YML` first, then stage the source and its generated siblings together — never commit a JSON change with stale generated files. +2. **Never stage `.dj/`.** It is DJ's local state and is gitignored. Respect the existing `.gitignore` (including any dashboards-as-code marker blocks). +3. **Match the project's commit style.** Scan `git log --oneline`; don't impose the DJ repo's `type(scope):` convention on a downstream project. +4. **Ask before pushing; guard secrets.** Stop at the commit unless the user asks to push; never `git push --force` a shared branch or `git reset --hard`. Follow the repo's `.gitignore`, don't stage hard-coded credentials, and confirm with the user when you're unsure whether a file holds a secret (a `profiles.yml` is fine only when it uses `env_var`). diff --git a/templates/skills/dj-govern-model/_SKILL.md b/templates/skills/dj-govern-model/_SKILL.md new file mode 100644 index 0000000..57eaa7d --- /dev/null +++ b/templates/skills/dj-govern-model/_SKILL.md @@ -0,0 +1,119 @@ +--- +name: dj-govern-model +description: >- + Audit governance posture of DJ models and sources -- ownership, PII / + classification / compliance metadata coverage, registered-group conformance, + and prod-write posture -- across a file, folder, dependency tree, or the whole + workspace. Use when the user wants to review data ownership, check PII / + sensitivity / compliance tagging, find models with no owner, or assess + governance coverage. Read-only: it reports gaps and recommendations, never + edits files and never blocks. +compatibility: DJ (Data JSON) Framework extension workspace with `.dj/schemas/` and `.agents/dj/AGENTS.md` +metadata: + dj-skill: '1.0' +--- + +# Audit DJ governance posture + +**Goal:** produce a **read-only** governance report for `.model.json` / `.source.json` +files in scope. Surface ownership and metadata-coverage gaps, registered-group +conformance, and prod-write posture. **Never edit any file. Never require +governance metadata** — these keys are optional and teams self-enforce their own +policy. This skill only reports; it hands off edits to the authoring / refactor +skills when the user asks to act. + +**Reading order:** `.agents/dj/AGENTS.md` (**Structural Governance**, **Project & +Environment Resolution**) → `.agents/dj/reference/meta-and-governance.md` +(**Custom Meta**, **Governance metadata conventions**) → this skill's checks below. + +## When this skill applies + +- The user mentions reviewing governance, data ownership, owners, stewardship, + PII, sensitivity, classification, compliance, or metadata coverage. +- The user asks "which models have no owner?", "is this PII tagged?", "what's our + governance coverage for group X?", or wants a pre-audit before a review cycle. +- Out of scope: applying tags or refactors. When the user wants to _fix_ a gap, + hand off — model metadata edits to `dj-create-new-model` / + `dj-review-and-refactor-model`, and reserved-key relocation to + `dj-review-and-refactor-model`. + +## Step 1 — Resolve scope and project + +- **Project.** If the workspace holds more than one dbt project, ask which one + (or all) to audit — do not silently pick a default. +- **Scope.** Default to the open `.model.json` if any; otherwise ask for a single + file, a folder, the dependency tree of a base model (from + `target/manifest.json` `child_map` / `parent_map`), or the whole workspace. + Confirm before a folder / tree / workspace pass. + +## Step 2 — Run the checks (read-only) + +Read each in-scope `.model.json` / `.source.json`. Capture findings as +`{ file, check, severity, detail }`. Severity is advisory only: `info` +(coverage note) or `warn` (conformance drift). **Nothing here is an error and +nothing blocks.** + +### A. Structural conformance (framework-enforced — flag drift only) + +- **Registered group.** The model's `group` must be registered in the project's + dbt group definitions — scan `.yml` files for a top-level `groups:` key (e.g. + `models/_groups.yml`, `models/groups.yml`, or per-folder `group_*.yml`) and the + `dbt_project.yml` models config. Flag any model whose group is not registered + (usually a hand-edited or converted file that drifted). +- **Path matches identity.** The file should sit at the framework-derived path + for its `type` + `group` + `topic` + `name` (see AGENTS.md **Structural + Governance**). Flag files whose location or name does not match — do not move + them; report so the user can re-sync. + +### B. Metadata coverage (advisory — report, never require) + +- **Ownership.** Report which models/sources declare `meta.owner` (and + `owner_slack`), and list those without. Present as coverage (e.g. "8/11 models + have an owner"), not pass/fail. +- **Sensitivity.** Report `pii` / `classification` / `compliance` presence at + model and column level. Only offer these keys the project already uses — scan + sibling models first; do not invent a taxonomy the project has not adopted. +- **Freshness.** Note models/sources with vs. without `freshness_sla`. + +Skip any dimension the project does not use at all (e.g. if no model tags +`classification`, report that as "not adopted" once, not as a gap per file). + +### C. Consistency + +- **Owner drift within a group.** Flag a group whose models declare conflicting + `owner` values — often one is stale. +- **Reserved-key collisions.** Flag governance-looking data authored under a + framework-reserved `meta` key instead of its structured sibling (see + `.agents/dj/reference/meta-and-governance.md` **Framework-reserved keys under `meta`**). + Recommend `dj-review-and-refactor-model` to relocate; do not edit here. + +### D. Prod-write posture + +- **Python models.** Note any `.python.json` whose `output.database` / + `output.schema` points at a production target, and whether `write_mode` is + destructive (`overwrite_partitions`). Report for awareness — do not change. +- **Lightdash-backed marts.** Note marts exposing a `lightdash` block with no + `lightdash.table.required_filters` (an unbounded explore). Recommend + `dj-create-new-model` to add a default window if the user wants one. + +## Step 3 — Render the report + +Print a single structured report, grouped by the four check categories, in this +order: **Structural conformance → Metadata coverage → Consistency → Prod-write +posture**. For each category, lead with a one-line summary (coverage numbers +where relevant), then list findings with `file` and `detail`. If a category is +clean, say so in one line. End with **Recommendations** — a short, prioritized +list that names the sibling skill to run for each actionable item. If there are +no findings, say so plainly and exit. + +## Hard rules + +- **Read-only.** Never create, edit, move, or delete a file. If the user asks to + fix something, hand off to the named authoring / refactor skill. +- **Never require governance metadata.** Absence of `owner` / `pii` / + `classification` / `compliance` is a coverage note, not a failure. Do not + push a project to adopt keys it has not chosen. +- **Do not run warehouse queries.** This audit reads JSON files only; it does not + connect to Trino or Lightdash. +- **Respect the framework as the source of truth for placement.** Structural + drift is reported for the user to re-sync, not corrected here. diff --git a/templates/skills/dj-initialize/_SKILL.md b/templates/skills/dj-initialize/_SKILL.md index dc04993..112894a 100644 --- a/templates/skills/dj-initialize/_SKILL.md +++ b/templates/skills/dj-initialize/_SKILL.md @@ -18,6 +18,7 @@ Set up all required and recommended configurations for the **DJ (Data JSON) Fram This skill is designed to work uniformly across all AI coding agents (Cursor, GitHub Copilot, Claude Code, Cline, Windsurf, etc.). Questions must be asked as **plain text** in the conversation — do not rely on agent-specific UI elements like structured forms or multi-choice widgets. Present options as numbered lists or bullet points that the user can respond to conversationally. **Question format:** + - Ask one question or a small related group (2-3 max) at a time - Present options as a numbered list when choices are finite - Always include a "skip" or "use default" option where applicable @@ -37,7 +38,7 @@ Follow these steps **in order**. At each step, report findings to the user and a 1. Find `dbt_project.yml` in the workspace (search with `**/dbt_project.yml`, exclude `node_modules`, `dbt_packages`, `.venv`, `target`) 2. Read the file and extract: `name`, `vars`, `dispatch`, `model-paths`, `macro-paths`, `target-path` -3. Check if `models/groups.yml` exists (or groups defined in `dbt_project.yml`) +3. Check whether any groups are registered — scan `.yml` files for a top-level `groups:` key (e.g. `models/_groups.yml`, `models/groups.yml`, or per-folder `group_*.yml`) and the `dbt_project.yml` models config 4. Check if `.gitignore` exists and whether it contains `.dj` 5. Check if `.vscode/settings.json` exists and has `dj.*` settings 6. Check if `target/manifest.json` exists @@ -46,6 +47,7 @@ Follow these steps **in order**. At each step, report findings to the user and a **Present a summary to the user:** > Here's what I found in your dbt project: +> > - Project name: `` > - Storage type: `` > - Dispatch: `` @@ -62,14 +64,17 @@ Follow these steps **in order**. At each step, report findings to the user and a Ask the user these questions (skip any already answered by the discovered config): 1. **Storage format**: "What storage format does your data lake use?" + - Options: `delta_lake` (Delta Lake + Hive metastore) or `iceberg` (Iceberg + Glue/Polaris) - Default: `delta_lake` 2. **dbt adapter**: "Which dbt adapter do you use?" + - Common: `dbt-trino`, `dbt-postgres`, `dbt-snowflake`, `dbt-bigquery` - This determines the pip package to install 3. **Trino usage** (optional): "Do you use Trino for querying? This is optional — DJ's core features (JSON sync, model creation, lineage) work without it. Trino adds catalog browsing and query execution." + - If yes, will configure Trino env vars later - If no or skip, skip all Trino-related steps entirely @@ -110,9 +115,9 @@ Check which DJ-relevant vars are missing and ask the user to confirm values: ```yaml vars: - storage_type: '' # Drives partitioning SQL generation - etl_schema: '' # ETL metadata schema (default: source_etl) - event_dates: '' # Date range for lookback models (format: YYYY-MM-DD~YYYY-MM-DD) + storage_type: '' # Drives partitioning SQL generation + etl_schema: '' # ETL metadata schema (default: source_etl) + event_dates: '' # Date range for lookback models (format: YYYY-MM-DD~YYYY-MM-DD) ``` **Ask**: "I'd like to add these variables to your `dbt_project.yml`. Here are the recommended values based on your answers. Should I proceed?" @@ -143,7 +148,7 @@ Wait for confirmation before adding. ### Step 6: Configure Groups -If no groups are defined (neither in `dbt_project.yml` models config nor in `models/groups.yml`): +If no groups are defined anywhere (no `.yml` with a top-level `groups:` key — e.g. `models/_groups.yml`, `models/groups.yml`, or per-folder `group_*.yml` — and none in the `dbt_project.yml` models config): **Ask**: "DJ's model creation wizard uses dbt groups to organize models. What business domains or teams does your project serve?" @@ -190,6 +195,7 @@ If `.gitignore` doesn't exist, ask before creating one. Check `.vscode/settings.json` for DJ settings. **Ask**: "I'll configure VS Code settings for DJ. Please confirm the Python venv path:" + - Default: `.venv` (relative to workspace root) Create or update `.vscode/settings.json` with: @@ -201,6 +207,7 @@ Create or update `.vscode/settings.json` with: ``` **Additionally ask**: + - "Do you want to restrict DJ to specific project names?" (for monorepos with multiple `dbt_project.yml` files) - If yes: `"dj.dbtProjectNames": [""]` - "What log level do you prefer?" (default: `info`) @@ -226,12 +233,14 @@ If the user indicated they use Trino: 6. `TRINO_PASSWORD` — (optional) password **Then ask**: "Where should I add these environment variables?" + - Options: `.env` file in project root, or suggest adding to shell profile (`~/.zshrc`, `~/.bashrc`) If `.env`: create the file with the values. If shell profile: provide the export commands for the user to add manually. Also check Trino CLI availability: + - "Is `trino-cli` on your PATH? (run `which trino-cli` to check)" - If not, suggest: `"dj.trinoPath": "/path/to/trino-cli"` in VS Code settings @@ -246,6 +255,7 @@ These are all optional. Ask about each briefly — if the user says no or skip, **Ask**: "Do you use Lightdash for BI dashboards?" If yes: + - Ensure `npm install -g @lightdash/cli` is done (or suggest it) - Ask for env vars: `LIGHTDASH_URL`, `LIGHTDASH_PREVIEW_NAME`, `LIGHTDASH_PROJECT` - Add to `.env` or suggest shell profile additions @@ -255,6 +265,7 @@ If yes: **Ask**: "Do you want DJ to generate Airflow DAGs for your models?" If yes, add to VS Code settings: + ```json { "dj.airflowGenerateDags": true, @@ -268,6 +279,7 @@ If yes, add to VS Code settings: **Ask**: "Would you like DJ to generate AI agent context files (`.agents/dj/AGENTS.md` and skill files)?" If yes, add to VS Code settings: + ```json { "dj.codingAgent": true @@ -283,6 +295,7 @@ If `target/manifest.json` is missing or the user wants to refresh it: **Ask**: "Would you like me to run `dbt parse` now to generate the manifest? (Required for lineage and column features)" If yes, run: + ```bash source .venv/bin/activate && dbt parse ``` @@ -298,19 +311,22 @@ Present a final summary of everything that was configured: > **DJ Initialization Complete** > > **Configured:** +> > - [x] Python venv at `.venv` with `` > - [x] `dbt_project.yml` vars: `storage_type`, `etl_schema` > - [x] Dispatch block added -> - [x] Groups defined in `models/groups.yml` +> - [x] Groups registered (in a `groups:` yaml such as `models/groups.yml`) > - [x] `.gitignore` updated with `.dj/` > - [x] VS Code settings configured > - [x] Manifest generated > > **Skipped:** +> > - [ ] Trino (user opted out) > - [ ] Lightdash (not needed) > > **Next steps:** +> > - Open VS Code Command Palette → `DJ: Refresh Projects` to load the new configuration > - Try creating your first model: Command Palette → `DJ: Create Model` > - Test Trino connection: Command Palette → `DJ: Test Trino Connection` diff --git a/templates/skills/dj-migrate-ephemerals-to-ctes/_SKILL.md b/templates/skills/dj-migrate-ephemerals-to-ctes/_SKILL.md index d37ceb0..3c69e76 100644 --- a/templates/skills/dj-migrate-ephemerals-to-ctes/_SKILL.md +++ b/templates/skills/dj-migrate-ephemerals-to-ctes/_SKILL.md @@ -16,7 +16,7 @@ metadata: **Goal:** dissolve qualifying ephemeral `.model.json` files into the `ctes[]` array of their downstream consumer, then remove the now-redundant file. Mutate **only** the JSON sources of truth — the framework's sync engine regenerates the `.sql` / `.yml` artifacts. -**Reading order:** `.agents/dj/AGENTS.md` (Model Types, Inline CTEs, Important Conventions) → `.dj/schemas/model.cte.schema.json` + `model.materialization.schema.json` → this skill's `references/transformation-matrix.md` for per-type recipes. +**Reading order:** `.agents/dj/AGENTS.md` (**Important Conventions**), `.agents/dj/reference/model-types.md`, and `.agents/dj/reference/ctes-and-subqueries.md` → `.dj/schemas/model.cte.schema.json` + `model.materialization.schema.json` → this skill's `references/transformation-matrix.md` for per-type recipes. ## When this skill applies @@ -26,6 +26,7 @@ metadata: ## Workflow +- [ ] **0. Resolve the target project.** If the workspace holds more than one dbt project, ask the user which one to migrate — do not silently scan a default. The `models/` searches and `target/manifest.json` lookups below are relative to that project (`model.${project}.${name}`). - [ ] **1. Inventory ephemerals.** Use ripgrep to find candidates -- never iterate-and-parse every `.model.json` (context-budget poison). Search for **all four** forms: 1. `rg -l '"materialization"\s*:\s*"ephemeral"' models/` 2. `rg -l '"materialization"\s*:\s*\{\s*"type"\s*:\s*"ephemeral"' models/` diff --git a/templates/skills/dj-migrate-ephemerals-to-ctes/references/transformation-matrix.md b/templates/skills/dj-migrate-ephemerals-to-ctes/references/transformation-matrix.md index 69da8a6..ae31492 100644 --- a/templates/skills/dj-migrate-ephemerals-to-ctes/references/transformation-matrix.md +++ b/templates/skills/dj-migrate-ephemerals-to-ctes/references/transformation-matrix.md @@ -289,7 +289,7 @@ Some ephemerals shouldn't be inlined into a CTE at all -- inlining forces the lo } ``` -**Strategy guidance** (per `AGENTS.md` "Materialization shorthand" section): +**Strategy guidance** (per `.agents/dj/reference/materialization.md`): - **Iceberg** projects: `dj_iceberg_partition_overwrite` is DJ-shipped and partition-safe; `merge` works for row-level upserts on `unique_key`. - **Delta Lake / Hive** projects: prefer `delete+insert` (partition-safe, no custom macro required). Avoid `merge` (Iceberg-only in dbt-trino). diff --git a/templates/skills/dj-resolve-merge-conflicts/_SKILL.md b/templates/skills/dj-resolve-merge-conflicts/_SKILL.md index 745dd22..a84ef9e 100644 --- a/templates/skills/dj-resolve-merge-conflicts/_SKILL.md +++ b/templates/skills/dj-resolve-merge-conflicts/_SKILL.md @@ -17,7 +17,7 @@ metadata: **Goal:** resolve git conflicts in a DJ workspace _correctly for this framework_. The only files you ever hand-merge are the JSON sources of truth (`.model.json` / `.source.json`). The generated `.sql` / `.yml` siblings are **regenerated from the JSON** -- never hand-merged. When the incoming branch looks old or diverged, pause and let the user choose between a full merge and a guided port of specific models. -**Reading order:** `.agents/dj/AGENTS.md` (Model Types, Important Conventions, sync flow) -> `.dj/schemas/` (validate resolved JSON) -> this skill's `references/staleness-and-porting.md` once you reach the staleness gate (Phase 1) or the port path (Phase 2b). +**Reading order:** `.agents/dj/AGENTS.md` (**Important Conventions**, sync flow) and `.agents/dj/reference/model-types.md` -> `.dj/schemas/` (validate resolved JSON) -> this skill's `references/staleness-and-porting.md` once you reach the staleness gate (Phase 1) or the port path (Phase 2b). For staging and committing once conflicts are resolved and re-synced, see `.agents/dj/reference/git-workflow.md`. ## When this skill applies diff --git a/templates/skills/dj-review-and-refactor-model/_SKILL.md b/templates/skills/dj-review-and-refactor-model/_SKILL.md index 026e0a2..5c2ee42 100644 --- a/templates/skills/dj-review-and-refactor-model/_SKILL.md +++ b/templates/skills/dj-review-and-refactor-model/_SKILL.md @@ -17,7 +17,7 @@ metadata: **Goal:** audit `.model.json` files against the latest DJ capabilities, render **all** findings upfront in two buckets (**Recommended** / **Needs your decision**), then apply **only** what the user confirms. Mutate the JSON sources of truth -- the framework regenerates `.sql` / `.yml` artifacts. -**Reading order:** `.agents/dj/AGENTS.md` (Advanced section, Materialization & Incremental Strategies, Custom Meta, Lightdash) → `.dj/schemas/` (`model.materialization.schema.json`, `model.from.rollup.schema.json`, `model.subquery.schema.json`, `model.cte.schema.json`, `lightdash.*.schema.json`) → this skill's `references/refactor-catalog.md` once the apply phase begins. +**Reading order:** `.agents/dj/reference/model-types.md` (Advanced), `.agents/dj/reference/materialization.md`, `.agents/dj/reference/meta-and-governance.md`, `.agents/dj/reference/lightdash-tags-tests.md` → `.dj/schemas/` (`model.materialization.schema.json`, `model.from.rollup.schema.json`, `model.subquery.schema.json`, `model.cte.schema.json`, `lightdash.*.schema.json`) → this skill's `references/refactor-catalog.md` once the apply phase begins. ## When this skill applies @@ -32,7 +32,7 @@ metadata: ## Workflow -- [ ] **1. Resolve scope.** Default to the open `.model.json` in the editor if any. Otherwise ask whether the user wants a single named file, all `.model.json` under a folder, the dependency tree of a base model (resolved from `target/manifest.json` via `child_map` / `parent_map`), or the entire workspace. Confirm before proceeding for folder / tree / workspace scope. +- [ ] **1. Resolve scope.** Default to the open `.model.json` in the editor if any. Otherwise ask whether the user wants a single named file, all `.model.json` under a folder, the dependency tree of a base model (resolved from `target/manifest.json` via `child_map` / `parent_map`), or the entire workspace. Confirm before proceeding for folder / tree / workspace scope. **If the workspace holds more than one dbt project, also confirm which project (or that all projects) is in scope — do not silently assume one.** - [ ] **2. Detect.** Read each in-scope `.model.json` and apply the catalog below. **Do not edit anything.** Capture each finding as `{ file, pattern, group, before, after, why? }`. Skip ephemeral inlining candidates entirely (they belong to `dj-migrate-ephemerals-to-ctes`). - [ ] **3. Render the review.** Print a single numbered report using the template below. Recommended items get numeric labels `[1]` `[2]` ...; Needs-your-decision items get letter labels `[A]` `[B]` .... If there are zero findings, say so plainly and exit -- do not invent work. - [ ] **4. Wait for confirmation.** Ask the user which items to apply (see "Confirmation prompt" below). **No edits until the user replies.** Treat any non-matching reply as "stop and ask again", not "apply all". @@ -47,13 +47,13 @@ Each row below is **one** finding type. Verbose before/after JSONC, detection he ### Group 1: Recommended (safe rewrites, behavior-preserving) -| # | Pattern | Why it's safe | -| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | -| R1 | Top-level `materialized` + optional `incremental_strategy` + optional `partitioned_by` → single structured `materialization` block | Schema marks the legacy keys `deprecated: true`; `materialization` takes precedence | -| R2 | `meta.dimension` / `meta.metrics` / `meta.metrics_merge` / `meta.case_sensitive` on a `select` item → `lightdash.dimension` / `lightdash.metrics` / etc. | Framework already raises a Warning diagnostic on these; rewrite clears it | -| R3 | `"group_by": [{ "type": "dims" }]` → `"group_by": "dims"` | Pure shorthand; same SQL | -| R4 | `exclude_datetime: true` + `exclude_portal_partition_columns: true` + `exclude_portal_source_count: true` (± `exclude_date_filter`) → `exclude_framework_artifacts: "columns"` (or `"all"` if `exclude_date_filter` is also set) | Combined-flag shortcut documented in `AGENTS.md`; same resolution | -| R5 | `where: { and: [{ expr: "x = 'y'" }] }` (single string expression, no other conditions) → `where: "x = 'y'"` | String shorthand; same SQL | +| # | Pattern | Why it's safe | +| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | +| R1 | Top-level `materialized` + optional `incremental_strategy` + optional `partitioned_by` → single structured `materialization` block | Schema marks the legacy keys `deprecated: true`; `materialization` takes precedence | +| R2 | `meta.dimension` / `meta.metrics` / `meta.metrics_merge` / `meta.case_sensitive` on a `select` item → `lightdash.dimension` / `lightdash.metrics` / etc. | Framework already raises a Warning diagnostic on these; rewrite clears it | +| R3 | `"group_by": [{ "type": "dims" }]` → `"group_by": "dims"` | Pure shorthand; same SQL | +| R4 | `exclude_datetime: true` + `exclude_portal_partition_columns: true` + `exclude_portal_source_count: true` (± `exclude_date_filter`) → `exclude_framework_artifacts: "columns"` (or `"all"` if `exclude_date_filter` is also set) | Combined-flag shortcut documented in `.agents/dj/reference/ctes-and-subqueries.md`; same resolution | +| R5 | `where: { and: [{ expr: "x = 'y'" }] }` (single string expression, no other conditions) → `where: "x = 'y'"` | String shorthand; same SQL | ### Group 2: Needs your decision (context attached; user picks) diff --git a/templates/skills/dj-review-and-refactor-model/references/refactor-catalog.md b/templates/skills/dj-review-and-refactor-model/references/refactor-catalog.md index 5c53532..1698d2c 100644 --- a/templates/skills/dj-review-and-refactor-model/references/refactor-catalog.md +++ b/templates/skills/dj-review-and-refactor-model/references/refactor-catalog.md @@ -83,7 +83,7 @@ For every pattern: preserve `ai_hint`, `lightdash.*`, `data_tests`, `tags`, `des - `select[i].meta.metrics_merge` is an object, OR - `select[i].meta.case_sensitive` is a boolean. -The reserved-keys list comes from the `Framework-reserved keys under meta` table in `templates/_AGENTS.md`. +The reserved-keys list comes from the `Framework-reserved keys under meta` table in `.agents/dj/reference/meta-and-governance.md`. **Before:** @@ -213,7 +213,7 @@ If `exclude_date_filter: true` is also set at the same scope, the After is `"exc - **Skip when `from.rollup` is at the same scope.** `exclude_framework_artifacts` (when its resolved value implies excluding `datetime`) is mutually exclusive with `from.rollup`. The validator already errors on the combo. Don't propose the rewrite there. - **Mixed `true` / `false` doesn't qualify.** If any of the three booleans is `false` (explicitly opting back in), don't collapse -- the user is opting into a partial set. - **CTE-scope vs model-scope.** Apply the rewrite at whichever scope all three are set. Do **not** hoist a CTE's three-flag set up to the model level (different semantics). -- **Remove all three (or four) legacy keys in the same edit.** Leaving any of them mixed with the new combined flag is confusing and (per AGENTS.md) lets the individual flag override per-column anyway. +- **Remove all three (or four) legacy keys in the same edit.** Leaving any of them mixed with the new combined flag is confusing and (per `.agents/dj/reference/ctes-and-subqueries.md`) lets the individual flag override per-column anyway. - **`exclude_daily_filter`** is **not** part of `exclude_framework_artifacts`; it stays as its own boolean. --- @@ -291,7 +291,7 @@ If `exclude_date_filter: true` is also set at the same scope, the After is `"exc - **`interval` values are exactly `hour` / `day` / `month` / `year`.** Not `weekly`, not `quarterly`. There is no `datetime_expr` field -- the upstream's `datetime` column with an `interval` drives the rollup. - **`exclude_datetime` / `exclude_framework_artifacts` are mutually exclusive with `from.rollup` at the same scope.** If the rollup model has either, the conversion errors at validation -- surface this in the Why-decide note. - **Joins.** If the rollup is consumed alongside joins, suggest `int_join_models` instead of `int_select_model`. Either type accepts `from.rollup`. -- See `model.from.rollup.schema.json` and `templates/_AGENTS.md` (Advanced section) for the full shape. +- See `model.from.rollup.schema.json` and `.agents/dj/reference/model-types.md` (Advanced) for the full shape. --- diff --git a/templates/skills/dj-review-python-model/_SKILL.md b/templates/skills/dj-review-python-model/_SKILL.md index 02d6c1d..d4e0868 100644 --- a/templates/skills/dj-review-python-model/_SKILL.md +++ b/templates/skills/dj-review-python-model/_SKILL.md @@ -40,65 +40,66 @@ Use this skill when the user mentions: review python model, audit python model, ### 1. Framework Compliance (F) -| Check | What to validate | -|-------|-----------------| -| F1 | `.python.json` has required fields: `name`, `group`, `topic` | -| F2 | Name/group/topic match pattern `^[a-z][a-z0-9_]*$` | -| F3 | `cells` array is present and non-empty in JSON | -| F4 | `.python.py` contains `def run_etl(context)` function | -| F5 | Uses `_trino_io` helpers (`from python_models._trino_io import ...`) — no inline Trino connection code (`trino.dbapi.connect`, `create_engine`, raw `requests.post` to Trino) | -| F6 | `OUTPUT_CONFIG` uses `PythonModelConfig` from `python_models._config` | -| F7 | `.python.py` content is derivable from JSON `cells` (no hand-edits that would be lost on next sync) | -| F8 | Runner cell (`run_etl(context)`) is the last code cell in JSON | -| F9 | ETL follows the standard function structure: `extract()`, `transform_and_load()`, `cleanup()`, `run_etl()` | +| Check | What to validate | +| ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| F1 | `.python.json` has required fields: `name`, `group`, `topic` | +| F2 | Name/group/topic match pattern `^[a-z][a-z0-9_]*$` | +| F3 | `cells` array is present and non-empty in JSON | +| F4 | `.python.py` contains `def run_etl(context)` function | +| F5 | Uses `_trino_io` helpers (`from python_models._trino_io import ...`) — no inline Trino connection code (`trino.dbapi.connect`, `create_engine`, raw `requests.post` to Trino) | +| F6 | `OUTPUT_CONFIG` uses `PythonModelConfig` from `python_models._config` | +| F7 | `.python.py` content is derivable from JSON `cells` (no hand-edits that would be lost on next sync) | +| F8 | Runner cell (`run_etl(context)`) is the last code cell in JSON | +| F9 | ETL follows the standard function structure: `extract()`, `transform_and_load()`, `cleanup()`, `run_etl()` | ### 2. Lineage Readiness (L) — end-to-end validation The DJ lineage engine discovers Python models by querying Iceberg `$properties` on output tables. It reads: + - `python_model_name` — model identity - `python_model_table` — output table name - `python_model_upstream_sources` — comma-separated `schema.table` pairs for upstream lineage edges The review validates the **full chain**: JSON metadata → `PythonModelConfig` → Iceberg table properties → lineage discoverability. -| Check | What to validate | -|-------|-----------------| -| L1 | `PythonModelConfig` instantiation emits all required properties: `python_model_name`, `python_model_type`, `python_model_namespace`, `python_model_table`, `python_model_description` | -| L2 | `python_model_upstream_sources` is set on the output table — code must call `ALTER TABLE ... SET PROPERTIES` or use PyIceberg catalog API to write this key listing all source tables the ETL reads from | -| L3 | Each entry in `python_model_upstream_sources` follows `schema.table` format (dot-separated, matching actual source tables) | -| L4 | **Completeness** — every table referenced in `extract()` / `transform_and_load()` SQL (FROM/JOIN clauses) appears in `python_model_upstream_sources` (no missing upstream edges) | -| L5 | **Accuracy** — every entry in `python_model_upstream_sources` corresponds to a table actually queried in the model (no stale/phantom entries) | -| L6 | `dags` field is populated in JSON (model is scheduled, discoverable by Airflow DAG lineage). Empty `dags` = utility module, not a lineage participant | -| L7 | `depends_on` correctly lists upstream Python models whose output tables this model reads (task dependency mirrors data dependency) | -| L8 | Model ID derivable from file path matches `python_model_name` in properties: `python______` | -| L9 | Companion `.python.py` exists alongside `.python.json` (required for Airflow `etl_helper.py` discovery via `def run_etl(`) | -| L10 | `OUTPUT_CONFIG.table_name` (or `model_name` fallback) matches the table name referenced in downstream `.source.json` files, if any exist in the project | +| Check | What to validate | +| ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| L1 | `PythonModelConfig` instantiation emits all required properties: `python_model_name`, `python_model_type`, `python_model_namespace`, `python_model_table`, `python_model_description` | +| L2 | `python_model_upstream_sources` is set on the output table — code must call `ALTER TABLE ... SET PROPERTIES` or use PyIceberg catalog API to write this key listing all source tables the ETL reads from | +| L3 | Each entry in `python_model_upstream_sources` follows `schema.table` format (dot-separated, matching actual source tables) | +| L4 | **Completeness** — every table referenced in `extract()` / `transform_and_load()` SQL (FROM/JOIN clauses) appears in `python_model_upstream_sources` (no missing upstream edges) | +| L5 | **Accuracy** — every entry in `python_model_upstream_sources` corresponds to a table actually queried in the model (no stale/phantom entries) | +| L6 | `dags` field is populated in JSON (model is scheduled, discoverable by Airflow DAG lineage). Empty `dags` = utility module, not a lineage participant | +| L7 | `depends_on` correctly lists upstream Python models whose output tables this model reads (task dependency mirrors data dependency) | +| L8 | Model ID derivable from file path matches `python_model_name` in properties: `python______` | +| L9 | Companion `.python.py` exists alongside `.python.json` (required for Airflow `etl_helper.py` discovery via `def run_etl(`) | +| L10 | `OUTPUT_CONFIG.table_name` (or `model_name` fallback) matches the table name referenced in downstream `.source.json` files, if any exist in the project | ### 3. Downstream Integration (D) -| Check | What to validate | -|-------|-----------------| -| D1 | Output table uses standard catalog/schema convention (`glue_development.opus_python_source` or project-configured equivalent) | -| D2 | Partition column `portal_partition_daily` is emitted in output SQL (`'{ds}' AS portal_partition_daily` or equivalent) | -| D3 | Column names in output SQL follow `snake_case` convention (no camelCase, no spaces, no special characters) | -| D4 | No `SELECT *` in production INSERT — explicit column enumeration for schema stability and consumer predictability | -| D5 | Table name matches model name for source discovery (`OUTPUT_CONFIG.table_name or model_name` → downstream `source.table`) | -| D6 | Output columns are stable — column order and types should be deterministic across runs | +| Check | What to validate | +| ----- | ----------------------------------------------------------------------------------------------------------------------------- | +| D1 | Output table uses standard catalog/schema convention (`glue_development.opus_python_source` or project-configured equivalent) | +| D2 | Partition column `portal_partition_daily` is emitted in output SQL (`'{ds}' AS portal_partition_daily` or equivalent) | +| D3 | Column names in output SQL follow `snake_case` convention (no camelCase, no spaces, no special characters) | +| D4 | No `SELECT *` in production INSERT — explicit column enumeration for schema stability and consumer predictability | +| D5 | Table name matches model name for source discovery (`OUTPUT_CONFIG.table_name or model_name` → downstream `source.table`) | +| D6 | Output columns are stable — column order and types should be deterministic across runs | ### 4. Performance & Best Practices (P) -| Check | What to validate | -|-------|-----------------| -| P1 | **SQL-first adherence** — transformations (filter, cast, aggregate, join, deduplicate) done in Trino SQL, not pandas/Python | -| P2 | Staging tables cleaned up in `cleanup()` — no orphaned `stg_tmp_*` tables left after ETL completes | -| P3 | Partition predicate pushdown — WHERE clauses on source tables include partition column filters | -| P4 | Batch staging — large datasets chunked before INSERT (not unbounded single INSERT of millions of rows) | -| P5 | Write mode appropriate for use case: `overwrite_partition` for idempotent daily loads, `append` for event streams, `overwrite` for full refresh | -| P6 | No full-table scans — queries against large tables include a partition filter or LIMIT | -| P7 | DataFrame operations restricted to external ingestion — any pandas/polars used only for API response parsing, not for SQL-expressible transforms | -| P8 | Explicit `CAST(... AS type)` in SQL for type safety — not relying on pandas `.astype()` or implicit coercion | -| P9 | Error handling — `run_etl()` has try/except with cleanup on failure (staging tables dropped even if transform fails) | -| P10 | Idempotency — re-running the model for the same `ds` produces identical results (no append-on-rerun for partition-based models) | +| Check | What to validate | +| ----- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| P1 | **SQL-first adherence** — transformations (filter, cast, aggregate, join, deduplicate) done in Trino SQL, not pandas/Python | +| P2 | Staging tables cleaned up in `cleanup()` — no orphaned `stg_tmp_*` tables left after ETL completes | +| P3 | Partition predicate pushdown — WHERE clauses on source tables include partition column filters | +| P4 | Batch staging — large datasets chunked before INSERT (not unbounded single INSERT of millions of rows) | +| P5 | Write mode appropriate for use case: `overwrite_partition` for idempotent daily loads, `append` for event streams, `overwrite` for full refresh | +| P6 | No full-table scans — queries against large tables include a partition filter or LIMIT | +| P7 | DataFrame operations restricted to external ingestion — any pandas/polars used only for API response parsing, not for SQL-expressible transforms | +| P8 | Explicit `CAST(... AS type)` in SQL for type safety — not relying on pandas `.astype()` or implicit coercion | +| P9 | Error handling — `run_etl()` has try/except with cleanup on failure (staging tables dropped even if transform fails) | +| P10 | Idempotency — re-running the model for the same `ds` produces identical results (no append-on-rerun for partition-based models) | ## Report template @@ -151,20 +152,20 @@ Render the full report in this structure. Adapt headings to the actual model. Om ### Severity classification -| Severity | Criteria | Report label | -|----------|----------|--------------| -| **Issue** | Blocks production deployment or breaks lineage/downstream consumers | `ISSUE` | -| **Warning** | Degrades performance, violates best practices, or creates maintenance risk | `WARNING` | -| **Suggestion** | Improvement opportunity; not blocking but raises quality | `SUGGESTION` | +| Severity | Criteria | Report label | +| -------------- | -------------------------------------------------------------------------- | ------------ | +| **Issue** | Blocks production deployment or breaks lineage/downstream consumers | `ISSUE` | +| **Warning** | Degrades performance, violates best practices, or creates maintenance risk | `WARNING` | +| **Suggestion** | Improvement opportunity; not blocking but raises quality | `SUGGESTION` | ### Recommendation priority -| Priority | Criteria | -|----------|----------| +| Priority | Criteria | +| -------- | ----------------------------------------------------------------------- | | CRITICAL | Lineage broken (L2-L5 failures), missing `run_etl`, no partition column | -| HIGH | Inline Trino code, missing cleanup, stale upstream_sources | -| MEDIUM | Non-SQL-first transforms, missing batch staging, no error handling | -| LOW | Naming style, column order, documentation gaps | +| HIGH | Inline Trino code, missing cleanup, stale upstream_sources | +| MEDIUM | Non-SQL-first transforms, missing batch staging, no error handling | +| LOW | Naming style, column order, documentation gaps | ## Hard rules (DO NOT) diff --git a/templates/skills/dj-run-dbt/_SKILL.md b/templates/skills/dj-run-dbt/_SKILL.md new file mode 100644 index 0000000..5be8bea --- /dev/null +++ b/templates/skills/dj-run-dbt/_SKILL.md @@ -0,0 +1,25 @@ +--- +name: dj-run-dbt +description: >- + Run a dbt command from the terminal in a DJ (Data JSON) Framework dbt + project — compile, parse, ls, deps, docs generate, test, or a + warehouse-writing run/build/seed/snapshot. Use when the user asks to + "compile this model", "run dbt", "parse the project", "build my models", + "dbt test", refresh the manifest, or otherwise execute the dbt CLI. Not for + authoring or editing models (-> dj-create-new-model), and not for analyzing + Trino query diagnostics (-> dj-trino-analyzer). +compatibility: DJ (Data JSON) Framework workspace with a dbt project and .agents/dj/AGENTS.md +metadata: + dj-skill: '1.0' +--- + +# Run dbt + +Execute dbt CLI commands correctly in this project. The full mechanics — activating the Python venv (`dj.pythonVenvPath`), running from the dbt project directory, and the read-only vs warehouse-writing command classes — live in `.agents/dj/reference/running-dbt.md`. Read it before running anything. + +Essentials: + +1. **Activate the venv first.** A terminal does not inherit DJ's Python environment. Resolve `dj.pythonVenvPath` from `.vscode/settings.json` (fall back to `.venv`), `source /bin/activate`, and verify `dbt --version`. +2. **Run from the dbt project directory** (the one with `dbt_project.yml`), not necessarily the workspace root. If more than one project exists, confirm which one. +3. **Sync before compiling.** dbt reads the generated `.sql`, not the `.model.json`. After a JSON edit, ask the user to run `DJ: Sync to SQL and YML` first. +4. **Read-only by default.** `parse` / `compile` / `ls` / `deps` / `docs generate` / `test` are safe. `run` / `build` / `seed` / `snapshot` / `run-operation` write to the warehouse — get explicit per-command confirmation and never target production. See **Command & Query Execution Safety** in `.agents/dj/AGENTS.md`. diff --git a/templates/skills/dj-run-trino/_SKILL.md b/templates/skills/dj-run-trino/_SKILL.md new file mode 100644 index 0000000..962e527 --- /dev/null +++ b/templates/skills/dj-run-trino/_SKILL.md @@ -0,0 +1,24 @@ +--- +name: dj-run-trino +description: >- + Run a read-only Trino SQL query from the terminal to inspect warehouse data or + schema in a DJ (Data JSON) Framework project. Use when the user asks to "query + Trino", "preview rows", "DESCRIBE"/"SHOW" a table, check what data or columns a + table has, or sanity-check a value. Not for diagnosing captured query + performance JSON (-> dj-trino-analyzer) or registering a table as a source + (-> dj-create-source). +compatibility: DJ (Data JSON) Framework workspace with a Trino CLI and .agents/dj/AGENTS.md +metadata: + dj-skill: '1.0' +--- + +# Run a Trino query + +Execute read-only Trino queries correctly in this project. The full mechanics — resolving the CLI from `dj.trinoPath`, the `TRINO_*` connection environment, the `--execute … --output-format=CSV_HEADER` invocation, and identifier quoting — live in `.agents/dj/reference/running-trino.md`. Read it before running anything. + +Essentials: + +1. **Resolve the CLI.** `dj.trinoPath` (default `trino-cli` on `PATH`). If it doesn't resolve, ask the user to install it or set the path. +2. **Confirm the connection.** The connection comes from `TRINO_HOST` / `TRINO_PORT` / `TRINO_USERNAME` / `TRINO_CATALOG` / `TRINO_SCHEMA` in the environment. If unset, ask for them and confirm the cluster is non-production before running. +3. **Read-only only.** Run `SELECT` / `SHOW` / `DESCRIBE` / `EXPLAIN`, always with a `LIMIT` and partition constraint. Any DDL/DML needs explicit confirmation and must never hit production. See **Command & Query Execution Safety** in `.agents/dj/AGENTS.md`. +4. **Prefer framework facilities.** Read `.source.json` / `.model.json` / `target/manifest.json` / `.dj/schemas/` before shelling out to the CLI. diff --git a/templates/skills/dj-trino-analyzer/_SKILL.md b/templates/skills/dj-trino-analyzer/_SKILL.md index be2f2d3..81e81b4 100644 --- a/templates/skills/dj-trino-analyzer/_SKILL.md +++ b/templates/skills/dj-trino-analyzer/_SKILL.md @@ -17,6 +17,8 @@ metadata: # Analyze a Trino query plan + runtime stats +This skill is **read-only** — it inspects the diagnostics JSON, not a live cluster, and never needs to modify anything. Most diagnoses end in the JSON. If you do reproduce or `EXPLAIN` a query, run **only** read-only statements (`SELECT` / `EXPLAIN` / `SHOW` / `DESCRIBE`) — never DDL/DML; a read-only query against production is fine. Target the same cluster and catalog the diagnostics came from — `summary.catalog`, `profileName`, and `coordinatorUrl` identify them — and reuse those rather than guessing (confirm with the user only when they're absent). See **Command & Query Execution Safety** in `.agents/dj/AGENTS.md`; for how to run those statements, see `.agents/dj/reference/running-trino.md`. + The DJ extension writes two files per analyzed query: - **`.dj/diagnostics/.json`** — sanitized, shaped for LLM diff --git a/templates/skills/dj-update-ai-hints/_SKILL.md b/templates/skills/dj-update-ai-hints/_SKILL.md index 2d8d0b5..4a8143c 100644 --- a/templates/skills/dj-update-ai-hints/_SKILL.md +++ b/templates/skills/dj-update-ai-hints/_SKILL.md @@ -16,7 +16,7 @@ description: Add or update Lightdash AI hints across a model's dependency tree. - **baseModel**: Base Model that we trying to update ai hints for. - **tag**: The tag to filter models to apply them for AI Agent in Lightdash. -## Key Model Type Identification: +## Key Model Type Identification - **Mart Models (mart\_\_)**: Final business-ready models - **Intermediate Models (int\_\_)**: Business logic and aggregations @@ -26,6 +26,7 @@ description: Add or update Lightdash AI hints across a model's dependency tree. ## Workflow +0. **Resolve the target project.** If the workspace holds more than one dbt project, ask the user which one to update — do not silently search a default. The `models/`, `sources/`, and `seeds/` paths below are relative to that project. 1. **Read in the Excel file** and load the sheet named using Python (preferably with 'pandas'). 2. **Identify and trace the complete dependency tree**: - **Start with the base model** with the name passed as parameter @@ -106,15 +107,15 @@ mart__analytics__billing__accounts_daily (ROOT) ## DO NOT's -1. ** DO NOT ** modify any files other than `.json` and files that are part of the dependency tree. -1. ** DO NOT ** add any columns or metrics. Just update existing. -1. ** DO NOT ** create additional scripts to achieve this. Just make the changes and Leave to user to whether commit them or revert back. -1. ** DO NOT ** modify or create `.yml` files. -1. ** DO NOT ** create empty dimension objects unnecessarily. -1. ** DO NOT ** remove comments or change formatting. Preserve JSONC structure completely. -1. ** DO NOT ** add tags to intermediate, staging, or source files - only the base model gets the specified tag. +1. **DO NOT** modify any files other than `.json` and files that are part of the dependency tree. +1. **DO NOT** add any columns or metrics. Just update existing. +1. **DO NOT** create additional scripts to achieve this. Just make the changes and Leave to user to whether commit them or revert back. +1. **DO NOT** modify or create `.yml` files. +1. **DO NOT** create empty dimension objects unnecessarily. +1. **DO NOT** remove comments or change formatting. Preserve JSONC structure completely. +1. **DO NOT** add tags to intermediate, staging, or source files - only the base model gets the specified tag. -### Success Metrics: +### Success Metrics - AI hints placed according to workflow - Only base model receives specified tag