Genie Code Skills Demo

Example Genie Code skills, custom instructions, and MCP setup for enforcing enterprise data engineering standards across Databricks development.

What is Genie Code?

Genie Code is Databricks' AI coding assistant. It can:

Author SDP pipelines -- streaming tables, materialized views, CDC, medallion architecture
Develop notebooks -- Python, SQL, Scala, R
Build DSML workflows -- MLflow experiment tracking, model training, feature engineering
Create AI/BI dashboards -- data visualizations and business intelligence
Develop Databricks Apps -- full-stack applications on the lakehouse
Manage Unity Catalog -- governance, permissions, lineage
Orchestrate jobs and workflows -- scheduling, dependencies, monitoring

Genie Code supports skills (task-specific instructions following the open Agent Skills standard) and MCP (Model Context Protocol) connections that fetch enterprise standards from external sources like GitHub.

Demo Coverage

This repo provides example skills, instructions, and tooling organized by domain. Each phase adds a new area of coverage.

Phase	Date	Domain	What's Included
Phase 1	April 2026	Data Engineering	SDP pipeline skills, PII management, table governance, MCP-based standards enforcement, sample financial data generation
Phase 2	May 2026	DSML	`sentiment-analysis` skill, AI-function patterns (`ai_analyze_sentiment`, `ai_classify`, `ai_extract`), AI/BI Bakehouse Marketplace install
Planned	--	Dashboards, Governance	Additional skill domains and MCP integrations

What This Repo Contains

Folder	Contents
`skills/data_eng/`	Skills for SDP pipelines, PII management, and table/column governance
`skills/dsml/`	DSML skills (currently `sentiment-analysis` for AI-function pipelines)
`instructions/`	Custom instruction templates (user-level and workspace-level)
`mcp/`	MCP connection setup: deploy script and config template
`sample_data_gen/`	Synthetic financial data generation notebook (uses `dbldatagen`)
`marketplace_data/`	Installer notebook + config template for sourcing datasets from the Databricks Marketplace (currently AI/BI Bakehouse)
`local_deployment/`	Gitignored. Your workspace-specific DAB config for deploying to your environment. See `local_deployment/README.md` for setup instructions.

How It Works

This demo shows three stages of Genie Code customization:

1. Baseline

Genie Code generates SDP pipeline code with no guidance. The output is functional but lacks naming conventions, audit columns, and PII handling.

2. Skills

Add table-governance, sdp-basics, and pii-management skills to your workspace. Genie Code now applies documentation standards, naming conventions, audit columns, TBLPROPERTIES, column descriptions, and PII annotations automatically.

3. MCP + Instructions

Connect a GitHub MCP server that points to the same skills in this repo. Add custom instructions that tell Genie Code to fetch and apply them dynamically. The result is automatic compliance with organizational policies, maintained centrally in version control.

Quick Start

Prerequisites

A Databricks workspace with Genie Code enabled
Databricks CLI installed and configured
A GitHub account (for MCP setup)
Python 3.9+ with databricks-sdk installed (for MCP deploy script)

1. Install Skills

Copy the skill files from skills/data_eng/ to your Databricks workspace:

Workspace/
  .assistant/
    skills/
      table-governance.md
      sdp-basics.md
      pii-management.md

Skills can be installed at the workspace level (Workspace/.assistant/skills/) or user level (/Users/{username}/.assistant/skills/).

Once installed, Genie Code picks them up automatically. You can also invoke them explicitly with @table-governance, @sdp-basics, or @pii-management.

2. Set Up MCP (Optional)

Connect Genie Code to a GitHub MCP server to fetch the same skills dynamically -- no need to copy files into the workspace manually. The MCP connection points directly to the skills/data_eng/ folder in this repo (or your fork of it).

a. Configure

cd mcp/
cp mcp_config.example.json mcp_config.json
# Edit mcp_config.json with your GitHub org, repo, and secret scope details

b. Deploy

pip install databricks-sdk
python deploy_mcp.py

The script will:

Create a Databricks secret scope and store your GitHub PAT
Print the SQL to create the MCP connection

c. Create the connection

Copy the printed SQL and run it in a Databricks SQL Editor.

3. Add Instructions

The instructions/ folder contains templates with placeholders. To create ready-to-use instruction files for your workspace:

Set up your local deployment folder (see local_deployment/README.md)
Choose your approach (direct skills or MCP) and scope (user-level or workspace-level)
Copy the matching ready-to-use file from local_deployment/instructions_to_use/:

Scope	Direct Skills	MCP
User-level	`user_instructions_skills.md`	`user_instructions_mcp.md`
Workspace-level	`workspace_instructions_skills.md`	`workspace_instructions_mcp.md`

Fill in any remaining placeholders (GitHub org/repo for MCP files)
Upload to your workspace:
- User-level → /Users/{username}/.assistant_instructions.md
- Workspace-level → Workspace/.assistant_workspace_instructions.md

Workspace instructions take priority over user instructions when both are present.

Try It Yourself -- Sample Data

Generate synthetic financial data to test the skills end-to-end. The data generation notebook uses dbldatagen (Databricks Labs Data Generator) for Spark-native synthetic data generation.

Option A: Deploy with DAB (Recommended)

Set up a local deployment folder with your workspace-specific config:

# See local_deployment/README.md for full setup instructions and examples
# Create databricks.yml at the repo root and resource configs in local_deployment/resources/

# Deploy and run from the repo root
databricks bundle deploy --profile <your-cli-profile>
databricks bundle run generate_financial_data --profile <your-cli-profile>

Option B: Run Interactively

Upload the notebook to your workspace manually and run it interactively -- set the catalog, schema, and volume widget values when prompted.

Build a Pipeline

With the sample data in your volume, use Genie Code:

"Create an SDP pipeline that reads the financial CSV files from /Volumes/{catalog}/{schema}/raw_data/ and builds a bronze-silver-gold medallion architecture."

Genie Code will apply the skills and standards automatically.

Run the Demo

For a guided walkthrough that progressively adds skills, instructions, and MCP, see docs/DEMO_SCRIPT.md. It includes specific prompts, setup/cleanup steps for each stage, and observation checklists showing what to look for.

Try It Yourself -- Bakehouse Sentiment Analysis

Stage 5 of the demo applies the new @sentiment-analysis skill to the AI/BI Bakehouse dataset from the Databricks Marketplace. The flow is:

Install the Marketplace share into a scratch catalog and mirror selected tables into a new schema inside your existing demo catalog. This keeps the demo data under one predictable namespace (<demo_catalog>.bakehouse.*).
Use Genie Code with @sentiment-analysis to generate bronze/silver/gold tables that turn <demo_catalog>.bakehouse.customer_reviews into a sentiment dataset using AI functions (ai_analyze_sentiment, ai_classify, ai_extract).

1. Install the Bakehouse share + mirror into the demo catalog

Copy the config template and fill in your workspace details:

cp marketplace_data/bakehouse_config.example.yaml local_deployment/bakehouse_config.yaml
# Edit local_deployment/bakehouse_config.yaml:
#   workspace_host, profile,
#   scratch_catalog_name  (where the share lands -- pick a unique per-user name),
#   demo_catalog          (your existing demo catalog),
#   demo_schema           (default: bakehouse)

Then either deploy via DAB:

databricks bundle deploy --profile <your-cli-profile>
databricks bundle run install_bakehouse --profile <your-cli-profile>

…or run the standalone script:

./marketplace_data/deploy.sh --run

The notebook is idempotent: it skips the Marketplace install when scratch_catalog_name already exists, and uses CREATE OR REPLACE TABLE for the mirror so re-runs are safe.

2. Generate the pipeline with Genie Code

In a Genie Code session with the skills available (workspace-uploaded or via MCP):

Build a bronze, silver, and gold pipeline from <demo_catalog>.bakehouse.customer_reviews. Add sentiment, topic, and extracted entities on silver using AI functions, then aggregate sentiment by franchise on gold. Apply @table-governance, @sdp-basics, @pii-management, and @sentiment-analysis.

The result is a governed sentiment-analysis pipeline that follows every project standard: layer prefixes, audit columns, AI cost guardrails, sentiment label constraints, and PII annotations on the raw review_text.

Project Structure

genie-code-skills-demo/
├── .cursor/rules/                          # Cursor IDE rules
│   ├── public-repo-compliance.md
│   ├── coding-standards.md
│   └── branch-conventions.md
├── skills/
│   ├── data_eng/                           # Data engineering skills (also served via MCP)
│   │   ├── table-governance.md             # Table/column documentation, UC tags, PII labeling
│   │   ├── sdp-basics.md                   # SDP naming, audit columns, TBLPROPERTIES
│   │   └── pii-management.md               # PII detection and labelling
│   └── dsml/
│       └── sentiment-analysis.md           # AI-function patterns (ai_analyze_sentiment, ai_classify, ai_extract)
├── instructions/                           # Instruction TEMPLATES (with placeholders)
│   ├── .assistant_instructions.md          # User-level template
│   └── .assistant_workspace_instructions.md # Workspace-level template
├── mcp/
│   ├── mcp_config.example.json             # MCP config template (points to skills/ folder)
│   └── deploy_mcp.py                       # MCP connection deploy script
├── sample_data_gen/
│   ├── generate_financial_data.py          # Databricks notebook (dbldatagen, parameterized)
│   ├── deploy_config.example.yaml          # Deploy config template (placeholders)
│   └── deploy.sh                           # Deploy script (reads local config)
├── marketplace_data/
│   ├── install_bakehouse.py                # Databricks notebook (uses databricks-sdk Marketplace API)
│   ├── bakehouse_config.example.yaml       # Bakehouse install config template (placeholders)
│   └── deploy.sh                           # Deploy/run script for the install notebook
├── docs/
│   └── DEMO_SCRIPT.md                     # Five-stage guided demo walkthrough
├── local_deployment/                       # GITIGNORED (except README)
│   ├── README.md                           # How to set up your own local deployment
│   └── instructions_to_use/               # Ready-to-use instruction files (filled in)
│       ├── user_instructions_skills.md    # User-level, direct skills
│       ├── user_instructions_mcp.md       # User-level, MCP
│       ├── workspace_instructions_skills.md # Workspace-level, direct skills
│       └── workspace_instructions_mcp.md  # Workspace-level, MCP
├── README.md
├── LICENSE.md
├── NOTICE.md
├── SECURITY.md
├── CODEOWNERS
└── .gitignore

Customization

This repo is a starting point. To adapt it for your organization:

Add your own skills -- create new .md files in skills/data_eng/ or add new domain folders (e.g., skills/dsml/, skills/dashboards/)
Update existing skills -- edit the skill files in skills/data_eng/ to match your organization's naming conventions, PII policies, and quality rules
Fork and serve via MCP -- fork this repo, customize the skills, and point your MCP connection to your fork. Changes in GitHub are picked up automatically by Genie Code.
Customize instructions -- edit the templates in instructions/ to include your team's specific pipeline names, routing tables, and preferences

How to Get Help

Databricks support does not cover this content. For questions or bugs, please open a GitHub issue and the team will help on a best-effort basis.

License

© 2026 Databricks, Inc. All rights reserved. The source in this repository is provided subject to the Databricks License. All included or referenced third-party libraries are subject to the licenses set forth below.

Library	Description	License	Source
databricks-sdk	Databricks SDK for Python	Apache 2.0	PyPI
dbldatagen	Databricks Labs Data Generator	Apache 2.0	PyPI

Navigation Menu

Uh oh!

Repository files navigation

Genie Code Skills Demo

What is Genie Code?

Demo Coverage

What This Repo Contains

How It Works

1. Baseline

2. Skills

3. MCP + Instructions

Quick Start

Prerequisites

1. Install Skills

2. Set Up MCP (Optional)

3. Add Instructions

Try It Yourself -- Sample Data

Option A: Deploy with DAB (Recommended)

Option B: Run Interactively

Build a Pipeline

Run the Demo

Try It Yourself -- Bakehouse Sentiment Analysis

1. Install the Bakehouse share + mirror into the demo catalog

2. Generate the pipeline with Genie Code

Project Structure

Customization

How to Get Help

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.cursor/rules		.cursor/rules
docs		docs
instructions		instructions
local_deployment		local_deployment
marketplace_data		marketplace_data
mcp		mcp
sample_data_gen		sample_data_gen
skills		skills
.gitignore		.gitignore
CODEOWNERS		CODEOWNERS
LICENSE.md		LICENSE.md
NOTICE.md		NOTICE.md
README.md		README.md
SECURITY.md		SECURITY.md

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Genie Code Skills Demo

What is Genie Code?

Demo Coverage

What This Repo Contains

How It Works

1. Baseline

2. Skills

3. MCP + Instructions

Quick Start

Prerequisites

1. Install Skills

2. Set Up MCP (Optional)

3. Add Instructions

Try It Yourself -- Sample Data

Option A: Deploy with DAB (Recommended)

Option B: Run Interactively

Build a Pipeline

Run the Demo

Try It Yourself -- Bakehouse Sentiment Analysis

1. Install the Bakehouse share + mirror into the demo catalog

2. Generate the pipeline with Genie Code

Project Structure

Customization

How to Get Help

License

About

Resources

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages