GitHub Models with Power BI Development Example

This repository demonstrates how to use GitHub Models to automatically generate and validate AI-powered comments for Power BI semantic model TMDL files. It includes Promptfoo tests for prompt quality assurance and GitHub Actions automation for continuous validation.

Getting Started -- Import to Your Own Repository

Because this template uses GitHub Actions secrets (the MODELS_TOKEN), we recommend creating your own copy of this repository. GitHub does not allow forking a public repo as private, so use one of the options below to import it instead.

Option A: Import via the CLI

# 1. Create a new repository on GitHub (do NOT initialize it with a README)
gh repo create <your-username>/github-models-power-bi-dev-example --public --confirm
# Or use --private if you prefer a private repository

# 2. Clone this template repo as a bare repository
git clone --bare https://github.com/kerski/github-models-power-bi-dev-example.git
cd github-models-power-bi-dev-example.git

# 3. Push the mirror to your new repository
git push --mirror https://github.com/<your-username>/github-models-power-bi-dev-example.git

# 4. Clean up the bare clone
cd ..
rm -rf github-models-power-bi-dev-example.git

# 5. Clone your new repository and start working
git clone https://github.com/<your-username>/github-models-power-bi-dev-example.git
cd github-models-power-bi-dev-example

Option B: Import via the GitHub UI

1. Go to github.com/new/import.
2. Paste the source URL: https://github.com/kerski/github-models-power-bi-dev-example.
3. Set the Owner to your account or organization.
4. Choose a repository name (e.g., github-models-power-bi-dev-example).
5. Select Public or Private based on your needs.
6. Click Begin import and wait for it to finish.
7. Once complete, clone your new repository locally.

Once your repository is ready, continue with the Setup Guide below.

What's Included

• GitHub Actions Workflows (.github/workflows/): Automated TMDL commenting and Promptfoo test validation
• Prompt Definitions (prompts/): YAML-based prompt configurations for GitHub Models
• Promptfoo Tests (tests/promptfoo/): Evaluation framework for testing prompt quality
• Sample Semantic Models: Example Power BI models in TMDL format for testing

Repository Scope

• Semantic models (*.SemanticModel) and reports (*.Report) are source artifacts and should stay in place.
• Prompt and evaluation assets are under:
  • ./prompts
  • ./tests/promptfoo

Prerequisites

• A GitHub repository with admin access (to configure secrets and permissions)
• GitHub Actions must be enabled on your repository
• Access to GitHub Models (available to GitHub users)

Setup Guide

Step 1: Enable GitHub Models

Ensure GitHub Models is enabled for your organization and repository. No custom model deployment is needed — workflows use the standard mistral-ai/codestral-2501 model available via GitHub Models.

Step 2: Set Up the MODELS_TOKEN

The workflows require a GitHub personal access token with Models access:

1. Go to GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens.
2. Click Generate new token.
3. Give it a descriptive name (e.g., GitHub Models Access).
4. Set an appropriate expiration date.
5. Under Resource owner, select your account or organization.
6. Under Repository access, select the repository where you imported this template.
7. Expand Account permissions and set:
   • Models → Read-only (required for accessing GitHub Models)
8. Click Generate token and copy the value.
9. In your repository, go to Settings → Secrets and variables → Actions → Secrets.
10. Click New repository secret.
11. Set the name to MODELS_TOKEN and paste the token value.

Step 3: Grant GitHub Actions Permission to Create Pull Requests

The Comment TMDL Files workflow can optionally create pull requests with AI-generated comments:

1. Go to your repository Settings → Actions → General.
2. Scroll down to the Workflow permissions section.
3. Select Read and write permissions.
4. Check the box: Allow GitHub Actions to create and approve pull requests.
5. Click Save.

Note: The workflow only creates pull requests. A human reviewer is still required to review and approve before merging.

Step 4: Test the Workflows

You can trigger the workflows in multiple ways:

• Comment TMDL Files: Push changes to *.tmdl files, comment on a PR, or manually trigger via Actions
• Promptfoo Tests: Push changes to prompts/ or tests/promptfoo/, or manually trigger via Actions

Workflows included:

• .github/workflows/comment-tmdl.yml - AI-powered TMDL commenting
• .github/workflows/promptfoo-tests.yml - Prompt quality validation

Running Promptfoo Tests Locally

To run prompt evaluation tests locally:

Prerequisites

• Node.js 25 (matches CI; Node 18+ may work but CI is pinned to 25)
• npm
• PowerShell 7+ (pwsh)
• GitHub CLI (gh)
• GitHub CLI extension: Install with gh extension install github/gh-models
• A MODELS_TOKEN with models:read permission

Setup and Run

# Navigate to the promptfoo test directory
cd tests/promptfoo

# Install dependencies
npm ci

# Create local environment file
Copy-Item .env.example .env

# Set your MODELS_TOKEN in .env or in your current shell:
$env:MODELS_TOKEN = "ghp_your_token_here"

# Run tests
npm test

# View interactive results (optional)
npm run view

Help and AI enablement

For questions and support on Power Query linting and becoming AI-enabled, reach out via Power Query Lint:

• https://powerquerylint.com