Fabric Data Agent CI/CD Testing Template

Automated testing framework for Microsoft Fabric Data Agents using PromptFoo with Azure DevOps CI/CD integration.

Template Repository: This repository is designed to be copied and customized for your own Fabric Data Agent projects. Test cases live in testing/promptfooconfig.yaml, and generated results are written to testing/test_results/.

Quick Start

# 1. Clone or copy this repository
git clone https://github.com/clientfirsttech/fabric-data-agent-testing-template my-fabric-agent-testing
cd my-fabric-agent-testing

# 2. Install dependencies
npm install

# 3. Configure your Data Agent (see Configuration section)
cp .env.example .env
# Edit .env with your Service Principal credentials
# Edit testing/promptfooconfig.yaml with your Data Agent URLs

# 4. Run tests locally
npm test

Prerequisites

Required Software

Software	Version	Purpose
Node.js	22.22.2	PromptFoo runtime
Git	Latest	Version control

Required Access

Microsoft Fabric Workspace with Data Agent(s) deployed
Azure AD / Entra ID application registration (Service Principal)
Access to underlying data sources used by the Data Agent

Getting Started -- Clone to a Public Repository

This template can be imported into a public repository.

To keep clean history and preserve branches/tags, mirror it instead of manually copying files.

Option A: Import via the CLI

# 1. Create a new public repo on GitHub (do NOT initialize it with a README)
gh repo create <your-public-repo-name> --public --confirm

# 2. Mirror-clone this template
git clone --bare https://github.com/clientfirsttech/fabric-data-agent-testing-template.git
cd fabric-data-agent-testing-template.git

# 3. Push to your new public repository
git push --mirror https://github.com/<your-org-or-user>/<your-public-repo-name>.git

# 4. Clone your public repository for day-to-day work
cd ..
git clone https://github.com/<your-org-or-user>/<your-public-repo-name>.git
cd <your-public-repo-name>

After this, continue with either Azure DevOps or GitHub Actions setup below.

Azure DevOps Setup

This section explains how to set up Azure DevOps CI/CD pipelines for automated testing of your Fabric Data Agents.

Step 1: Create Azure DevOps Project

Go to Azure DevOps
Create a new project or use an existing one
Navigate to Repos and import or connect your public copy of this repository (see Getting Started -- Clone to a Public Repository)

Step 2: Create Variable Group

The pipeline requires secrets stored in a Variable Group. Create one with these steps:

Go to Pipelines > Library
Click + Variable group
Name it: Data Agent Variables
Add the following variables:

Variable	Value	Secret?
`FABRIC_CLIENT_ID`	Your Azure AD App Client ID	No
`FABRIC_CLIENT_SECRET`	Your Azure AD App Client Secret	Yes (click lock icon)
`FABRIC_TENANT_ID`	Your Azure AD Tenant ID	No
`FABRIC_SCOPE`	`https://analysis.windows.net/powerbi/api/.default`	No

Click Save

Step 3: Create the Pipeline

Go to Pipelines > Pipelines
Click New Pipeline
Select your repository source (Azure Repos Git, GitHub, etc.)
Select Existing Azure Pipelines YAML file
Select /pipelines/azure-pipelines.yml
Click Continue and then Run

Step 4: Authorize Variable Group

On first run, you may need to authorize the pipeline to access the Variable Group:

When the pipeline pauses for permission, click View
Click Permit to allow access to Data Agent Variables

Step 5: Configure Pipeline Triggers

The default pipeline triggers on changes to:

main and dev branches
Files matching artifacts/*.DataAgent/**

To customize triggers, edit pipelines/azure-pipelines.yml:

trigger:
  branches:
    include:
      - main
      - dev
      - feature/*  # Add more branches as needed
  paths:
    include:
      - 'artifacts/**'
      - 'testing/**'  # Also trigger on test changes

Pipeline Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Azure DevOps Pipeline                     │
├─────────────────────────────────────────────────────────────┤
│  1. Install Node.js 22.22.2                                 │
│  2. Install npm dependencies (npm ci)                       │
│  3. Run PromptFoo evaluations                               │
│  4. Parse results and log failures                          │
│  5. Publish test artifacts                                  │
└─────────────────────────────────────────────────────────────┘

Pipeline Output

Console: Test pass/fail summary
Build Artifacts: JSON results at promptfoo-results
Pipeline Errors: Each failed test appears as an Azure DevOps error

GitHub Actions Setup

This section explains how to set up GitHub Actions CI/CD for automated testing of your Fabric Data Agents. The workflow file is already included at .github/workflows/promptfoo.yml.

Step 1: Use Your Public Repository

Complete Getting Started -- Clone to a Public Repository first, then open your public repository on GitHub.

Step 2: Add Repository Secrets

The workflow requires secrets stored in your GitHub repository. Add them with these steps:

Go to your repository on GitHub
Navigate to Settings > Secrets and variables > Actions
Click New repository secret for each of the following:

Secret	Value
`FABRIC_CLIENT_ID`	Your Azure AD App Client ID
`FABRIC_CLIENT_SECRET`	Your Azure AD App Client Secret
`FABRIC_TENANT_ID`	Your Azure AD Tenant ID
`FABRIC_SCOPE`	`https://analysis.windows.net/powerbi/api/.default`

Click Add secret after entering each value

Step 3: Verify the Workflow File

The workflow file at .github/workflows/promptfoo.yml is already included in this template. No additional setup is required — GitHub Actions will automatically detect and run it.

Step 4: Configure Workflow Triggers

The default workflow triggers on pushes and pull requests to:

main and dev branches
Files matching artifacts/*.DataAgent/**

It also supports manual runs via workflow_dispatch.

To customize triggers, edit .github/workflows/promptfoo.yml:

on:
  push:
    branches:
      - main
      - dev
      - feature/*  # Add more branches as needed
    paths:
      - 'artifacts/**'
      - 'testing/**'  # Also trigger on test changes
  pull_request:
    branches:
      - main
      - dev
  workflow_dispatch:

Step 5: Run the Workflow

Automatic: Push a commit or open a pull request matching the configured triggers
Manual: Go to Actions > PromptFoo Evaluation > Run workflow

Workflow Architecture

┌─────────────────────────────────────────────────────────────┐
│                    GitHub Actions Workflow                    │
├─────────────────────────────────────────────────────────────┤
│  1. Checkout repository                                      │
│  2. Install Node.js 22.22.2 (with npm cache)                │
│  3. Install npm dependencies (npm ci)                        │
│  4. Run PromptFoo evaluations                                │
│  5. Parse results and log failures                           │
│  6. Upload test artifacts                                    │
└─────────────────────────────────────────────────────────────┘

Workflow Output

Console: Test pass/fail summary in the Actions run log
Annotations: Each failed test appears as an error annotation on the run
Artifacts: JSON results uploaded as promptfoo-results (retained per GitHub's default retention policy)

Service Principal Setup

The testing framework uses a Service Principal (Azure AD App Registration) to authenticate with the Fabric API.

Step 1: Create App Registration

Go to Azure Portal > Microsoft Entra ID > App registrations
Click + New registration
Configure:
- Name: Fabric-DataAgent-Testing (or your preferred name)
- Supported account types: Single tenant
- Redirect URI: Leave blank
Click Register
Note the Application (client) ID and Directory (tenant) ID

Step 2: Create Client Secret

In your app registration, go to Certificates & secrets
Click + New client secret
Add a description and set expiration
Click Add
Copy the secret value immediately - it won't be shown again

Step 3: Grant Fabric API Permissions

In your app registration, go to API permissions
Click + Add a permission
Select APIs my organization uses
Search for and select Power BI Service (or 00000009-0000-0000-c000-000000000000)
Select Delegated permissions or Application permissions:
- Workspace.Read.All
- Dataset.Read.All (if querying datasets)
- DataAgent.Execute.All
- Eventhouse.Read.All (if querying Eventhouse)
Click Add permissions
Click Grant admin consent (requires admin privileges)

Step 4: Grant Workspace Access

Critical Prerequisite: The Service Principal must have access to:

Fabric Workspace containing the Data Agent:
- Go to your Fabric Workspace > Manage access
- Add your Service Principal with Contributor or Member role
Underlying Data Sources ⚠️ REQUIRED - The Data Agent runs queries under the calling identity (SPN):
- Lakehouse: Add Service Principal to the Lakehouse with read access
- Warehouse: Grant SELECT permissions to required schemas/tables

Semantic Model / Dataset: Grant the Service Principal explicit access to the dataset used by the agent
KQL Database: Add Service Principal as a viewer
External Sources: Configure appropriate credentials/connections

Important: The Fabric Data Agent executes queries using the Service Principal's identity. The agent can only access and query data that the SPN has been explicitly granted access to. Without proper data source permissions, queries will fail even if the SPN can access the workspace.

Important limitations and requirements:

RLS/OLS-backed semantic models in Import mode are not supported in this testing scenario and will not work as expected with Service Principal execution.

For Direct Lake semantic models with Service Principal, configure a fixed identity cloud connection.

Reference: https://learn.microsoft.com/en-us/fabric/fundamentals/direct-lake-overview#:~:text=If%20you%27re%20using%20a%20service%20principal%2C%20you%20must%20use%20a%20fixed%20identity%20cloud%20connection.

Step 5: How Authentication Works

The testing provider automatically handles authentication using the client credentials flow (OAuth 2.0):

✅ Authenticates to Microsoft Entra ID with SPN credentials (client ID + secret)
✅ Requests access token for Fabric API scope: https://analysis.windows.net/powerbi/api/.default
✅ Uses token as bearer token when calling Fabric Data Agent API
✅ Token is automatically refreshed when expired (cached for performance)

You only need to configure the credentials - the provider handles all token management.

Step 6: Note Your Credentials

You'll need these values for configuration:

Value	Where to Find
`FABRIC_CLIENT_ID`	App registration > Overview > Application (client) ID
`FABRIC_CLIENT_SECRET`	App registration > Certificates & secrets > Client secret value
`FABRIC_TENANT_ID`	App registration > Overview > Directory (tenant) ID
`FABRIC_SCOPE`	Usually `https://analysis.windows.net/powerbi/api/.default`

Local Development Setup

Step 1: Clone the Repository

git clone <repository-url>
cd fabric-agent-ci-cd

Step 2: Install Node.js Dependencies

npm install

Step 3: Configure Environment Variables

# Copy the example environment file
cp .env.example .env

# Edit .env with your Service Principal credentials
# Windows:
notepad .env

# macOS/Linux:
nano .env

Fill in your .env file:

FABRIC_CLIENT_ID=your-client-id
FABRIC_CLIENT_SECRET=your-client-secret
FABRIC_TENANT_ID=your-tenant-id
FABRIC_SCOPE=https://analysis.windows.net/powerbi/api/.default

Step 4: Configure Data Agent URLs

Edit testing/promptfooconfig.yaml to add your Data Agent URLs:

providers:
  - id: "file://./provider.ts"
    config:
      fabric_urls:
        agent1: "https://api.fabric.microsoft.com/v1/workspaces/<workspace-id>/dataagents/<dataagent-id>/aiassistant/openai"

Running Tests

Run All Tests

cd testing
promptfoo eval

View Results in Browser

promptfoo view

Run with npm Scripts

From the project root:

# Run tests (clears cache first)
npm test

# Run tests using cache (faster, for iterative development)
npm run test:cached

# Run tests with watch mode
npm run test:watch

# View results
npm run test:view

Test Output

Console: Summary table showing pass/fail status
HTML Report: testing/test_results/promptfoo/latest_results.html
Logs: testing/fabric_provider.log

Quick PowerShell Query (SPN)

Use the helper script to sign in with the Service Principal from .env and run EVALUATE DateDim against a semantic model.

# Replace with your Fabric workspace and semantic model IDs
powershell -ExecutionPolicy Bypass -File .\scripts\query-datedim.ps1 `
  -WorkspaceId "<workspace-guid>" `
  -DatasetId "<semantic-model-guid>"

Notes:

The script reads FABRIC_CLIENT_ID, FABRIC_CLIENT_SECRET, and FABRIC_TENANT_ID from .env.
It uses the Power BI PowerShell module (MicrosoftPowerBIMgmt) and calls the Execute Queries API.
Install module once if needed: Install-Module MicrosoftPowerBIMgmt -Scope CurrentUser

Configuration

Adding or Updating Agents

Agents are configured in testing/promptfooconfig.yaml under the fabric_urls section:

providers:
  - id: "file://./provider.ts"
    label: "Fabric Data Agent"
    config:
      fabric_urls:
        agent1: "https://api.fabric.microsoft.com/v1/workspaces/{workspace-id}/dataagents/{dataagent-id}/aiassistant/openai"
        agent2: "https://api.fabric.microsoft.com/v1/workspaces/{workspace-id}/dataagents/{dataagent-id}/aiassistant/openai"
        # Add more agents as needed:
        agent3: "https://api.fabric.microsoft.com/v1/workspaces/..."

Each key (e.g., agent1, agent2) is referenced in your test data to route questions to the correct agent.

Finding the Workspace ID and Agent (Skill) ID

To get the IDs needed for the agent URL, open your Data Agent in the Fabric workspace and look at the browser URL:

https://app.fabric.microsoft.com/groups/{workspace-id}/dataagents/{dataagent-id}

URL Component	Description	Example
`{workspace-id}`	GUID after `/groups/` or `/workspaces/`	`86dcc410-06b4-46e4-ab9d-bcc849af41a8`
`{dataagent-id}`	GUID after `/dataagents/`	`01f0e400-7fc3-4dc9-9152-caa46b933fdd`

Then construct the API URL:

https://api.fabric.microsoft.com/v1/workspaces/{workspace-id}/dataagents/{dataagent-id}/aiassistant/openai

The testing/promptfooconfig.yaml file also controls:

Timeouts: API call and provider timeouts
Retries: Number of retry attempts on failure

Adding Test Cases

Test cases are defined directly in testing/promptfooconfig.yaml under the tests section:

tests:
  - description: "Test relevant data question"
    vars:
      fabric_url_key: agent1  # Matches key in fabric_urls
      question: "What are the top 5 errors?"
    assert:
      - type: javascript
        value: "output.length > 0"

  - description: "Test irrelevant question rejection"
    vars:
      fabric_url_key: agent1
      question: "Tell me a joke"
    assert:
      - type: javascript
        value: "output.includes('not relevant')"

Multi-Turn Conversations (Follow-up Questions)

The provider now supports conversation memory across test turns.

Use these variables in your test cases:

conversation_id: Reuses the same Fabric thread across tests with the same ID
reset_conversation (optional): Set to true on the first turn to force a fresh thread

Example with your scenario:

tests:
  - description: "Turn 1: ask yearly total"
    vars:
      fabric_url_key: agent1
      conversation_id: chars-last-year
      reset_conversation: true
      question: "What was the total number of characters last year?"
    assert:
      - type: javascript
        value: "output.length > 0"

  - description: "Turn 2: follow-up by quarter"
    vars:
      fabric_url_key: agent1
      conversation_id: chars-last-year
      question: "How about by quarter"
    assert:
      - type: javascript
        value: "/quarter|q1|q2|q3|q4/i.test(output)"

How it works:

No conversation_id: single-turn behavior (thread is cleaned up after the call)
With conversation_id: thread is reused so follow-up prompts keep context

Assertion Types

PromptFoo supports various assertion types:

Assertion Type	Example	Description
`javascript`	`value: "output.length > 0"`	Custom JavaScript expression
`javascript`	`value: "output.includes('text')"`	Check output contains text
`javascript`	`value: "output.toLowerCase().includes('error')"`	Case-insensitive contains
`javascript`	`value: "JSON.parse(output).field === 'value'"`	Parse JSON and check field
`contains`	`value: "expected text"`	Simple contains check
`icontains`	`value: "expected text"`	Case-insensitive contains
`regex`	`value: "pattern"`	Match a regular expression

Learn more: See the PromptFoo Assertions Guide for all available assertion types.

See the PromptFoo Configuration Guide for advanced configuration options.

Troubleshooting

Common Issues

Issue	Solution
`401 Unauthorized`	Verify Service Principal credentials and workspace access
`403 Forbidden`	Grant admin consent for API permissions
`Data not found` errors	Service Principal needs access to underlying Lakehouse/Warehouse
Provider timeout	Increase `timeout` in `promptfooconfig.yaml`
`Cannot find module`	Run `npm install` in project root

Logs

Provider logs: testing/fabric_provider.log
PromptFoo logs: ~/.promptfoo/logs/

Using This Template

This repository is designed to be copied and customized for your own Fabric Data Agent testing needs.

Template Repository Structure

fabric-agent-ci-cd/
├── .env.example
├── .gitignore
├── package.json
├── README.md
├── .github/
│   └── workflows/
│       └── promptfoo.yml
├── pipelines/
│   └── azure-pipelines.yml
└── testing/
    ├── promptfooconfig.yaml
    ├── provider.ts
    └── test_results/

What to Customize

File	What to Change
`.env`	Your Service Principal credentials (copy from `.env.example`)
`testing/promptfooconfig.yaml`	Data Agent URLs (`fabric_urls`) and test cases (`tests`)
`pipelines/azure-pipelines.yml`	Branch triggers and paths for Azure DevOps (if needed)
`.github/workflows/promptfoo.yml`	Branch triggers and paths for GitHub Actions (if needed)

Files NOT to Commit

The following are automatically excluded via .gitignore:

Path	Reason
`artifacts/`	Workspace-specific Fabric artifacts
`node_modules/`	Node.js dependencies (regenerated via npm install)
`.env`	Contains secrets - use `.env.example` as template
`testing/test_results/`	Generated test output
`*.log`	Log files

Test Data Format

Tests are defined inline in testing/promptfooconfig.yaml under the tests section. This template does not use a separate CSV test file.

Steps to Use This Template

Clone the repository

git clone <this-repo-url> my-agent-testing
cd my-agent-testing

Configure credentials

cp .env.example .env
# Edit .env with your Service Principal values

Configure your Data Agent URLs

Edit testing/promptfooconfig.yaml:

providers:
  - id: "file://./provider.ts"
    config:
      fabric_urls:
        my_agent: "https://api.fabric.microsoft.com/v1/workspaces/YOUR-WORKSPACE-ID/dataagents/YOUR-DATAAGENT-ID/aiassistant/openai"

Add test cases

Edit testing/promptfooconfig.yaml and add tests:

tests:
  - description: "Test your question"
    vars:
      fabric_url_key: my_agent
      question: "Your test question here"
    assert:
      - type: javascript
        value: "output.length > 0"

Install dependencies and test locally

npm install
cd testing && promptfoo eval

Set up CI/CD — choose one or both:
- Azure DevOps: see Azure DevOps Setup above
- GitHub Actions: see GitHub Actions Setup above

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 16 Commits
.github/workflows		.github/workflows
pipelines		pipelines
testing		testing
.env.example		.env.example
.gitignore		.gitignore
LICENSE.txt		LICENSE.txt
README.md		README.md
package-lock.json		package-lock.json
package.json		package.json

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Fabric Data Agent CI/CD Testing Template

Quick Start

Table of Contents

Prerequisites

Required Software

Required Access

Getting Started -- Clone to a Public Repository

Option A: Import via the CLI

Azure DevOps Setup

Step 1: Create Azure DevOps Project

Step 2: Create Variable Group

Step 3: Create the Pipeline

Step 4: Authorize Variable Group

Step 5: Configure Pipeline Triggers

Pipeline Architecture

Pipeline Output

GitHub Actions Setup

Step 1: Use Your Public Repository

Step 2: Add Repository Secrets

Step 3: Verify the Workflow File

Step 4: Configure Workflow Triggers

Step 5: Run the Workflow

Workflow Architecture

Workflow Output

Service Principal Setup

Step 1: Create App Registration

Step 2: Create Client Secret

Step 3: Grant Fabric API Permissions

Step 4: Grant Workspace Access

Step 5: How Authentication Works

Step 6: Note Your Credentials

Local Development Setup

Step 1: Clone the Repository

Step 2: Install Node.js Dependencies

Step 3: Configure Environment Variables

Step 4: Configure Data Agent URLs

Running Tests

Run All Tests

View Results in Browser

Run with npm Scripts

Test Output

Quick PowerShell Query (SPN)

Configuration

Adding or Updating Agents

Finding the Workspace ID and Agent (Skill) ID

Adding Test Cases

Multi-Turn Conversations (Follow-up Questions)

Assertion Types

Troubleshooting

Common Issues

Logs

Using This Template

Template Repository Structure

What to Customize

Files NOT to Commit

Test Data Format

Steps to Use This Template

License

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages