Skip to content
box

GitHub Action

Test LLM outputs

v1 Latest version

Test LLM outputs

box

Test LLM outputs

Automatically run a before/after comparison of edited prompts and posts the result to your Pull Request

Installation

Copy and paste the following snippet into your .yml file.

 
              
- name: Test LLM outputs

              
  uses: promptfoo/promptfoo-action@v1
Learn more about this action in promptfoo/promptfoo-action

Choose a version

Github Action for LLM Prompt Evaluation

This Github Action uses promptfoo to produce a before/after view of edit prompts.

When you change a prompt, an eval will automatically be posted on the pull request:

pull request llm eval

The provided link opens the promptfoo web viewer, which allows you to interactively explore the before vs. after:

promptfoo web viewer

Configuration

The action can be configured using the following inputs:

Parameter Description Required
github-token The Github token. Used to authenticate requests to the Github API. Yes
prompts The glob patterns for the prompt files. These patterns are used to find the prompt files that the action should evaluate. Yes
config The path to the configuration file. This file contains settings for the action. Yes
cache-path The path to the cache. This is where the action stores temporary data. No
promptfoo-version The version of promptfoo to use. Defaults to latest No
no-share No sharing option for promptfoo. Defaults to false No
use-config-prompts Use prompt files set at config file. Defaults to false No

The following API key parameters are supported:

Parameter Description
openai-api-key The API key for OpenAI. Used to authenticate requests to the OpenAI API.
azure-api-key The API key for Azure OpenAI. Used to authenticate requests to the Azure OpenAI API.
anthropic-api-key The API key for Anthropic. Used to authenticate requests to the Anthropic API.
huggingface-api-key The API key for Hugging Face. Used to authenticate requests to the Hugging Face API.
aws-access-key-id The AWS access key ID. Used to authenticate requests to AWS services.
aws-secret-access-key The AWS secret access key. Used to authenticate requests to AWS services.
replicate-api-key The API key for Replicate. Used to authenticate requests to the Replicate API.
palm-api-key The API key for Palm. Used to authenticate requests to the Palm API.
vertex-api-key The API key for Vertex. Used to authenticate requests to the Vertex AI API.

Here is a generic Github Action configuration using "typpo/promptfoo-action@v1" with a cache step:

name: 'Prompt Evaluation'
on:
  pull_request:
    paths:
      - 'prompts/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read # Required for actions/checkout
      pull-requests: write # Ability to post comments on Pull Requests
    steps:
      # Required for promptfoo-action's git usage
      - uses: actions/checkout@v4

      # This cache is optional, but you'll save money and time by setting it up!
      - name: Set up promptfoo cache
        uses: actions/cache@v3
        with:
          path: ~/.cache/promptfoo
          key: ${{ runner.os }}-promptfoo-v1
          restore-keys: |
            ${{ runner.os }}-promptfoo-

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          prompts: 'prompts/**/*.json'
          config: 'prompts/promptfooconfig.yaml'
          cache-path: ~/.cache/promptfoo

If you are using an OpenAI model, remember to create the secret in Repository Settings > Secrets and Variables > Actions > New repository secret.

For more information on how to set up the promptfoo config, see documentation.