feat: Public API for Experiment Runs + GitHub Action (Contribution Proposal)

[cpoepke]

Problem

Currently, experiments (prompt + dataset + model evaluation runs) can only be triggered through the Langfuse UI via a tRPC procedure (experimentsRouter.createExperiment in web/src/features/experiments/server/router.ts) or run via SDK's. There is no public API to programmatically trigger experiment runs, which means:

• CI/CD pipelines cannot automatically run experiments on code changes
• Scheduled/recurring evaluation runs are not possible without manual intervention
• Teams cannot integrate experiment execution into their existing automation workflows

Proposed Solution

Two deliverables:

1. A new public API endpoint that reuses the existing experiment execution pipeline with zero changes to the worker
2. A GitHub Action that wraps the API for one-off CI/CD triggered runs

POST /api/public/v2/datasets/{datasetName}/runs

Request Body

{
  "runName": "nightly-eval-2024-01-15",
  "promptName": "my-prompt",
  "promptVersion": "3",
  "description": "Nightly regression run",
  "modelConfig": {
    "provider": "openai",
    "model": "gpt-4o",
    "modelParams": {
      "temperature": 0,
      "max_tokens": 1024
    }
  },
  "structuredOutputSchema": {}
}

Field	Type	Required	Description
runName	string	Yes	Unique name for this experiment run
promptName	string	Yes	Name of the prompt to evaluate
promptVersion	string	No	Optional version of the prompt to use, default is latest (e.g. 3, latest)
description	string	No	Optional description (max 1000 chars)
modelConfig.provider	string	Yes	LLM provider (e.g., openai, anthropic)
modelConfig.model	string	Yes	Model identifier (e.g., gpt-4o)
modelConfig.modelParams	object	No	Model parameters (temperature, max_tokens, top_p, etc.)
structuredOutputSchema	object	No	JSON schema for structured output

Response

{
  "id": "clxyz...",
  "datasetId": "abc123",
  "datasetName": "my-eval-dataset",
  "runName": "nightly-eval-2024-01-15",
  "status": "queued",
  "createdAt": "2024-01-15T00:00:00.000Z"
}

Returns immediately with queued status. Callers poll GET /api/public/datasets/{name}/runs/{runName} for results.

Architecture

Public API endpoint (NEW)
    |
    v
Validate config, resolve prompt by name+version, create DatasetRuns record (EXISTING, no changes)
    |
    v
ExperimentCreateQueue (EXISTING, no changes)
    |
    v
experimentCreateQueueProcessor → createExperimentJobClickhouse (EXISTING, no changes)

Only new code: API endpoint, Zod schemas, and tests.

Implementation Plan

Step	File	Description
1	web/src/features/public-api/types/datasets.ts	Add Zod v4 schemas for request body and response
2	web/src/pages/api/public/datasets/[name]/runs/index.ts	Add POST handler alongside existing GET using withMiddlewares + createAuthedProjectAPIRoute
3	fern/apis/server/definition/datasets.yml	Add Fern API spec for the new endpoint
4	web/src/__tests__/async/experiment-run-api.servertest.ts	Integration tests

GitHub Action: langfuse/run-experiment

A standalone GitHub Action (separate repository) that wraps the new API endpoint.

- uses: langfuse/run-experiment@v1
  with:
    langfuse-host: https://cloud.langfuse.com
    langfuse-public-key: ${{ secrets.LANGFUSE_PUBLIC_KEY }}
    langfuse-secret-key: ${{ secrets.LANGFUSE_SECRET_KEY }}
    dataset-name: my-eval-dataset
    prompt-name: my-prompt
    prompt-version: 3
    model-provider: openai
    model-name: gpt-4o
    model-params: '{"temperature": 0}'
    run-name: "ci-${{ github.sha }}"
    wait-for-completion: true
    timeout-minutes: 30

Questions for Maintainers

1. RBAC scope: Reuse existing promptExperiments:CUD or create a new scope?
2. GitHub Action repo: Should the action live in a separate langfuse/run-experiment-action repository?

[hassiebp]

@cpoepke Thanks a lot for raising this!

CI/CD pipelines cannot automatically run experiments on code changes

What are the code changes here exactly that would trigger such a run? The on-platform prompt experiments are using prompt, dataset and model configuration that are all on the Langfuse platform for a run, no external code that can change that should trigger a run.

Generally, we would recommend here to use the Experiment runner SDK in order to create your own custom experiment pipeline that runs on your triggers: https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk#experiment-runner-sdk

[cpoepke]

Thank you @hassiebp !

The code change would be based on the system under eval, hence no trigger in Langfuse.

Indeed, the SDK's would be the fallback. The idea is here, for easier integration into CI/CD pipelines of software development and to measure quality of an overall AI system, to regularly run evals against them. Having an endpoint and a GitHub Action would lower the barrier to set up such feedback loops.

The wish would be to not replicate the setup already existing in Langfuse. Thinking here about, dataset management, prompt management and scalable test execution, which already exists and does not need to be replicated 1:1 in a pipeline, which also does not scale elastically.

[cpoepke]

@hassiebp , is it fine to go ahead with that proposal and to prepare a PR for this? Or are there any other open questions or recommendations?

Can also scope it down to just the endpoint and put the GitHub Action into another repository.

[wochinge]

Hi @cpoepke,

we just published a Langfuse GitHub Action for running experiments in CI/CD: https://github.com/langfuse/experiment-action 🎉
There’s also a short changelog post with context and examples: https://langfuse.com/changelog/2026-05-05-experiment-ci-cd-gates

Would be great if you could try it out when you have a chance and share any feedback 🙌🏻