Analytics and Prompt Evolution

Aludel combines a rolling operational dashboard with prompt-version analysis. The goal is to show whether quality improved and what that improvement cost in money, time, or stability.

Dashboard

The dashboard shows recent 7-day and 30-day windows plus lifetime context:

• total executions and recent activity
• pass rate and weighted quality
• total cost and cost per run
• average latency, P50, and P95
• cost and latency per passed test
• pass-rate standard deviation and stability sample size
• regression, improvement, and insufficient-data signals
• cost breakdown by provider or prompt
• latency breakdown by provider
• pass rates by prompt
• recent evaluations and quick actions

Interactive sections can be expanded without leaving the dashboard.

Prompt Evolution

Open a prompt and select Evolution. Every immutable version remains visible even when it has no suite data.

Choose overall or provider-specific charts for:

• pass rate
• structured-output score
• average cost
• average latency

The detailed breakdown includes suite-run counts, cost/latency efficiency, version-over-version deltas, stability, and regression signals.

Pareto Frontier

Select a suite to compare prompt versions on one consistent workload. A version is on the Pareto frontier when no other eligible version is at least as good on quality, cost, and latency and strictly better on one dimension.

Versions without enough quality, cost, or latency evidence are marked ineligible instead of being ranked using guessed values.

Example decision:

• version 3 has the best pass rate but costs twice as much
• version 4 keeps the same pass rate with lower latency
• version 4 dominates version 3 and remains on the frontier

The frontier does not choose a winner; it narrows the options that are not unambiguously worse.

Failure Reflection

When a selected suite has failed evidence:

1. choose a provider
2. generate a suggestion
3. review the proposed template, rationale, and summarized failures
4. accept or dismiss the suggestion

The generated template must preserve the source version's variables. Accepting creates a new immutable prompt version. Dismissing records the decision without changing the prompt. Only pending suggestions can transition.

Library API

Analyze one suite over a bounded period and review a generated revision through the same contracts used by the dashboard:

metrics = Aludel.Prompts.get_evolution_metrics(prompt.id, suite_id: suite.id, days: 30)
analysis = Aludel.Prompts.Optimization.analyze(metrics)

Enum.each(analysis.frontier, fn candidate ->
  IO.inspect(candidate.version_number, label: "frontier version")
end)

{:ok, suggestion} =
  Aludel.Prompts.Optimization.generate_suggestion(
    source_version.id,
    suite.id,
    provider.id
  )

{:ok, accepted} =
  Aludel.Prompts.Optimization.accept_suggestion(suggestion.id, prompt.id)

Use dismiss_suggestion/2 instead of accept_suggestion/2 to retain the review decision without creating a version.

Dashboard statistics are also directly queryable:

comparisons = Aludel.Stats.Overview.rolling_comparisons(DateTime.utc_now())
provider_costs = Aludel.Stats.Costs.cost_by_provider()
provider_latency = Aludel.Stats.Latency.latency_by_provider()
recent_activity = Aludel.Stats.Activity.list_recent_activity(20)

Export Evolution Data

Use the export menu to download JSON or CSV. Exports contain version metrics, deltas, signals, and provider breakdowns. CSV output quotes control characters and neutralizes spreadsheet formula prefixes in text fields.

Related Pages

• Prompts
• Evaluation Suites
• Exports and CI