Home

Welcome to the Aludel wiki!

Aludel is a Phoenix-native workbench for evaluating prompts with real visibility. It helps teams compare provider behavior, track prompt history, and turn important scenarios into repeatable regression suites.

flowchart LR
    A[Design Prompt] --> B[Run Across Providers]
    B --> C[Review Responses]
    C --> D[Validate With Test Suites]
    D --> E[Reuse Datasets and Gate CI]
    E --> F[Analyze and Improve Prompt Versions]

Loading

What Is Forged Here

• Multi-provider prompt execution from one workspace
• Native provider calls or app-callback execution against your real workflow
• Versioned prompt templates with {{variable}} interpolation
• Repeatable evaluation suites with string, field, deep JSON, and model-based rubric assertions
• Bounded repeated sampling with configurable pass reducers and complete attempt evidence
• Validated quality-policy authoring, immutable version history, and per-rule run evidence for pass rate, evaluator score, cost, and latency gates
• Native ExUnit assertions for generated output and persisted suite quality gates
• Versioned JSON and YAML suite manifests for scripts and CI
• Dashboard previews and downloads for console, versioned JSON, JUnit XML, and GitHub annotation reports, plus custom reporters through the API
• Reusable single-turn and multi-turn datasets with provenance and metadata filters
• Versioned curated red-team cases with deterministic canaries, optional judges, and idempotent dataset materialization
• Bounded generated adversarial candidates with review records, partial-failure reporting, budgets, and checksums
• Optional document attachments for suite test cases
• Separate typed project groupings for prompts and suites
• Rolling quality, cost, latency, stability, regression, and Pareto analysis
• Failure-grounded prompt suggestions with explicit acceptance or dismissal
• Copy actions, raw exports, and headless CI execution
• Configurable document storage backends for local development and cloud deployments

The Workbench

Get Started

• Installation - How to light the furnace: embedded, standalone, Docker, and storage setup
• Quick Start - The first working loop, from provider setup to results
• Providers - OpenAI, Anthropic, Google Gemini, Ollama, xAI, Groq, and OpenRouter
• Demo Data - Populate every major workflow with deterministic examples

Use the Features

• Prompts - Templates, projects, immutable versions, diffs, and suggestions
• Runs and Execution - Multi-provider runs, callback mode, status, and artifacts
• Evaluation Suites - Assertions, imports, documents, retries, and suite history
• Metric Context - Normalized prompt, output, document, provider, and execution evidence
• Evaluator Execution Details - Lifecycle status, timing, model identity, usage, cost, and safe errors
• Rubric Judges - Score semantic quality with custom criteria and a separate judge provider
• Judge Catalog - Versioned correctness, relevance, faithfulness, safety, refusal, PII, and hallucination templates
• Repeated Sampling - Reduce multiple model attempts into reliable suite decisions
• Quality Policies - Author, version, inspect, and automate quality, metadata-group, evaluator-score, cost, and latency gates
• ExUnit Evaluations - Assert inline output or execute and gate persisted suites from tests
• File-Based Suites - Run persisted suites from versioned JSON or YAML manifests
• Evaluation Reporters - Send one normalized result to local tools and CI systems
• Datasets - Reusable ordered examples, multi-turn messages, metadata, and provenance
• Red-Team Datasets - Browse and materialize curated adversarial cases with judges, provenance, and deduplication
• Generated Red-Team Cases - Generate, review, explicitly approve, and atomically import bounded cases
• Analytics and Prompt Evolution - Dashboard windows, deltas, signals, Pareto analysis, and reflection
• Exports and CI - JSON/CSV exports and mix aludel.eval
• Documents and Storage - File validation, conversion, local storage, S3, and GCS
• Embedding and Access - Router options, host authentication, resolvers, CSP, and read-only access

Understand the System

• Architecture - The chambers, conduits, and execution flow beneath the surface
• Data Model - Core entities, relationships, and storage choices
• Roadmap - Milestones, priorities, and the next stages of the workbench

Suggested Path

1. Installation
2. Quick Start
3. Providers
4. Evaluation Suites
5. Datasets
6. Analytics and Prompt Evolution
7. Embedding and Access
8. Architecture
9. Data Model

Community

• Issues
• Discussions