Revert "Revert "Add composite scores"" 😬

src/docs.json

@@ -842,6 +842,7 @@
                         "pages": [
                           "langsmith/code-evaluator",
                           "langsmith/llm-as-judge",
+                          "langsmith/composite-evaluators",
                           "langsmith/summary",
                           "langsmith/evaluate-pairwise"
                         ]

src/langsmith/composite-evaluators.mdx

@@ -0,0 +1,234 @@
+---
+title: Composite evaluators
+sidebarTitle: Composite evaluators
+---
+
+_Composite evaluators_ are a way to combine multiple evaluator scores into a single [score](/langsmith/evaluation-concepts#evaluator-outputs). This is useful when you want to evaluate multiple aspects of your application and combine the results into a single result.
+
+## Create a composite evaluator using the UI
+
+You can create composite evaluators on a [tracing project](/langsmith/observability-concepts#projects) (for [online evaluations](/langsmith/evaluation-concepts#online-evaluation)) or a [dataset](/langsmith/evaluation-concepts#datasets) (for [offline evaluations](/langsmith/evaluation-concepts#offline-evaluation)). With composite evaluators in the UI, you can compute a weighted average or weighted sum of multiple evaluator scores, with configurable weights.
+
+<div style={{ textAlign: 'center' }}>
+<img
+    className="block dark:hidden"
+    src="/langsmith/images/create_composite_evaluator-light.png"
+    alt="LangSmith UI showing an LLM call trace called ChatOpenAI with a system and human input followed by an AI Output."
+/>
+
+<img
+    className="hidden dark:block"
+    src="/langsmith/images/create_composite_evaluator-dark.png"
+    alt="LangSmith UI showing an LLM call trace called ChatOpenAI with a system and human input followed by an AI Output."
+/>
+</div>
+
+
+### 1. Navigate to the tracing project or dataset
+
+To start configuring a composite evaluator, navigate to the **Tracing Projects** or **Dataset & Experiments** tab and select a project or dataset.
+- From within a tracing project: **+ New** > **Evaluator** > **Composite score**
+- From within a dataset: **+ Evaluator** > **Composite score**
+
+### 2. Configure the composite evaluator
+
+1. Name your evaluator.
+2. Select an aggregation method, either **Average** or **Sum**.
+    - **Average**: ∑(weight*score) / ∑(weight).
+    - **Sum**: ∑(weight*score).
+3. Add the feedback keys you want to include in the composite score.
+4. Add the weights for the feedback keys. By default, the weights are equal for each feedback key. Adjust the weights to increase or decrease the importance of specific feedback keys in the final score.
+5. Click **Create** to save the evaluator.
+
+<Tip> If you need to adjust the weights for the composite scores, they can be updated after the evaluator is created. The resulting scores will be updated for all runs that have the evaluator configured. </Tip>
+
+### 3. View composite evaluator results
+
+Composite scores are attached to a run as **feedback**, similar feedback from a single evaluator. How you can view them depends on where the evaluation was run:
+
+**On a tracing project**:
+- Composite scores appear as feedback on runs.
+- [Filter for runs](/langsmith/filter-traces-in-application) with a composite score, or where the composite score meets a certain threshold.
+- [Create a chart](/langsmith/dashboards#custom-dashboards) to visualize trends in the composite score over time.
+
+**On a dataset**:
+- View the composite scores in the experiments tab. You can also filter and sort experiments based on the average composite score of their runs.
+- Click into an experiment to view the composite score for each run.
+
+<Note> If any of the constituent evaluators are not configured on the run, the composite score will not be calculated for that run. </Note>
+
+## Create composite feedback with the SDK
+
+This guide describes setting up an evaluation that uses multiple evaluators and combines their scores with a custom aggregation function.
+
+### 1. Configure evaluators on a dataset
+Start by configuring your evaluators. In this example, the application generates a tweet from a blog introduction and uses three evaluators — summary, tone, and formatting — to assess the output.
+
+If you already have your own dataset with evaluators configured, you can skip this step.
+
+<Accordion title="Configure evaluators on a dataset.">
+
+```python
+# Import dependencies
+import os
+from dotenv import load_dotenv
+from langsmith import traceable, wrappers
+from openai import OpenAI
+from langsmith import Client
+from pydantic import BaseModel
+import json
+
+# Load environment variables from .env file
+load_dotenv()
+
+# Access environment variables
+openai_api_key = os.getenv('OPENAI_API_KEY')
+langsmith_api_key = os.getenv('LANGSMITH_API_KEY')
+langsmith_project = os.getenv('LANGSMITH_PROJECT', 'default')
+
+
+# Create a dataset. Only need to do this once.
+client = Client()
+
+examples = [
+  {
+    "inputs": {"blog_intro": "Today we’re excited to announce the general availability of LangGraph Platform — our purpose-built infrastructure and management layer for deploying and scaling long-running, stateful agents. Since our beta last June, nearly 400 companies have used LangGraph Platform to deploy their agents into production. Agent deployment is the next hard hurdle for shipping reliable agents, and LangGraph Platform dramatically lowers this barrier with: 1-click deployment to go live in minutes, 30 API endpoints for designing custom user experiences that fit any interaction pattern, Horizontal scaling to handle bursty, long-running traffic, A persistence layer to support memory, conversational history, and async collaboration with human-in-the-loop or multi-agent workflows, Native LangGraph Studio, the agent IDE, for easy debugging, visibility, and iteration "},
+  },
+  {
+    "inputs": {"blog_intro": "Klarna has reshaped global commerce with its consumer-centric, AI-powered payment and shopping solutions. With over 85 million active users and 2.5 million daily transactions on its platform, Klarna is a fintech leader that simplifies shopping while empowering consumers with smarter, more flexible financial solutions. Klarna’s flagship AI Assistant is revolutionizing the shopping and payments experience. Built on LangGraph and powered by LangSmith, the AI Assistant handles tasks ranging from customer payments, to refunds, to other payment escalations. With 2.5 million conversations to date, the AI Assistant is more than just a chatbot; it’s a transformative agent that performs the work equivalent of 700 full-time staff, delivering results quickly and improving company efficiency."},
+  },
+]
+
+dataset = client.create_dataset(dataset_name="Blog Intros")
+client.create_examples(
+  dataset_id=dataset.id,
+  examples=examples,
+)
+
+# Define a target function. In this case, we're using a simple function that generates a tweet from a blog intro.
+def generate_tweet(inputs: dict) -> dict:
+    instructions = (
+      "Given the blog introduction, please generate a catchy yet professional tweet that can be used to promote the blog post on social media. Summarize the key point of the blog post in the tweet. Use emojis in a tasteful manner."
+    )
+    messages = [
+        {"role": "system", "content": instructions},
+        {"role": "user", "content": inputs["blog_intro"]},
+    ]
+    result = oai_client.responses.create(
+        input=messages, model="gpt-5-nano"
+    )
+    return {"tweet": result.output_text}
+
+# Define evaluators. In this case, we're using three evaluators: summary, formatting, and tone.
+def summary(inputs: dict, outputs: dict) -> bool:
+    """Judge whether the tweet is a good summary of the blog intro."""
+    instructions = "Given the following text and summary, determine if the summary is a good summary of the text."
+
+    class Response(BaseModel):
+        summary: bool
+
+    msg = f"Question: {inputs['blog_intro']}\nAnswer: {outputs['tweet']}"
+    response = oai_client.responses.parse(
+        model="gpt-5-nano",
+        input=[{"role": "system", "content": instructions,}, {"role": "user", "content": msg}],
+        text_format=Response
+    )
+
+    parsed_response = json.loads(response.output_text)
+    return parsed_response["summary"]
+
+def formatting(inputs: dict, outputs: dict) -> bool:
+    """Judge whether the tweet is formatted for easy human readability."""
+    instructions = "Given the following text, determine if it is formatted well so that a human can easily read it. Pay particular attention to spacing and punctuation."
+
+    class Response(BaseModel):
+        formatting: bool
+
+    msg = f"{outputs['tweet']}"
+    response = oai_client.responses.parse(
+        model="gpt-5-nano",
+        input=[{"role": "system", "content": instructions,}, {"role": "user", "content": msg}],
+        text_format=Response
+    )
+
+    parsed_response = json.loads(response.output_text)
+    return parsed_response["formatting"]
+
+def tone(inputs: dict, outputs: dict) -> bool:
+    """Judge whether the tweet's tone is informative, friendly, and engaging."""
+    instructions = "Given the following text, determine if the tweet is informative, yet friendly and engaging."
+
+    class Response(BaseModel):
+        tone: bool
+
+    msg = f"{outputs['tweet']}"
+    response = oai_client.responses.parse(
+        model="gpt-5-nano",
+        input=[{"role": "system", "content": instructions,}, {"role": "user", "content": msg}],
+        text_format=Response
+    )
+    parsed_response = json.loads(response.output_text)
+    return parsed_response["tone"]
+
+# Calling evaluate() with the dataset, target function, and evaluators.
+results = client.evaluate(
+    generate_tweet,
+    data=dataset.name,
+    evaluators=[summary, tone, formatting],
+    experiment_prefix="gpt-5-nano",
+)
+```
+</Accordion>
+
+### 2. Create composite feedback
+
+Create composite feedabck that aggregates the individual evaluator scores using your custom function. This example uses a weighted average of the individual evaluator scores.
+
+<Accordion title="Create a composite feedback.">
+
+```python
+from typing import Dict
+import math
+import pandas as pd
+
+# Set weights for the individual evaluator scores
+DEFAULT_WEIGHTS: Dict[str, float] = {
+    "feedback.summary": 0.7,
+    "feedback.tone": 0.2,
+    "feedback.formatting": 0.1,
+}
+WEIGHTED_KEY = "weighted_summary"
+
+# Pull experiment results
+EXPERIMENT_ID = list(client.list_projects(reference_dataset_name=DATASET_NAME, limit=1))[0].id
+df = client.get_test_results(project_id=EXPERIMENT_ID)
+
+# Skip rows with any missing metric
+required = list(DEFAULT_WEIGHTS.keys())
+missing_cols = [c for c in required if c not in df.columns]
+if missing_cols:
+    raise ValueError(f"Missing columns in DataFrame: {missing_cols}")
+
+# Calculate weighted score. This can be any aggregation function
+def row_score(row: pd.Series) -> float:
+    if row[required].isna().any():
+        return float("nan")
+    return float(sum(row[c] * DEFAULT_WEIGHTS[c] for c in required))
+
+df[WEIGHTED_KEY] = df.apply(row_score, axis=1)
+
+# Write feedback back to LangSmith
+for _, row in df.iterrows():
+    run_id = row["id"]
+    score = row[WEIGHTED_KEY]
+    if isinstance(score, (int, float)) and not math.isnan(score):
+        client.create_feedback(
+            run_id=run_id,
+            key=WEIGHTED_KEY,
+            score=float(score)
+        )
+```
+</Accordion>
+
+
+