Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions datasets/automatically-from-traces.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ title: Automatically build datasets from real-time traces
description: Automatically build datasets from real-time traces to power LLM evaluations, regression tests, and AI agent testing workflows.
---

You can keep continously populating the dataset with new data arriving from production by using **Triggers**, mapping trace fields to any dataset columns you prefer.
You can keep continously populating the dataset with new data arriving from production by using **Automations**, mapping trace fields to any dataset columns you prefer.

Simply go to the Messages page and select a filter (for example, by model), the Add Trigger button will be enabled:
Simply go to the Messages page and select a filter (for example, by model), the Add Automation button will be enabled:

<div style={{ display: "flex", justifyContent: "center" }}>
<Frame caption="Add Trigger" style={{ maxWidth: "300px" }}>
<Frame caption="Add Automation" style={{ maxWidth: "300px" }}>
<img
className="block"
src="/images/add-trigger.png"
Expand All @@ -25,4 +25,4 @@ For Action, select **Add To Dataset**, and chose the right fields to map from th
alt="LangWatch"
/>
</Frame>
Hit save, and that's it! Now every time a new message matches the filter, the trigger will be fired and the dataset will be populated with the new row.
Hit save, and that's it! Now every time a new message matches the filter, the automation will be fired and the dataset will be populated with the new row.
2 changes: 1 addition & 1 deletion datasets/dataset-images.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ With the your images column type set to type set to `image (URL)`, you will be a
Below is an example of on how to set the column type to `image (URL)` in the screenshot below, you can also set this type when creating a new dataset.

<div style={{ display: "flex", justifyContent: "center" }}>
<Frame caption="Add Trigger" style={{ maxWidth: "300px" }}>
<Frame caption="Image column type" style={{ maxWidth: "300px" }}>
<img
className="block"
src="/images/dataset-image-select.png"
Expand Down
2 changes: 1 addition & 1 deletion datasets/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ To add data by selecting a group, simply click the "Add to Dataset" button after

### Continuously

You can keep continuously populating the dataset with new data arriving from production by using **Triggers**. See [Automatically building a dataset from traces](/datasets/automatically-from-traces) for more details.
You can keep continuously populating the dataset with new data arriving from production by using **Automations**. See [Automatically building a dataset from traces](/datasets/automatically-from-traces) for more details.


### CSV Upload
Expand Down
6 changes: 3 additions & 3 deletions evaluations/online-evaluation/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ trace.end();
Once enabled, scores will appear on:
- **Traces** - Individual trace scores visible in trace details
- **Analytics** - Aggregate metrics over time
- **Alerts** - Configure triggers for low scores
- **Alerts** - Configure automations for low scores

## Adding Scores via Code

Expand Down Expand Up @@ -169,10 +169,10 @@ See the full [Evaluators List](/evaluations/evaluators/list).
href="/evaluations/online-evaluation/setup-monitors"
/>
<Card
title="Triggers & Alerts"
title="Automations & Alerts"
description="Get notified when quality drops."
icon="bell"
href="/features/triggers"
href="/features/automations"
/>
<Card
title="Guardrails"
Expand Down
2 changes: 1 addition & 1 deletion evaluations/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ BUILD → TEST → DEPLOY → MONITOR
| Run quality checks in CI/CD | [Experiments CI/CD](/evaluations/experiments/ci-cd) |
| Monitor production quality over time | [Online Evaluation](/evaluations/online-evaluation/overview) |
| Block harmful or policy-violating content | [Guardrails](/evaluations/guardrails/overview) |
| Get alerts when quality drops | [Online Evaluation](/evaluations/online-evaluation/overview) + [Triggers](/features/triggers) |
| Get alerts when quality drops | [Online Evaluation](/evaluations/online-evaluation/overview) + [Automations](/features/automations) |

## Quick Start

Expand Down
6 changes: 3 additions & 3 deletions introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,10 @@ Ready to start taking control of your LLM application quality? Quick start with
horizontal
/>
<Card
title="Alerts & Triggers"
description="Set up alerts and triggers for your LLM applications."
title="Alerts & Automations"
description="Set up alerts and automations for your LLM applications."
icon="bell"
href="/features/triggers"
href="/features/automations"
arrow
horizontal
/>
Expand Down
20 changes: 10 additions & 10 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,10 @@ Ready to start taking control of your LLM application quality? Quick start with
horizontal
/>
<Card
title="Alerts & Triggers"
description="Set up alerts and triggers for your LLM applications."
title="Alerts & Automations"
description="Set up alerts and automations for your LLM applications."
icon="bell"
href="/features/triggers"
href="/features/automations"
arrow
horizontal
/>
Expand Down Expand Up @@ -261,10 +261,10 @@ See what's happening inside your LLM applications. LangWatch tracks every intera
horizontal
/>
<Card
title="Triggers & Alerts"
title="Automations & Alerts"
description="Get notified when things go wrong, or when costs spike unexpectedly."
icon="bell"
href="/features/triggers"
href="/features/automations"
horizontal
arrow
/>
Expand Down Expand Up @@ -13999,7 +13999,7 @@ title: Automatically build datasets from real-time traces
description: Automatically build datasets from real-time traces to power LLM evaluations, regression tests, and AI agent testing workflows.
---

You can keep continously populating the dataset with new data arriving from production by using **Triggers**, mapping trace fields to any dataset columns you prefer.
You can keep continously populating the dataset with new data arriving from production by using **Automations**, mapping trace fields to any dataset columns you prefer.

Simply go to the Messages page and select a filter (for example, by model), the Add Trigger button will be enabled:

Expand Down Expand Up @@ -14089,7 +14089,7 @@ To add data by selecting a group, simply click the "Add to Dataset" button after

### Continuously

You can keep continuously populating the dataset with new data arriving from production by using **Triggers**. See [Automatically building a dataset from traces](/datasets/automatically-from-traces) for more details.
You can keep continuously populating the dataset with new data arriving from production by using **Automations**. See [Automatically building a dataset from traces](/datasets/automatically-from-traces) for more details.


### CSV Upload
Expand Down Expand Up @@ -16391,10 +16391,10 @@ See the full [Evaluators List](/evaluations/evaluators/list).
href="/evaluations/online-evaluation/setup-monitors"
/>
<Card
title="Triggers & Alerts"
title="Automations & Alerts"
description="Get notified when quality drops."
icon="bell"
href="/features/triggers"
href="/features/automations"
/>
<Card
title="Guardrails"
Expand Down Expand Up @@ -17151,7 +17151,7 @@ BUILD → TEST → DEPLOY → MONITOR
| Run quality checks in CI/CD | [Experiments CI/CD](/evaluations/experiments/ci-cd) |
| Monitor production quality over time | [Online Evaluation](/evaluations/online-evaluation/overview) |
| Block harmful or policy-violating content | [Guardrails](/evaluations/guardrails/overview) |
| Get alerts when quality drops | [Online Evaluation](/evaluations/online-evaluation/overview) + [Triggers](/features/triggers) |
| Get alerts when quality drops | [Online Evaluation](/evaluations/online-evaluation/overview) + [Automations](/features/automations) |

## Quick Start

Expand Down
4 changes: 2 additions & 2 deletions observability/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,10 @@ See what's happening inside your LLM applications. LangWatch tracks every intera
horizontal
/>
<Card
title="Triggers & Alerts"
title="Automations & Alerts"
description="Get notified when things go wrong, or when costs spike unexpectedly."
icon="bell"
href="/features/triggers"
href="/features/automations"
horizontal
arrow
/>
Expand Down
6 changes: 3 additions & 3 deletions platform/rbac.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Permissions are applied to the following resource types:
| `annotations` | Data annotations and labeling |
| `evaluations` | Creating and managing evaluations |
| `datasets` | Managing datasets |
| `triggers` | Creating and managing triggers |
| `automations` | Creating and managing automations |
| `workflows` | Creating and managing workflows |
| `prompts` | Creating and managing prompts |

Expand All @@ -61,7 +61,7 @@ Full administrative access to all team resources and settings.
| Annotations | `view`, `manage` |
| Evaluations | `view`, `manage` |
| Datasets | `view`, `manage` |
| Triggers | `view`, `manage` |
| Automations | `view`, `manage` |
| Workflows | `view`, `manage` |
| Prompts | `view`, `manage` |
| Scenarios | `view`, `manage` |
Expand All @@ -81,7 +81,7 @@ Read-only access to most resources for observation and reporting purposes.
| Annotations | `view` |
| Evaluations | `view` |
| Datasets | `view` |
| Triggers | `view` |
| Automations | `view` |
| Workflows | `view` |
| Prompts | `view` |
| Scenarios | `view` |
Expand Down