Skip to content

Commit

Permalink
feat(aim): implemented reviews on ui doc, drop filters
Browse files Browse the repository at this point in the history
Update drop filters doc, updated language to include six events instead of four
  • Loading branch information
akristen committed Mar 18, 2024
1 parent 764a937 commit ee0bae9
Show file tree
Hide file tree
Showing 4 changed files with 21 additions and 30 deletions.
29 changes: 10 additions & 19 deletions src/content/docs/ai-monitoring/drop-sensitive-data.mdx
Expand Up @@ -6,14 +6,18 @@ freshnessValidatedDate: never

import aiDropFilterModal from 'images/ai_screenshot-crop_drop-filter-modal.webp'

import aiDropFilterModalCrop from 'images/ai_screenshot-crop_drop-filter-modal.webp'
import aiDropFilterTable from 'images/ai_screenshot-crop_drop-filter-table.webp'

Our agents collect and store your data in the New Relic database (NRDB). When an agent captures an interaction with your AI assistant, it stores the contents of those prompts and responses as text string attributes within data events. This gives you greater insight into the cost, accuracy, and performance of your AI-powered app, but you may not want to store interactions that contain sensitive data. We recommend creating drop filters.

Drop filters evaluate data sent by the agent and removes sensitive data before it's stored in NRDB. This lets you still analyze an interaction with your AI assistant without storing your customer's personal data in our databases.

## Create drop filters [#create-filter]

<Callout variant="tip">
If you've disabled `ai_monitoring.record_content.enabled`, event data containing end user prompts and AI responses will be dropped before they're stored in NRDB. You can read more about agent configurations at our [AI monitoring configuration doc](/docs/ai-monitoring/configure-ai-monitoring).
</Callout>

A single drop filter targets a specified attribute within one event type, but sensitive information from a single AI interaction is stored in multiple events. To ensure that information is completely dropped before it enters NRDB, you need four separate drop filters.

<img
Expand All @@ -23,7 +27,7 @@ A single drop filter targets a specified attribute within one event type, but se
/>

1. Go to **[one.newrelic.com](https://one.newrelic.com) > All capabilities > AI monitoring > Drop filters**, then click **Create drop filter**.
1. Create a filter name. Because one kind of data requires at least four drop filters, we recommend a naming convention that helps you track the events you're dropping data from.
1. Create a filter name. Because one kind of data requires at least six drop filters, we recommend a naming convention that helps you track the events you're dropping data from.
1. Paste in the base NRQL query:
```sql
SELECT <COLUMN> FROM <EVENT_TYPE> WHERE <COLUMN> RLIKE <REGEX>
Expand Down Expand Up @@ -62,27 +66,14 @@ In a typical AI interaction, a prompt or request undergoes certain processes (li
<img
title="Create drop filters for every event type"
alt="When you click Create a drop filter, a side modal appears to guide you through dropping text from events and attributes."
src={aiDropFilterModalCrop}
src={aiDropFilterTable}
/>

A complete set of drop filters for a piece of sensitive information should include queries for four events: `LlmChatCompletionMessage`, `LlmFeedbackMessage`, `LlmEmbedding`, and `LlmTool`. Your drop filters correspond to event and column pairs in a given row, as seen in the table. A few things to remember:
A complete set of drop filters for a piece of sensitive information should include queries for the six events provided in the drop filter table. Your drop filters correspond to event and column pairs in a given row. A few things to remember:

* In other words, the attribute `content` only appears in the `LlmChatCompletionMessage` event and not in the `LlmEmbedding` event. The attribute `messages` only appears in `LlmFeedbackMessage` but not in `LlmTool`.
* The column `content` appears in the `LlmChatCompletionMessage` event and not in the `LlmEmbedding` event.
* The column `messages` only appears in `LlmFeedbackMessage` but not in `LlmTool`.
* The exception to this rule is the attribute `input`, which appears in both `LlmEmbedding` and `LlmTool`.
* In the case of `LlmTool`, you need to create queries to account for two attributes `input` and `output`. For example:

```sql
SELECT output FROM LlmTool WHERE output RLIKE ^([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})$
```
```sql
SELECT input FROM LlmTool WHERE input RLIKE ^([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})$
```

Keep in mind that data can vary across app environments, and can affect how many drop filters you create. For example:

* If you use LangChain, you need to create filters for `LlmTool`.
* If you've updated your app code to correlate feedback with responses, you need to create filters for `LlmFeedbackMessage`.
* In most cases, you need to create two or more filters for `LlmChatCompletionMessage` for at least the user question and the model's response.

### Regex

Expand Down
22 changes: 11 additions & 11 deletions src/content/docs/ai-monitoring/view-ai-data.mdx
Expand Up @@ -26,7 +26,7 @@ import aiTraceWaterfallPageErrorDetails from 'images/ai_screenshot-crop_trace-wa

import aiAiModelComparisonPage from 'images/ai_screenshot-full_ai-model-comparison-page.webp'

Enabling AI monitoring allows the agent to recognize and capture performance metrics and trace data about your app's AI-layer. You can track token usage, number of completions, and response time from your AI-powered app. When you see an error or an inaccurate response, scope to a trace-level view on a given prompt-response interaction to identify problems in the logic of your AI service.
Enabling AI monitoring allows the agent to recognize and capture performance metrics and trace data about your app's AI-layer. With AI monitoring, you can track token usage, number of completions, and AI response time from your AI-powered app. When you see an error or an inaccurate response, you can scope to a trace-level view on a given prompt-response interaction to identify problems in the logic of your AI service.

<img
title="AI responses data"
Expand All @@ -37,8 +37,8 @@ Enabling AI monitoring allows the agent to recognize and capture performance met
You can view your data by going to **[one.newrelic.com](https://one.newrelic.com) > All Capabilities > AI monitoring**. You can see your data from three different pages:

* **AI responses**: Overview aggregated data from all your AI entities. Track your AI responses, times, and tokens, or see data about individual prompts and responses.
* **AI entities**: View a table summary of all entities reporting AI data. See entities with standard APM data like error rate, throughput, and app response time. When you select an entity, you can start exploring the APM **AI responses** page.
* **Compare models**: Compare token usage, response time, and error rate between different models. If you're conducting A/B tests, get all the information you need to make decisions about your AI-powered app.
* **AI entities**: View a table summary of all entities reporting AI data. See entities with standard APM data such as error rate, throughput, and app response time. When you select an entity, you can start exploring the APM **AI responses** page.
* **Compare models**: Compare token usage, response time, and error rate between different models. If you're conducting A/B tests, you can get all the information you need to make decisions about your AI-powered app.

## AI responses page [#ai-responses]

Expand All @@ -54,15 +54,15 @@ If you own several apps with various implementations of different AI frameworks,
src={aiTimeseriesBillboard}
/>

The three tiles show general performance metrics about your AI's responses. These tiles may not tell you the exact cause behind a problem, but they're useful for identifying anomalous fluctuations in your app's performance.
The three tiles show general performance metrics about your AI's responses. These tiles may not tell you the exact cause behind a problem, but they're useful for identifying anomalies in your app's performance.

<img
title="Three tiles to show spikes and drops in response usage"
alt="A cropped screenshot displaying billboard info about AI data"
src={aiCroppedImageofAIBillboards}
/>

* If you notice a drop in total responses or increase in average response time, it can indicate that some technology in your AI toolchain has malfunctioned and prevented your AI-powered app from posting a response.
* If you notice a drop in total responses or an increase in average response time, it can indicate that some technology in your AI toolchain has prevented your AI-powered app from posting a response.
* A drop or increase in average token usage per response can give you insight into how your model creates a response. Maybe it's pulling too much context, thus driving up token cost while generating its response. Maybe its responses are too spare, leading to lower token costs and unhelpful responses.

### Adjust the timeseries graphs
Expand All @@ -77,7 +77,7 @@ You can refer to the time series graphs to better visualize when an anamolous be

* Adjust the timeseries graph by dragging over a spike or drop. This scopes the timeseries to a certain time window.
* Select the drop down to run comparative analysis for different performance parameters. You can choose between total responses, average response time, or average tokens per response.
* If you've enabled the feedback feature, you can scope the graphs to analyze responses by positive and negative feedback.
* If you've enabled the [feedback feature](/docs/ai-monitoring/configure-ai-monitoring), you can scope the graphs to analyze responses by positive and negative feedback.

### Evaluate individual AI responses

Expand All @@ -89,9 +89,9 @@ Your AI response table organizes data about interactions between your end user a
src={aiResponseTable}
/>

You can adjust the table columns by clicking the cog icon in the upper right. This functionality lets you choose the kinds of data you want to analyze.
You can adjust the table columns by clicking the cog icon in the upper right. This lets you choose the kinds of data you want to analyze.

The Responses is an entry point into viewing trace data about an individual response. Click a row in the table to open the trace view of a particular response.
The response table is an entry point into viewing trace data about an individual response. Click a row in the table to open the trace view of a particular response.

### AI response trace view

Expand All @@ -101,7 +101,7 @@ The Responses is an entry point into viewing trace data about an individual resp
src={aiTraceViewAiResponse}
/>

The AI response trace view gives you trace-level insights into how your app generates responses. You may want to look at the trace view to identify where an error occurred, or maybe you want to understand what led to negative feedback from a high token response.
The AI response trace view gives you trace-level insights into how your app generates responses. You may want to look at the trace view to identify where an error occurred, or maybe you want to understand what led to negative feedback from a high token response. From the trave view, you can:

* Choose between traces or logs. When you select logs, query within logs for text strings or attributes you want to investigate further.
* Toggle between response details or metadata. The response details column shows the user prompt and AI response so you can maintain context for your traces and spans. Metadata provides a list view for entity GUID, model, tokens, and vendor.
Expand Down Expand Up @@ -144,13 +144,13 @@ Selecting an AI entity takes you to the APM summary page for that app. From the

### APM AI responses page [#apm-ai-response]

Selecting an AI entity takes you to the APM summary page. To find your AI data, choose **AI Responses** in the left nav. We recommend using this page when you've identified that a particular AI entity has contributed to anomalous behavior.
Selecting an AI entity takes you to the APM summary page. To find your AI data, choose **AI Responses** in the left nav. We recommend using this page when you've identified that a particular AI entity has contributed to anomalies.

* The APM version of AI responses contains the same tiles, timeseries graphs, and response tables collected as the top-level AI responses page.
* Instead of showing aggregated data, the APM AI responses page shows data scoped to the service you selected from AI entities.
* While the top-level AI responses page lets you filter by service across all AI entities, the APM AI responses page limits filter functionality to the app's own attributes.

To review how to explore your AI data, you can follow the same patterns explained in [AI responses section](#ai-responses) of this doc.
To review how to explore your AI data, you can follow the same patterns explained in the previous [AI responses section](#ai-responses).

## Model comparison page [#model-comparison]

Expand Down
Binary file modified src/images/ai_screenshot-crop_drop-filter-modal.webp
Binary file not shown.
Binary file not shown.

0 comments on commit ee0bae9

Please sign in to comment.