Skip to content

Commit

Permalink
feat(aim): See description
Browse files Browse the repository at this point in the history
- Finalized first draft content forr drop rules
- fixed install link in yml
- Updated some copy in intro doc
  • Loading branch information
akristen committed Feb 5, 2024
1 parent 58552d9 commit 5e48b20
Show file tree
Hide file tree
Showing 3 changed files with 52 additions and 25 deletions.
63 changes: 44 additions & 19 deletions src/content/docs/ai-observability/drop-sensitive-data.mdx
Expand Up @@ -6,9 +6,9 @@ freshnessValidatedDate: never

import apmAimDropFilters from 'images/apm_screenshot-crop_aim-drop-filters.webp'

AI Monitoring captures event data about how your AI assistant responds to prompts. This event data can contain metrics about latency,
AI Monitoring captures event data about how your AI assistant responds to prompts. When an end user interacts with an AI assistant, prompts and responses are recorded as strings inside event attributes. You may find that some prompts or responses contain information you won't want stored in NRDB.

After AI Monitoring captures event data, you can direct the agent to drop that data so it isn't stored in NRDB. Drop filters apply regex to NRQL queries so AI Monitoring can match data to the query, then drop it. This doc walks you through how to set up drop filters.
With drop filters, you can apply regex to NRQL queries so AI Monitoring can match data to the query, then drop it if needed.

<img
title="Drop filters page"
Expand All @@ -17,12 +17,12 @@ After AI Monitoring captures event data, you can direct the agent to drop that d
/>

<figcaption>
To find this page, go to **[one.newrelic.com](https://one.newrelic.com) > All capabilities > AI Monitoring > Drop filters**
Go to **[one.newrelic.com](https://one.newrelic.com) > All capabilities > AI Monitoring > Drop filters**
</figcaption>

## Build your drop filters [#sec-compliance]

An AI assistant's message to an end user can be stored in NRDB under five separate event types:
Prompts and responses are stored as attributes under five separate event types:

<table>

Expand Down Expand Up @@ -71,11 +71,25 @@ An AI assistant's message to an end user can be stored in NRDB under five separa

</table>

When you create a drop filter for one kind of sensitive data, it's critical that you apply the rule to the four other events. This ensures that no artifact of that sensitve data persists in our database.
Let's say that you want to drop any prompts or responses that reference account information, like a shipping address. Creating drop filters for this type of sensitive information requires four phases:

1. Identify the information you want to filter, then find the regex for it.
2. Take a NRQL query for an event type and apply regex to the query.
3. Repeat step two for the four other event types, totaling five separate queries.
4. Repeat steps one through three for other kinds of information you want dropped.

## Cautions: Drop rules and data compliance [#caution]

When creating drop rules, you're responsible for ensuring that the rules accurately identify and discard the data that meets the conditions you've established. You're also responsible for monitoring the rule, as well as the data you disclose to New Relic.

* New Relic can't guarantee that this functionality completely resolves your data disclosure concerns.
* New Relic doesn't review or monitor how effective your drop rules are.

Keep in mind that creating rules about sensitive data can leak information about the kinds of data you maintain, including the format of your data or systems (for example, through referencing email addresses or specific credit card numbers). Any user with the relevant role-based access control permissions can view and edit all information in the rules you create.

## Create drop filters [#create]

Follow these procedures when creating a drop filter. Let's say you want to drop any instance of a customer's personal address whenever it appears in a prompt or response.
The below procedures give you an example workflow for dropping information from all event types.

<Steps>
<Step>
Expand All @@ -89,36 +103,43 @@ Follow these procedures when creating a drop filter. Let's say you want to drop

## Prepare your NRQL query [#nrql]

All drop rules build off the same NRQL query:

```sql
SELECT content FROM <EVENT_TYPE_HERE> WHERE content RLIKE <YOUR_REGEX_HERE>
```

Let's say you want to create a filter for a shipping address. Your first query would look like this:

```sql
SELECT content FROM LlmChatCompletion WHERE content RLIKE r''.*/\d{1,}(\s{1}\w{1,})(\s{1}?\w{1,})/g
```

</Step>
<Step>

## Add drop filter to four additional tables

Drop filters use NRQL queries to drop sensitive data. Here are three examples of a NRQL query using regex to locate and drop sensitive data:
## Add drop filter to four additional events

* EXAMPLE
* EXAMPLE
* EXAMPLE
Since your first query used `LlmChatCompletion`, you need to update the `FROM` clause for the other events:

There's a secion at the end of this procedure with regex samples to get you started with drop filters.

New Relic stores its data in five different tables. When you create a drop filter to drop one kind of data (say, a birth date), you need to repeat the process four additional times to account for those five tables.
* `LlmTool`
* `LlmChain events`
* `LlmChatCompletionSummary`
* `LlmChatCompletionMessage`

</Step>
<Step>

## Test your drop filter
## Repeat for other types of information.

To test your drop filter, ELEPHANT
You've now dropped address information from all five event types. You can refer to the sample regex section to create more sets of drop rules for other information.

</Step>
</Steps>

## Sample expressions [#regex-ref]

REGEX INTRO HERE
We've provided some sample regex to help you create your drop rules. While not an exhaustive list, it covers some standard types of information you may want dropped.

<CollapserGroup>
<Collapser
Expand Down Expand Up @@ -266,4 +287,8 @@ REGEX INTRO HERE

## What's next? [#whats-next]

jhbhkjhkjhkjh
Now that you've secured your customer data, you can explore other AI Monitoring features:

* Explore your [AI responses](/docs/ai-observability/view-ai-data/ai-response-performance) or [compare model performance](/docs/ai-observability/view-ai-data/compare-models) in New Relic.
* Update your app code to [correlate user feedback to AI responses](/docs/ai-observability/user-feedback).
* If you've enabled logging, [create drop rules for log messages](/docs/logs/ui-data/drop-data-drop-filter-rules).
6 changes: 4 additions & 2 deletions src/content/docs/ai-observability/intro-to-ai-monitoring.mdx
Expand Up @@ -6,7 +6,7 @@ freshnessValidatedDate: never

import apmAiMonitoringAllCapabilities from 'images/apm_screenshot-full_ai-monitoring-all-capabilities.gif'

Your AI-powered app introduces new technologies into your environment that are hard to monitor, let alone understand. AI Monitoring bridges unfamiliar technologies with familiar solutions, deepening your knowledge about your AI's behavior.
Your AI-powered app introduces new technologies into your environment that are hard to monitor, let alone understand. With AI Monitoring, you can bridge unfamiliar technologies to familiar solutions, deepening your knowledge about your AI's behavior so you can improve its performance.

<img
title="View your app's AI data"
Expand All @@ -31,7 +31,9 @@ AI Monitoring exposes your AI data as a subset of APM data. To get started, you

## Check app compatibility [#compatibility]

AI Monitoring supports some languages and some libraries. Refer to the below table to determine if your AI-powered app is compatible with your AI-powered app. For example:
AI Monitoring supports some languages and some libraries. Refer to the below table to determine if your AI-powered app is compatible with your AI-powered app.

For example:

* Our Python agent supports instrumentation for Amazon Bedrock, OpenAI, and LangChain libraries.
* Our Java agent supports instrumentation for Amazon Bedrock, but not OpenAI or LangChain.
Expand Down
8 changes: 4 additions & 4 deletions src/nav/ai-observability.yml
Expand Up @@ -4,17 +4,17 @@ pages:
- title: Introduction to AI Monitoring
path: /docs/ai-observability/intro-to-ai-monitoring
- title: Install AI Monitoring
path: /docs/install/aim
- title: Configure APM agents for AI Monitoring
path: /docs/ai-observability/configure-for-aim
path: /install/aim
- title: View AI data
pages:
- title: Evaluate AI response data
path: /docs/ai-observability/view-ai-data/ai-response-performance
- title: Compare performance between models
path: /docs/ai-observability/view-ai-data/compare-models
- title: Configure APM agents for AI Monitoring
path: /docs/ai-observability/configure-for-aim
- title: Correlate user feedback with AI responses
path: /docs/ai-observability/user-feedback
- title: Filter out sensitive data
- title: Filter out sensitive information
path: /docs/ai-observability/drop-sensitive-data

0 comments on commit 5e48b20

Please sign in to comment.