Skip to content

Commit

Permalink
feat(aim): see description
Browse files Browse the repository at this point in the history
- Added two new screenshots for the aggregated AI responses view option
- Added content to aggregated AI responses view
- Added Ruby install and config options to the install flow but not the config doc flow
  • Loading branch information
akristen committed Feb 16, 2024
1 parent 72d661a commit a3b049b
Show file tree
Hide file tree
Showing 9 changed files with 1,035 additions and 15 deletions.
Expand Up @@ -9,11 +9,6 @@ You can set limits with certain event attributes in your APM agent. These config
* Limits tell your agent how much to sample events or spans.
* A higher limit means less sampling, which gives a more complete picture of your app's performance, but also means higher ingest costs and higher memory usage.

<Callout variant="tip">
Keep in mind that if you disabled distributed tracing, you need to turn it back on for AI Monitoring:

`distributed_tracing.enabled = true`
</Callout>

## Adjust agent configuration settings [#adjust-AIM]

Expand Down
47 changes: 37 additions & 10 deletions src/content/docs/ai-observability/ai-response-performance.mdx
Expand Up @@ -6,6 +6,10 @@ freshnessValidatedDate: never

import apmAIEntitiesPage from 'images/apm_screenshot-crop_AI-entities-page.webp'

import aiTopleveAiResponsesSummary from 'images/ai_screenshot-crop_topleve-ai-responses-summary.webp'

import aiCorrelateTimeseries from 'images/ai_screenshot-crop_correlate-timeseries.webp'

import apmFindAiResponsesFromApm from 'images/apm_screenshot-crop_find-ai-responses-from-apm.webp'

import apmAiResponsesDefaultView from 'images/apm_screenshot-crop_ai-responses-default-view.webp'
Expand All @@ -18,7 +22,7 @@ With AI monitoring enabled, your agent can recognize and capture performance met

## Start with AI Entities [#ai-entities]

to see a table of all your AI-powered apps. From the AI Entities page, you have three entry points into viewing and analyzing your data:
to see a table of all your AI-powered apps. From the AI entities page, you have three entry points into viewing and analyzing your data:

<img
title="AI Entities page"
Expand All @@ -29,24 +33,50 @@ to see a table of all your AI-powered apps. From the AI Entities page, you have
Start exploring your AI data from the AI entities page: Go to **[one.newrelic.com](https://one.newrelic.com) > All Capabilities > AI Monitoring**, then choose one of our entry points to explore your data.
</figcaption>

* **AI Entities**: Check your app's performance at a glance with data about response time, throughput, and error rate. Choose an AI-powered app to view AI data scoped to that app.
* **AI Responses**: Overview aggregated data about all your AI entities. Check how many tokens all your AI apps use together, or monitor how your AI apps are performing in general.
* **Compare Models**: Compare token usage, response time, and error rate between different models. If you're conducting A/B tests, get all the information you need to make decisions about your AI-powered app.
* **AI entities**: Check your app's performance at a glance with data about response time, throughput, and error rate. Choose an AI-powered app to view AI data scoped to that app.
* **AI responses**: Overview aggregated data about all your AI entities. Update the filter bar to scope your data to specific services, models, or error types.
* **Compare models**: Compare token usage, response time, and error rate between different models. If you're conducting A/B tests, get all the information you need to make decisions about your AI-powered app.

<Tabs>
<TabsBar>
<TabsBarItem id="app-scoped">
View AI data scoped by application
</TabsBarItem>
<TabsBarItem id="aggregated-data">
Overview aggregated data about your AI apps
</TabsBarItem>
<TabsBarItem id="app-scoped">
View AI data scoped by application
</TabsBarItem>
<TabsBarItem id="compare-models">
Compare your models
</TabsBarItem>
</TabsBar>

<TabsPages>
<TabsPageItem id="aggregated-AI">

<img
title="AI Entities page"
alt="A screenshot of the first page you see when you click AI Monitoring. View aggregated data, compare your AI models, or create drop filters."
src={aiTopleveAiResponsesSummary}
/>
<figcaption>
View your aggregated AI data: Go to **[one.newrelic.com](https://one.newrelic.com) > All Capabilities > AI Monitoring**, then choose **AI Responses**.
</figcaption>

The top-level **AI responses** page shows your AI data in aggregate. Aggregated data takes the average total responses, response times, and token usage per response across all entities reporting AI data.

<img
title="Correlate time series graphs"
alt="A crropped screenshot that shows two time series graphs with data trends for feedback and response time"
src={aiCorrelateTimeseries}
/>

You can toggle the time series graphs to correlate AI behaviors with different metrics. For example, you might want to see if higher token usage correlates with higher response times from your AI. If you've enabled the feedback feature, you can see if negative or positive user feedback trends with response times.

## Scope your data with the filter bar [#filter-bar]

Let's say you own several apps with various implementations of different AI frameworks, like GPT-4 and Amazon BedRock. You can get a general sense for how the AI layers of your app are performing, then scope that data with attributes in the filter bar.

</TabsPageItem>
<TabsPageItem id="entity-scoped">

When you choose an AI entity, you'll be taken to the APM summary page. This page contains summary tiles and time series graphs that measure the health of your entire app. To find your AI data, choose **AI Responses** in the left nav:
Expand Down Expand Up @@ -107,9 +137,6 @@ The **Responses** table is an entry point into viewing trace data about an indiv



</TabsPageItem>
<TabsPageItem id="aggregated-AI">
find use case
</TabsPageItem>
<TabsPageItem id="compare-models">
add content
Expand Down
Binary file not shown.
Binary file not shown.
21 changes: 21 additions & 0 deletions src/install/aim/agent-lang/ruby-config.mdx
@@ -0,0 +1,21 @@
---
componentType: default
headingText: Configure the Ruby agent
freshnessValidatedDate: never
---

AI Monitoring introduces two new configuration options to the Ruby agent:

* `ai_monitoring.enabled`, default: `true`
Indicates whether LLM instrumentation will be registered. If this is set to `false`, no metrics, events, or spans related to AI Monitoring will be sent. Currently, this equates to the same thing as `instrumentation.ruby_openai`, but as we add support for more AI monitoring libraries, you can use this configuration as a overall switch to disable instrumentation for them as well. This configuration is set to `false` when high security mode is enabled.

* `instrumentation.ruby_openai`, default: `auto`
This configuration controls the way instrumentation for the `ruby-openai` gem is installed in your application. `auto` will default to adding the instrumentation via module prepending and is the same as setting this value to `prepend`. Other valid options are `chain`, to install the instrumentation with alias method chaining, and `disabled` which will skip instrumentation. The instrumentation will only attempt to install if `ai_monitoring.enabled` is true, and your application uses a version of `ruby-openai` >= 3.4.0.

In addition, you may want to adjust these existing configuration options to improve your AI Monitoring experience:

* `distributed_tracing.enabled: true` When Distributed Tracing is enabled on your application, you'll be able to connect your AI Monitoring events with your traces.

* `span_events.max_samples_stored: 10_000` Increasing this configuration to the maximum value of 10,000 will minimize sampling of AI-related distributed traces.

* `custom_insights_events.max_samples_stored: 100_000` Increasing this configuration to the maximum value of 100,000 will minimize sampling of AI-related events and any custom events your application currently sends.
7 changes: 7 additions & 0 deletions src/install/aim/agent-lang/ruby-install.mdx
@@ -0,0 +1,7 @@
---
componentType: default
optionType: agent-lang
headingText: Install the Ruby agent
---

blahblahblah
4 changes: 4 additions & 0 deletions src/install/aim/intro.mdx
Expand Up @@ -18,3 +18,7 @@ To get started, you'll install one of our language agents, then enable AI monito
<figcaption>
Go to **[one.newrelic.com](https://one.newrelic.com) > All Capabilities > AI Monitoring**
</figcaption>

<Callout variant="tip">
Keep in mind that you need to enable distributed tracing for AI monitoring: `distributed_tracing.enabled = true` If `distributed_tracing.enabled` is set to `false`, your AI monitoring data will be limited.
</Callout>

0 comments on commit a3b049b

Please sign in to comment.