ai-response-performance.mdx

title	metaDescription	freshnessValidatedDate
View AI data in New Relic	AI monitoring lets you observe the AI-layer of your tech stack, giving you a holistic overview of the health and performance of your AI-powered app.	never

import aiIntroAiUi from 'images/ai_screenshot-full_intro-ai-ui.webp'

import aiTimeseriesBillboard from 'images/ai_screenshot-crop_timeseries-billboard.webp'

import aiCroppedImageofAIBillboards from 'images/ai_screenshot-crop_billboard.webp'

import aiCroppedImageofAItimeseries from 'images/ai_screenshot-crop_Cropped-image-of-AI-timeseries.webp'

import aiAIEntitiesPage from 'images/ai_screenshot-crop_AI-entities-page.webp'

import aiTopleveAiResponsesSummary from 'images/ai_screenshot-crop_topleve-ai-responses-summary.webp'

import aiResponseTable from 'images/ai_screenshot-crop_response-table.webp'

import aiTraceViewAiResponse from 'images/ai_screenshot-full_trace-view-ai-response.webp'

import aiTraceWaterfallPageSpanDetails from 'images/ai_screenshot-crop_trace-waterfall-page-span-details.webp'

import aiTraceWaterfallPageErrorDetails from 'images/ai_screenshot-crop_trace-waterfall-page-error-details.webp'

Enabling AI monitoring allows the agent to recognize and capture performance metrics and trace data about your app's AI-layer. You can track token usage, number of completions, and response time from your AI-powered app. When you see an error or an inaccurate response, scope to a trace-level view on a given prompt-response interaction to identify problems in the logic of your AI service.

You can view your data by going to one.newrelic.com > All Capabilities > AI monitoring. You can see your data from three different pages:

• AI responses: Overview aggregated data from all your AI entities. Track your AI responses, times, and tokens, or see data about individual prompts and responses.
• AI entities: View a table summary of all entities reporting AI data. See entities with standard APM data like error rate, throughput, and app response time. When you select an entity, you can start exploring the APM AI responses page.
• Compare models: Compare token usage, response time, and error rate between different models. If you're conducting A/B tests, get all the information you need to make decisions about your AI-powered app.

AI responses page [#ai-responses]

The top-level AI responses page shows your AI data in aggregate. Aggregated data takes the average total responses, response times, and token usage per response across all entities reporting AI data. On this page, response refers to an output from your AI-powered app when given a prompt.

If you own several apps with various implementations of different AI frameworks, you can get a general sense for how your AI models perform.

Track total responses, average response time, and token usage

The three tiles show general performance metrics about your AI's responses. These tiles may not tell you the exact cause behind a problem, but they're useful for identifying anomalous fluctuations in your app's performance.

• If you notice a drop in total responses or increase in average response time, it can indicate that some technology in your AI toolchain has malfunctioned and prevented your AI-powered app from posting a response.
• A drop or increase in average token usage per response can give you insight into how your model creates a response. Maybe it's pulling too much context, thus driving up token cost while generating its response. Maybe its responses are too spare, leading to lower token costs and unhelpful responses.

Adjust the timeseries graphs

You can refer to the time series graphs to better visualize when an anamolous behavior first appears.

• Adjust the timeseries graph by dragging over a spike or drop. This scopes the timeseries to a certain time parameter.
• Select the drop down to run comparative analysis for different performance parameters. You can choose between total responses, average response time, or average tokens per response.
• If you've enabled the feedback feature, you can scope the graphs to analyze responses by positive and negative feedback.

Scope your data by service, model, or text string

Evaluate individual AI responses

Your AI response table organizes data about interactions between your end user and AI app. You can view when an interaction occurred, prompts paired with their responses, completion and token count, and which model received a prompt.

You can adjust the table columns by clicking the cog icon in the upper right. This functionality lets you choose the kinds of data you want to analyze.

The Responses is an entry point into viewing trace data about an individual response. Click a row in the table to open the trace view of a particular response.

AI response trace view

The AI response trace view gives you trace-level insights into how your app generates responses. You may want to look at the trace view to identify where an error occurred, or maybe you want to understand what led to negative feedback from a high token response.

• Choose between traces or logs. When you select logs, query within logs for text strings or attributes you want to investigate further.

• Toggle between response details or metadata. The response details column shows the user prompt and AI response so you can maintain context for your traces and spans. Metadata provides a list view for entity GUID, model, tokens, and vendor.

• When an error occurs, the waterfall view highlights its row in red. Select the row to open up span data, including the span's error details.

    <SideBySide>
        <Side>
  
        <img
            title="Span details modal"
            alt="A screenshot that shows span details "
            src={aiTraceWaterfallPageSpanDetails}
        /> 
  
        </Side>
        <Side>
        <img
            title="Error details modal"
            alt="A screenshot that shows error details "
            src={aiTraceWaterfallPageErrorDetails}
        /> 
        </Side>
    </SideBySide>
  

AI entities page [#entities]

The AI entities page organizes all your entities currently reporting AI data into a table. This page displays your AI apps alongside response time, throughput, and error rate.

<img
    title="AI entities page"
    alt="A screenshot of the first page you see when you click AI Monitoring. View aggregated data, compare your AI models, or create drop filters."
    src={aiAIEntitiesPage}
/>

View the entities that report AI data: Go to **[one.newrelic.com](https://one.newrelic.com) > All Capabilities > AI Monitoring**

Selecting an AI entity takes you to the APM summary page for that app. From the APM summary page, select AI monitoring in the left nav.

APM AI responses page [#apm-ai-response]

Selecting an AI entity takes you to the APM summary page. To find your AI data, choose AI Responses in the left nav. We recommend using this page when you've identified that a particular AI entity has contributed to anomalous behavior.

• The APM version of AI responses contains the same tiles, timeseries graphs, and response tables collected as the top-level AI responses page.
• Instead of showing aggregated data, the APM AI responses page shows data scoped to the service you selected from AI entities.
• While the top-level AI responses page lets you filter by service across all AI entities, the APM AI responses page limits filter functionality to the app's own attributes.

To review how to explore your AI data, you can follow the same patterns explained in AI responses section of this doc.

Model comparison page [#model-comparison]

What's next? [#whats-next]

Now that you know how to find your data, you can explore other features that AI monitoring has to offer.

• Want to drop sensitive information before it enters into NRDB? Learn to set up a drop filter
• If you want forward user feedback information about your app's AI responses to New Relic, follow our instructions to update your app code to get user feedback in the UI.
• To adjust your data ingest, explore the AI monitoring configurations to see if you need to report more or less data.