| title | metaDescription | freshnessValidatedDate |
|---|---|---|
Configure AI monitoring |
You can apply certain configurations to your APM agents to change how your AI data appears in New Relic. |
never |
Once you install AI monitoring, you can configure the default behavior of the agent or update your app to collect different kinds of data.
Update default agent behavior for AI monitoring at these agent configuration docs:
If you've disabled ai_monitoring.record_content.enabled, you may still want to know the number of tokens your app uses in an interaction. If you haven't disabled content from being recorded by the agent, you don't need to invoke the token count API.
Update your app code with the below code snippets:
```js
// register callback to set token counts
newrelic.setLlmTokenCount(callback)
/**
* Sample callback to calculate token count. Must be a synchronous method.
* This callback is optional, and is designed for customers that cannot send
* the content in the Llm events but still want token counts in the case
* they are not present.
* @param {string} model name of model
* @param {string} content content of call
* @returns {int} token count for content, customers must use relevant libraries to calculate this
*/
function callback(model, content) {
// put your business logic to calculate token counts
return 10
}
```
</Collapser>
<Collapser
id="python-token-api"
title="Python"
>
INSERT_TEXT_HERE
</Collapser>
<Collapser
id="ruby-token-api"
title="Ruby"
>
```ruby
# Write a proc to calculate token counts
# The proc should accept a hash as an argument with two keys:
:model => [String]
# The name of the LLM model
:content => [String]
# The message content/prompt or
# embedding input
#
# It should return an integer representing the token count for
# the content.
callback_proc = proc do |hash|
# your calculation behavior
hash[:model] + hash[:content]
10
end
#register this callback proc using the API
#if you're using Rails, this might be placed in an initializer
NewRelic::Agent.set_llm_token_count_callback(callback_proc)
```
</Collapser>
AI monitoring can correlate trace IDs between a generated message from your AI and the feedback an end user submitted. When you update your code to correlate messages with feedback, you need to take the trace ID and pass it to the feedback call, as these two events occur at two different endpoints.
To pass the trace ID when recording a feedback event, refer to the below code samples for supported languages:
```go
/* given app, a newrelic application value, and a chat completion message response resp of type nropenai.ChatCompletionResponseWrapper, */
rating := "4"
category := "informative"
message := "The response was concise yet thorough."
customMetadata := map[string]any{
"llm.foo": "bar"
}
```
</Collapser>
<Collapser
id="nodejs-feedback-feature"
title="Node.js"
>
```js
const responses = new Map()
// This appears in a handler. It stores a reference to traceId via `newrelic.getTraceMetadata()
fastify.post('/chat-completion', async(request, reply) => {
const { message = 'Say this is a test', model = 'gpt-4' } = request.body || {}
// Assigns conversation_id via custom attribute API
const conversationId = uuid()
newrelic.addCustomAttribute('llm.conversation_id', conversationId)
const chatCompletion = await openai.chat.completions.create({
temperature: 0.5,
messages: [{ role: 'user', content: message }],
model
});
const { traceId } = newrelic.getTraceMetadata()
responses.set(chatCompletion.id, { traceId })
return reply.send(chatCompletion)
})
// To post feedback. it'll get the traceId from some context and post it via `newrelic.recordLlmFeedback`
fastify.post('/feedback', (request, reply) => {
const { category = 'feedback-test', rating = 1, message = 'Good talk', metadata, id } = request.body || {}
const { traceId } = responses.get(id)
if (!traceId) {
return reply.code(404).send(`No trace id found for ${message}`)
}
newrelic.recordLlmFeedbackEvent({
traceId,
category,
rating,
message,
metadata
})
return reply.send('Feedback recorded')
})
```
```python
import newrelic.agent
def message(request):
trace_id = newrelic.agent.current_trace_id()
def feedback(request):
record_feedback(trace_id=request.trace_id, rating=request.rating)
```
NewRelic::Agent.record_llm_feedback_event(trace_id: trace_id, rating: rating)
```
</Collapser>