[Feature]: Azure, OpenAI - Log Response Headers

[ishaan-jaff]

The Feature

User request

Motivation, pitch

cc @Manouchehri

Twitter / LinkedIn details

No response

[Manouchehri]

Probably wouldn't hurt to do for other providers as well. Off the top of my head, I don't think any popular LLM API is returning private/sensitive content in their response headers.

[ishaan-jaff]

Starting with just OpenAI/Azure because we're going to need to move all OpenAI python calls

from chat.completions.create
to chat.completions.with_raw_response.create

[ishaan-jaff]

Tradeoffs of this move: openai/openai-python#416 (comment) It looks like with_raw_response will currently return an object with only sync methods

[ishaan-jaff]

another approach is using: https://www.python-httpx.org/advanced/#event-hooks

[ishaan-jaff]

Useful details here: langchain-ai/langchain#9601

[Manouchehri]

Feedback from Langfuse's Marc (when I asked if he thought it was a good idea to also feed upstream traces to a different project):

I'd suggest to add it to the metadata on the same generation in Langfuse though. Splitting the data into two projects makes this a kind of "exotic" setting that not many users will benefit from

I'm inclined to agree with him, not a good idea if it's just me asking.

For logging the request and response metadata, can we do it like this?

https://github.com/orgs/langfuse/discussions/1070#discussioncomment-8369918

{
  "request": {
    "headers": {
      "host": "example.com"
    },
    "url": "https://api.openai.com/v1/chat/completions"
  },
  "response": {
    "headers": {
      "age": "1337"
    },
    "status": 200
  }
}

[ishaan-jaff]

Attempt 1 PR: #1873
TLDR: we can't use chat.completions.with_raw_response.create it's not async and impacts throughput

[Manouchehri]

Hmm, I noticed that Sentry is able to log the response object/headers. Maybe we can steal/borrow their method of doing it?

[Manouchehri]

This is possible to do with: https://til.simonwillison.net/httpx/openai-log-requests-responses

[ishaan-jaff]

we do this now