Proposal: Log raw LLM requests and responses in tracing for improved debugging

[waldekmastykarz]

Proposal

Enhance the tracing functionality by logging raw HTTP requests and responses sent to and received from LLMs. This would aid in debugging issues related to prompt input, model responses, and integration errors.

Rationale

• Debugging LLM interactions is challenging when only high-level errors or summaries are available.
• Capturing the full raw request and response enables developers to reproduce issues, inspect prompt formatting, and analyze model behavior.
• Tracing raw data helps identify encoding issues, unintentional prompt modifications, or unexpected output formats.

[pelikhan]

would you want those in separate files?

[pelikhan]

Open telemetry?

[waldekmastykarz]

It would be perfectly fine to include them in the trace. OpenTelemetry would be icing on the cake.

[pelikhan]

The streaming responses are quite verbose, I probably need to offload them to another file or they bloat the trace file.

[waldekmastykarz]

If GenAIScript supports web proxies, then using Dev Proxy might be a solution without any changes to the product. Dev Proxy allows you to inspect web requests using Chrome Dev Tools.

[pelikhan]

Yes it should support proxies so this could be knocking 2 birds in one: test proxy support, also test throttling support.

[waldekmastykarz]

Let me test it on v2 and I'll report back