[Bug]: incorrect completion_start_time for streaming request #8999
Description
What happened?
Pull request #8156 removed the call to logging_obj.success_handler in the file:
litellm/litellm_core_utils/streaming_handler.py
Due to this removal, the following function from litellm/litellm_core_utils/litellm_logging.py is no longer invoked during streaming responses:
def _success_handler_helper_fn(
self,
result=None,
start_time=None,
end_time=None,
cache_hit=None,
standard_logging_object: Optional[StandardLoggingPayload] = None,
):
try:
if start_time is None:
start_time = self.start_time
if end_time is None:
end_time = datetime.datetime.now()
if self.completion_start_time is None:
self.completion_start_time = end_time
self.model_call_details["completion_start_time"] = (
self.completion_start_time
)
As a result, completion_start_time is not accurately recorded, impacting the calculation of the "time to first token" metric.
Relevant log output
Are you a ML Ops Team?
No
What LiteLLM version are you on ?
v1.61.20
Twitter / LinkedIn details
No response