Skip to content

Release b188

Choose a tag to compare

@github-actions github-actions released this 07 Dec 06:42

feat: Support stream usage statistics

  • Added stream_options to ChatCompletionRequest and CompletionRequest models.
  • Automatically inject stream_options: {"include_usage": true} in build_request_body_generic for streaming requests if not provided.
  • Enhanced stream_logger_task to parse timings from llama.cpp responses and convert them to standard OpenAI usage format for logging.