Skip to content

metrics: TTFT in streaming mode #128

@tao12345666333

Description

@tao12345666333

for now we haven't tried streaming mode. In the buffered mode, the response from LLM has to be fully received before the response is received. If you can add an issue to track TTFT in streaming mode, that'll be great.

Originally posted by @rootfs in #126 (comment)

Metadata

Metadata

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions