Skip to content
Discussion options

You must be logged in to vote

For standard embedding and reranking endpoints, I would monitor input-token throughput, work items,
latency and failures
, with separate dashboards per model and route. Embeddings return vectors, while
reranking returns relevance scores. Those outputs have different units from generated-text tokens.
In upstream vLLM 0.19.0, the OpenAI-style embedding response reports equal prompt_tokens and total_tokens.
vLLM embedding response implementation.

For embeddings, track texts processed per second, input tokens per second, vectors returned, vector
dimension, request latency and error rate. For reranking, track query–document pairs evaluated per
second, input tokens, candidate documents per reque…

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by lizhaoyangre
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants