Repository navigation
rerank and embedding model monitor #5361
|
Hi everyone, I wanted to ask, when using vllm-ascend to run reranking and embedding models and offering services externally through the higressAI gateway, what metrics do you monitor as the basis for inputs and outputs? I noticed that for large language models like DeepSeek, you can monitor the number of input and output tokens, but for reranking and embedding models, only input tokens are tracked, and output tokens aren't counted. |
Replies: 1 comment
|
For standard embedding and reranking endpoints, I would monitor input-token throughput, work items, For embeddings, track texts processed per second, input tokens per second, vectors returned, vector Higress |
For standard embedding and reranking endpoints, I would monitor input-token throughput, work items,
latency and failures, with separate dashboards per model and route. Embeddings return vectors, while
reranking returns relevance scores. Those outputs have different units from generated-text tokens.
In upstream vLLM 0.19.0, the OpenAI-style embedding response reports equal
prompt_tokensandtotal_tokens.vLLM embedding response implementation.
For embeddings, track texts processed per second, input tokens per second, vectors returned, vector
dimension, request latency and error rate. For reranking, track query–document pairs evaluated per
second, input tokens, candidate documents per reque…