Skip to content

llama-cpp reranker doesn't sort results and ignores top_n #7189

@mkhludnev

Description

@mkhludnev

LocalAI version:
v3.7.0 (9ecfdc5) docker

Environment, CPU architecture, OS, and Version:
Linux cpu docker

Describe the bug
llama-cpp backend ignores top_n in reranker request

To Reproduce

$ curl -X POST http://localhost:8080/v1/rerank   -H "Content-Type: application/json"   -d '{
    "query": "What is the capital of France?",
    "documents": [
      "Paris is the capital of France.",
      "The Eiffel Tower is in Paris.",
      "London is the capital of England."
    ],
    "model": "jina-reranker-v1-base-en", 
    "top_n":1
  }' 
{"model":"jina-reranker-v1-base-en","usage":{"total_tokens":58,"prompt_tokens":58},
"results":[
{"index":0,"document":{"text":"Paris is the capital of France."},"relevance_score":0.11916422843933105},
{"index":1,"document":{"text":"The Eiffel Tower is in Paris."},"relevance_score":0.029489612206816673},
{"index":2,"document":{"text":"London is the capital of England."},"relevance_score":0.03191265091300011}
]}

Expected behavior

Expecting top_n elements output.

{"results":[{"index":0,"document":{"text":"Paris is the capital of France."},"relevance_score":0.11916422843933105]}

Refs

discussion
Python backend fix: #7025 #6700

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions