Curations no longer work when rerank_hybrid_matches=true

[tstarmer]

Bug Description

When using hybrid search (embedding field + text fields in query_by) with rerank_hybrid_matches=true, overrides/curations are not applied to the search results. The curated documents that should be pinned at specific positions are not in the expected position.

Reproduction Steps

#!/bin/bash

set -e

export TYPESENSE_API_KEY=xyz
export TYPESENSE_HOST=http://localhost:8108

# Clean up any existing container
docker stop typesense-hybrid-rerank-bug 2>/dev/null || true
docker rm typesense-hybrid-rerank-bug 2>/dev/null || true
rm -rf "$(pwd)"/typesense-data-hybrid-rerank
mkdir "$(pwd)"/typesense-data-hybrid-rerank

# Start Typesense
docker run -d -p 8108:8108 --name typesense-hybrid-rerank-bug \
    -v"$(pwd)"/typesense-data-hybrid-rerank:/data \
    typesense/typesense:28.0 \
    --data-dir /data \
    --api-key=$TYPESENSE_API_KEY \
    --enable-cors > /dev/null

# Wait for Typesense to be ready
until curl -s "$TYPESENSE_HOST/health" -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" > /dev/null 2>&1; do
  sleep 1
done

# Wait for Typesense to be fully initialized
sleep 5

echo "Typesense $(curl -s "$TYPESENSE_HOST/debug" -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | jq -r '.version') ready"

# Create collection with embedding field
COLLECTION_RESPONSE=$(curl -s "$TYPESENSE_HOST/collections" \
    -X POST \
    -H "Content-Type: application/json" \
    -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
    -d '{
  "name": "test_articles",
  "fields": [
    {"name": "title", "type": "string"},
    {"name": "content", "type": "string"},
    {"name": "embedding", "type": "float[]", "embed": {"from": ["title", "content"], "model_config": {"model_name": "ts/all-MiniLM-L12-v2"}}}
  ]
}')

if echo "$COLLECTION_RESPONSE" | jq -e '.name' > /dev/null 2>&1; then
    echo "Collection created"
else
    echo "Failed to create collection: $COLLECTION_RESPONSE"
    exit 1
fi

# Import documents
IMPORT_RESPONSE=$(curl -s "$TYPESENSE_HOST/collections/test_articles/documents/import?action=create" \
    -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
    -H "Content-Type: text/plain" \
    -X POST \
    -d '{"id": "1", "title": "University Libraries", "content": "The library system includes Main Library and other specialized collections"}
{"id": "2", "title": "Library Services Guide", "content": "A comprehensive guide to using the library resources"}
{"id": "3", "title": "Campus Dining Options", "content": "Explore the various dining halls and restaurants on campus"}
{"id": "4", "title": "Student Life at University", "content": "Overview of student organizations and campus activities"}')

SUCCESS_COUNT=$(echo "$IMPORT_RESPONSE" | grep -o '"success":true' | wc -l | tr -d ' ')
if [ "$SUCCESS_COUNT" -eq 4 ]; then
    echo "Imported 4 documents"
else
    echo "Failed to import documents properly"
    exit 1
fi

# Wait for embeddings to be generated
sleep 3

# Create override/curation that should pin document "1" at position 1 for query containing "library"
curl -s "$TYPESENSE_HOST/collections/test_articles/overrides/library-curation" -X PUT \
    -H "Content-Type: application/json" \
    -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" \
    -d '{
  "rule": {"query": "library", "match": "contains"},
  "includes": [{"id": "1", "position": 1}]
}' > /dev/null

echo -e "\n=== TEST 1: Keyword-only search ==="
curl -s "$TYPESENSE_HOST/collections/test_articles/documents/search?q=library&query_by=title,content&query_by_weights=10,5" \
    -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | jq -c '[.hits[] | {id: .document.id, title: .document.title}]'

echo -e "\n=== TEST 2: Keyword search with rerank_hybrid_matches=true ==="
curl -s "$TYPESENSE_HOST/collections/test_articles/documents/search?q=library&query_by=title,content&query_by_weights=10,5&rerank_hybrid_matches=true" \
    -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | jq -c '[.hits[] | {id: .document.id, title: .document.title}]'

echo -e "\n=== TEST 3: Hybrid search WITHOUT rerank ==="
curl -s "$TYPESENSE_HOST/collections/test_articles/documents/search?q=library&query_by=title,content,embedding&query_by_weights=10,5,1" \
    -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | jq -c '[.hits[] | {id: .document.id, title: .document.title}]'

echo -e "\n=== TEST 4: Hybrid search WITH rerank_hybrid_matches=true (BUG) ==="
curl -s "$TYPESENSE_HOST/collections/test_articles/documents/search?q=library&query_by=title,content,embedding&query_by_weights=10,5,1&rerank_hybrid_matches=true" \
    -H "X-TYPESENSE-API-KEY: ${TYPESENSE_API_KEY}" | jq -c '[.hits[] | {id: .document.id, title: .document.title}]'

echo -e "\nExpected: Document id:1 at position 1 in all tests"
echo "BUG: If Test 4 does NOT show id:1 first, override is ignored with hybrid+rerank"
echo -e "\nCleanup: docker stop typesense-hybrid-rerank-bug && docker rm typesense-hybrid-rerank-bug"

This seems to be the case in v28 and v29

[alangmartini]

Fixed in v30