[CRITICAL] Server mode search completely broken - 0 results on all queries

[jsbattig]

Bug Description

Server mode is completely unable to index repositories or execute search queries. The server defaults to the legacy QdrantContainerBackend instead of the modern FilesystemVectorStore, causing cascading failures:

1. Tries to connect to Ollama for embeddings (Qdrant mode requirement)
2. Ollama connection fails with "Connection refused"
3. No indexing occurs (0 files indexed, 0 embeddings created)
4. All search queries return 0 results
5. Stats service cannot check indexing status

Environment

• Version: cd1cdfe (feature/epic-477-mcp-oauth-integration)
• OS: Linux 5.14.0-570.55.1.el9_6.x86_64
• Mode: Server mode (multi-user)
• Configuration: Production deployment at linner.ddns.net:8383
• Repository: my-tries (activated from golden repo "tries")

Steps to Reproduce

1. Start cidx-server in server mode
2. Activate a repository (e.g., my-tries)
3. Attempt to sync repository via MCP: sync_repository(user_alias="my-tries")
4. Attempt semantic search via MCP: search_code(query_text="...", search_mode="semantic")
5. Check repository statistics via MCP: get_repository_statistics(repository_alias="my-tries")

Expected Behavior

1. Server should use FilesystemVectorStore backend (current production standard)
2. Server should use VoyageAI for embeddings (production provider)
3. Sync should index all supported files (Pascal .pas, Python, etc.)
4. Search queries should return relevant results from indexed content
5. Statistics should show >0 indexed files and embeddings

Actual Behavior

Backend Selection:

INFO - No vector_store config - defaulting to QdrantContainerBackend (backward compatibility)
INFO - Using backend: QdrantContainerBackend
INFO - Using collection: code_index__nomic_embed_text

Embedding Provider Failure:

ERROR - Semantic search failed: Failed to connect to Ollama: [Errno 111] Connection refused
ERROR - Failed to search repository 'my-tries': Semantic search failed: Failed to connect to Ollama: [Errno 111] Connection refused

Indexing Status:

{
  "files": {
    "total": 61,
    "indexed": 0,
    "by_language": {"text": 6, "markdown": 1}
  },
  "storage": {
    "embedding_count": 0
  },
  "activity": {
    "last_sync_at": null,
    "sync_count": 0
  },
  "health": {
    "issues": ["Low indexing coverage: 0.0%"]
  }
}

Stats Service Warnings:

WARNING - Cannot check indexing status: Real Qdrant file indexing check not yet implemented

Error Messages / Logs

From /var/log/systemd/cidx-server.service:

2025-11-17 19:12:37 - code_indexer.backends.backend_factory - INFO - No vector_store config - defaulting to QdrantContainerBackend (backward compatibility)
2025-11-17 19:12:37 - code_indexer.server.services.search_service - ERROR - Semantic search failed for repo /home/jsbattig/.cidx-server/data/activated-repos/admin/my-tries: Failed to connect to Ollama: [Errno 111] Connection refused
2025-11-17 19:14:56 - code_indexer.server.services.stats_service - WARNING - Cannot check indexing status for *.pas files: Real Qdrant file indexing check not yet implemented

Impact Assessment

Severity: Critical - Core functionality completely broken
Affected Users: All server mode users (MCP integration, team deployments)
Frequency: 100% - Affects every search query and indexing operation
Workaround: None - Server mode unusable for semantic search

Root Cause Analysis

Initial Hypothesis: Server mode configuration missing vector_store settings, causing fallback to legacy backend incompatible with current deployment.

Investigation Findings:

1. Backend Factory Logic (src/code_indexer/backends/backend_factory.py):

   • When vector_store config is missing → defaults to QdrantContainerBackend
   • This is "backward compatibility" mode for old configs
   • Server repos likely created before FilesystemVectorStore migration

2. Repository Config Location:

   • Golden repos: ~/.cidx-server/data/golden-repos/<alias>/
   • Activated repos: ~/.cidx-server/data/activated-repos/<user>/<alias>/
   • Config file: .code-indexer/config.yml in each repo directory

3. Missing Configuration:

   • Repositories don't have vector_store section in config
   • No embedding_provider configuration
   • Defaults trigger legacy Qdrant + Ollama mode

Confirmed Root Cause: Repository configurations created before v7.x migration to FilesystemVectorStore are missing required vector_store and embedding_provider settings, causing server to fall back to incompatible legacy backend.

Fix Implementation

Changes Required

• Update repository config schema to include FilesystemVectorStore settings
• Add VoyageAI embedding provider configuration for server mode
• Implement config migration for existing repositories
• Fix stats service to work with FilesystemVectorStore
• Add validation to prevent Qdrant backend in production

Testing Required

• Unit tests: Backend factory selects FilesystemVectorStore when configured
• Integration tests: Repository indexing with VoyageAI + FilesystemVectorStore
• E2E tests: Full MCP search workflow (activate → sync → search → results)
• Manual validation: Test on production server with real repository

Implementation Status

• Core fix implemented
• Tests added and passing
• Code review approved
• Manual testing completed
• Documentation updated
• Regression tests added

Completion: 0/6 tasks complete (0%)

Verification Evidence

[To be added after fix - should include:]

• Repository statistics showing >0 indexed files
• Successful search query results
• Server logs showing FilesystemVectorStore + VoyageAI usage
• Health check showing all services healthy

---

Reported By: Claude Code (investigating MCP integration)
Assigned To: [Unassigned]
Found In: cd1cdfe (feature/epic-477-mcp-oauth-integration)
Fixed In: [To be updated]

[jsbattig]

✅ ISSUE RESOLVED

Fix Status: Complete and verified in production

Solution Summary

Fixed server mode search by ensuring activated repositories have proper with FilesystemVectorStore configuration.

Root Cause:

• Activated repositories lacked config files
• Backend defaulted to legacy QdrantContainerBackend
• Ollama connection failures prevented indexing

Implementation:

1. Modified to create during repository activation
2. Config includes: ,
3. Critical fix: Changed from config.yml to config.json (ConfigManager requirement)

Commits:

• 444e0ba: Initial fix (created config.yml - WRONG)
• 94c8c74: Critical correction (config.json - CORRECT)

Verification Evidence

CLI Query Test (on server):

$ cd ~/.cidx-server/data/activated-repos/admin/my-tries
$ cidx query "hash trie node removal algorithm" --quiet --limit 5
✅ Returned 5 results with similarity scores 0.751-0.713

MCP API Query Test (via claude.ai):

{
  "total_results": 5,
  "results": [
    {"file_path": "hash_trie.pas", "similarity_score": 0.729},
    {"file_path": "hash_trie.pas", "similarity_score": 0.701},
    {"file_path": "hash_trie.pas", "similarity_score": 0.690},
    {"file_path": "hash_trie.pas", "similarity_score": 0.638},
    {"file_path": "hash_trie.pas", "similarity_score": 0.634}
  ],
  "query_metadata": {
    "execution_time_ms": 1018,
    "repositories_searched": 1
  }
}

Index Verification:

$ ls ~/.cidx-server/data/activated-repos/admin/my-tries/.code-indexer/index/
voyage-code-3/  # 1.9MB indexed data

$ cat ~/.cidx-server/data/activated-repos/admin/my-tries/.code-indexer/config.json
{
  "vector_store": {"provider": "filesystem"},
  "embedding_provider": "voyage-ai",
  "voyage_ai": {"model": "voyage-code-3"}
}

Test Results

✅ All 9 new tests passing
✅ CLI query returns results
✅ MCP API query returns results
✅ FilesystemVectorStore backend selected
✅ VoyageAI embeddings working
✅ 18 files indexed, 58 chunks

Success Criterion Met: Query command returns actual semantic search results (not 0).

Closing issue - Fix deployed to production and fully verified.

[jsbattig]

Verified working in production. MCP semantic search returning actual results.