Comprehensive Chat Agent Overhaul, Testing Consolidation, and Evaluation Methodology

[kargig]

Summary

This PR delivers a massive upgrade to the agentic chat subsystem, improving its accuracy, contextual awareness, and reliability. It transitions the LLM away from ambiguous, "catch-all" tools towards highly specific tool schemas with dedicated Python executors, resulting in a strict upgrade to chat quality (achieving a 100% pass rate on quantitative evaluations).

Additionally, this PR completely reorganizes the fragmented chat test suite for better maintainability, resolves several critical runtime bugs, and introduces a robust, double-blind LLM-as-a-judge evaluation methodology for future chat development.

Changes Made

🤖 Agent & Prompt Engineering Improvements

• Specific Tool Schemas: Replaced generic tools (like search_certifications) with explicit, dedicated tools: compare_certifications, get_certification_path, get_dive_site_details, search_diving_trips, get_user_dive_logs, and get_reviews_and_comments.
• Anti-Hallucination Guardrails: Updated the system prompt to strictly forbid the LLM from hallucinating coordinates for locations. The system now enforces reliance on the backend's deterministic Nominatim geocoding.
• Enhanced Formatting: Instructed the LLM to automatically categorize lists of dive sites and explicitly include crucial metadata like max_depth, difficulty, and shore_direction.
• Proactive Cross-Referencing: The LLM now proactively cross-references local Diving Centers when recommending dive sites.
• Confident Fallbacks: Replaced rigid, apologetic error messages with confident fallback suggestions (e.g., pivoting to diving centers when dive sites aren't found).
• Page Context Enrichment: Fixed the page context resolver to correctly map dive_site.name and inject rich physics metadata (depths, duration, serialized gas info) into the context window, enabling seamless SAC calculations on specific dive log pages.

🗺️ Dynamic Geocoding & Spatial Search

• Dynamic Search Radius: Replaced the hardcoded 100km fallback radius with a dynamic calculation using the Haversine formula (calculate_distance in geo_utils.py). The search radius now scales proportionally to the size of the Nominatim bounding box (clamped between 5km and 200km).
• Scope Fix: Resolved an UnboundLocalError in discovery.py by moving get_empirical_region_bounds and get_external_region_bounds imports to the module level.

🎯 Intent Extraction Refinements

• Dedicated Executors: Shifted from a massive monolithic others.py conditional block to dedicated executor modules (user_data.py and reviews.py), respecting global privacy settings like disable_diving_center_reviews.
• Gear Rental (SearchGearRentalTool): Created a dedicated tool schema. Updated the fallback logic to explicitly join(GearRentalCost) to only recommend centers verified to offer rentals.
• Career Path (CAREER_PATH): Implemented regex-based tokenization and stop-word filtering to accurately extract specific certification entities.
• Comparisons (COMPARISON): Improved the sorting heuristic to prioritize exact whole-word matches over partial substring matches. Increased the result cap from 10 to 20 to prevent real database seed data from crowding out exact matches.

🐛 Bug Fixes & Reliability

• Fixed shadowing bug in discovery executor where the date argument conflicted with the date class import, crashing trip searches.
• Resolved missing argument in weather enrichment pipeline by correctly passing intent_location to the enricher function.
• Updated ChatIntermediateAction schema to capture tool_name and raw tool_result, enabling a high-fidelity audit trail.

🧪 Testing & Evaluation Methodology

• Suite Reorganization: Merged 11 scattered and overlapping test files into 3 logically organized files (test_chat_agent.py, test_chat_executors.py, test_chat_api.py) that align with the system's modular architecture.
• Fixture Fixes: Resolved data-dependency bugs in recommendation fixtures to ensure reliable test execution.
• LLM-as-a-Judge Evaluation Pipeline: Introduced analyze_chat_quality_diff.py and evaluate_qualitative.py scripts to automate double-blind A/B testing of chatbot responses.
• Documentation: Added comprehensive Markdown documentation in docs/development/chat_evaluation_methodology.md establishing the standard operating procedure for running the new evaluation pipeline.

Testing

• Automated Tests: Ran the full backend test suite (./docker-test-github-actions.sh). All tests pass (1430/1430). The newly consolidated test files correctly handle DB seed data overlaps.
• Qualitative LLM Evaluation: Ran a comprehensive, double-blind A/B evaluation against 38 real-world prompts comparing the old architecture to this branch. This branch achieved a 100% quantitative pass rate and won the qualitative assessment 28-10, fixing previous regressions in Gear Rental, Certifications, and Regional Searches.
• Manual Verification: Manually tested complex prompts via a local testing script to verify new tools (like get_user_dive_logs and get_reviews_and_comments) correctly invoked backend logic and respected privacy constraints.

Related Issues

• Resolves intent parsing regressions and geocoding hallucination issues identified during the chat architecture migration.

Additional Notes

• Reviewers: Pay special attention to the new specialized tools in tools.py and the dynamic Haversine radius calculation in discovery.py.
• Deployment: No database migrations are required. The backend environment must have access to outbound internet for Nominatim API calls (already standard for geo_utils), and DEEPSEEK_API_KEY must be configured to run the new evaluation methodology scripts.