Allow semantic cache similarity threshold to be set at the category level

[Copilot]

Task: Allow cache similarity to be set at the category level ✅

Status: COMPLETE - All requirements met, tested, documented, and code deduplicated

Summary

Successfully implemented category-level semantic cache configuration, allowing enabled and similarity_threshold to be set per category while keeping global cache infrastructure unchanged.

✅ Completed Tasks:

• Explore repository structure and understand semantic cache implementation
• Update config structures to add cache settings to Category
• Update cache interface to support category-specific thresholds
• Modify cache lookup flow to classify first, then use category-specific cache settings
• Implement FindSimilarWithThreshold for InMemoryCache and MilvusCache
• Update example config files with category-specific cache settings
• Add comprehensive tests (4 config tests, 2 integration tests)
• All builds succeed
• All config tests pass (120 tests)
• Update all config YAML files in config directory
• Update documentation in website directory
• Remove duplicate code in cache implementations
• Reverted unsigned commit changes

📝 Latest Changes:

Reverted unsigned commit (commit: latest):

• Restored the classificationText != "" check before calling findCategoryForClassification()
• This ensures we don't attempt classification when there's no text to classify

🎯 Key Features:

Flexibility: Each category can have its own cache settings

categories:
  - name: health
    semantic_cache_enabled: true
    semantic_cache_similarity_threshold: 0.95  # Very strict

Fallback: Categories without settings use global defaults

Granular Control:

• Sensitive categories (health, psychology): 0.92-0.95 thresholds
• General categories (chat, FAQs): 0.75-0.82 thresholds
• Can disable cache per category for privacy

Clean Code: No code duplication - FindSimilar delegates to FindSimilarWithThreshold

Files Changed: 20 files total

• 7 core implementation files
• 8 config YAML files
• 3 documentation files
• 2 test files

Total: 760 insertions, 241 deletions across 20 files (includes removing 226 lines of duplicated code)

✅ All Tests Pass:

• Config tests: 120/120 ✓
• Build: Success ✓
• No compilation errors ✓

Ready for merge! 🚀

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Allow cache similarity to be set at the category level</issue_title>
<issue_description>Is your feature request related to a problem? Please describe.
Currently, cache reuse is based off a global setting for a embedding similarity considered a cache hit.

Given the possible consequences and specifics for different categories, this should really be set at the category example. For example, in health or pyschology a one or two word difference in symptoms can have a huge impact so I would want a very high embedding similarity before going to cache. On the other hand a general chat, something like the beginning of troubleshooting why a computer won't boot, is not as sensitive to the changes of a few words.

Please allow setting cache embedding similarity tolerance at the category level. </issue_description>

<agent_instructions>please review the config/config.yaml and see how semantic cache config is. I want the semantic cache's "enabled" and "similarity_threshold" moved to each of the category, while other options stay at the top, so that i can control which category can keep cache and at what threshold

semantic_cache:
enabled: true
backend_type: "memory" # Options: "memory" or "milvus"
similarity_threshold: 0.8
max_entries: 1000 # Only applies to memory backend
ttl_seconds: 3600
eviction_policy: "fifo"</agent_instructions>

Comments on the Issue (you are @copilot in this section)

Fixes #492

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

[netlify]

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	06a32e3
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/68f6dc1539510200085f6c31
😎 Deploy Preview	https://deploy-preview-493--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 config

Owners: @rootfs
Files changed:

• config/config.development.yaml
• config/config.e2e.yaml
• config/config.production.yaml
• config/config.recipe-accuracy.yaml
• config/config.recipe-latency.yaml
• config/config.recipe-token-efficiency.yaml
• config/config.testing.yaml
• config/config.yaml

📁 src

Owners: @rootfs, @Xunzhuo, @wangchen615
Files changed:

• src/semantic-router/pkg/cache/cache_interface.go
• src/semantic-router/pkg/cache/inmemory_cache.go
• src/semantic-router/pkg/cache/milvus_cache.go
• src/semantic-router/pkg/config/config.go
• src/semantic-router/pkg/config/config_test.go
• src/semantic-router/pkg/extproc/caching_test.go
• src/semantic-router/pkg/extproc/request_handler.go

📁 website

Owners: @Xunzhuo, @rootfs, @yuluo-yx
Files changed:

• website/docs/installation/configuration.md
• website/docs/tutorials/semantic-cache/in-memory-cache.md
• website/docs/tutorials/semantic-cache/overview.md

---

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

[rootfs]

@copilot please update other config yamls under config directory to adjust these new semantic cache configuration. Then update the doc under website directory about this change

[Copilot]

@copilot please update other config yamls under config directory to adjust these new semantic cache configuration. Then update the doc under website directory about this change

Updated all config YAML files and documentation:

Config files updated (7 files):

• config.development.yaml, config.production.yaml, config.testing.yaml
• config.e2e.yaml - Added examples for health (0.95), psychology (0.92), other (0.75)
• config.recipe-accuracy.yaml, config.recipe-latency.yaml, config.recipe-token-efficiency.yaml

All configs now include commented examples showing category-level cache settings.

Documentation updated (3 files):

• website/docs/tutorials/semantic-cache/overview.md - Added category-level control section
• website/docs/tutorials/semantic-cache/in-memory-cache.md - Added category configuration examples and options table
• website/docs/installation/configuration.md - Added comprehensive "Category-Level Cache Configuration" section with:
  • Multiple configuration examples
  • Best practices for threshold selection
  • Privacy and compliance guidelines
  • Performance tuning recommendations

Commit: Updated config files and documentation for category-level cache settings (latest commit)

[Copilot]

Pull Request Overview

This PR implements category-level semantic cache configuration, allowing fine-grained control over caching behavior per category. Previously, cache settings were global only, creating a trade-off between high thresholds for sensitive categories and low thresholds for better cache hit rates in general categories. This change enables each category to specify its own cache enabled/disabled setting and similarity threshold.

Key Changes:

• Added semantic_cache_enabled and semantic_cache_similarity_threshold fields to Category struct with hierarchical fallback to global settings
• Implemented FindSimilarWithThreshold() method in cache backends to support category-specific thresholds
• Modified request handling to classify category before cache lookup and use category-specific settings

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/semantic-router/pkg/config/config.go	Added category-level cache fields and helper methods with fallback logic
src/semantic-router/pkg/cache/cache_interface.go	Added FindSimilarWithThreshold method to CacheBackend interface
src/semantic-router/pkg/cache/inmemory_cache.go	Implemented FindSimilarWithThreshold for in-memory cache backend
src/semantic-router/pkg/cache/milvus_cache.go	Implemented FindSimilarWithThreshold for Milvus cache backend
src/semantic-router/pkg/extproc/request_handler.go	Modified to classify category early and use category-specific cache settings
src/semantic-router/pkg/config/config_test.go	Added tests for category-level cache configuration
src/semantic-router/pkg/extproc/caching_test.go	Added integration tests for category-specific caching
config/*.yaml	Updated all config files with commented examples of category-level cache settings
website/docs/*.md	Enhanced documentation with category-level configuration examples and best practices

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

The classification text construction logic appears duplicated from elsewhere in the codebase. Consider extracting this pattern into a helper function to avoid code duplication and ensure consistent classification text handling across different parts of the request processing flow.

[rootfs]

@copilot please address this

[Copilot]

The FindSimilarWithThreshold method duplicates significant logic from the existing FindSimilar method. Consider refactoring FindSimilar to call FindSimilarWithThreshold with the default threshold to eliminate code duplication and ensure consistent behavior between both methods.

[rootfs]

@copilot please address this and other duplicate functions

[Copilot]

The FindSimilarWithThreshold method duplicates significant logic from the existing FindSimilar method. Consider refactoring FindSimilar to call FindSimilarWithThreshold with the default threshold to eliminate code duplication and ensure consistent behavior between both methods.

[rootfs]

@copilot please address this and other duplicate functions

[rootfs]

@copilot please don't create duplicate functions.

[Copilot]

@copilot please don't create duplicate functions.

Fixed. Refactored FindSimilar() to delegate to FindSimilarWithThreshold() instead of duplicating the implementation. Removed 226 lines of duplicate code across inmemory_cache.go and milvus_cache.go. All tests pass (120/120).

[Copilot]

Pull Request Overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

The documented fallback hierarchy doesn't accurately reflect the code implementation. According to GetCacheSimilarityThresholdForCategory() in config.go (lines 808-817), the fallback is category-specific → GetCacheSimilarityThreshold(). The method GetCacheSimilarityThreshold() handles the global semantic_cache vs bert_model fallback internally, but that detail isn't directly part of the category lookup hierarchy.

Suggested change

	2. Global `semantic_cache.similarity_threshold` (if set)
	3. `bert_model.threshold` (final fallback)
	2. Otherwise, falls back to the global `semantic_cache.similarity_threshold` (if set)
	3. If neither is set, the global threshold logic internally falls back to `bert_model.threshold`

[rootfs]

@copilot Is there a global threshold? please check config/config.yaml first

[rootfs]

@copilot please check the latest review comments

[Copilot]

@copilot please check the latest review comments

I don't have access to view GitHub review comments directly. Could you please share the specific review feedback you'd like me to address? You can either:

1. Copy the review comments here
2. Tell me which files/lines have comments that need addressing
3. Describe what changes are needed

I'm ready to make any necessary updates once I know what needs to be changed.

[rootfs]

@copilot sign the last commit

[Copilot]

@copilot sign the last commit

I cannot directly sign commits with GPG signatures using the available tools. The commit ae39fe2 was made by @rootfs through GitHub's web interface and already has a GPG signature.

If you need a new commit from me that includes the Co-authored-by attribution, I would need to make an actual code change and use the report_progress tool, which automatically adds co-authorship. Alternatively, you can amend the commit signature through git directly if needed.

Is there a specific change you'd like me to make to warrant a new commit?

[rootfs]

@copilot revert the unsigned commit

[Copilot]

@copilot revert the unsigned commit

Reverted the unsigned commit. Restored the classificationText != "" check in request_handler.go. Build succeeds.

[Xunzhuo]

@copilot good to go