fix: duplicate UNIFIED_CLASSIFIER definition and optimize lock contention #516
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ntention - Remove duplicate UNIFIED_CLASSIFIER global state - Optimize PARALLEL_LORA_ENGINE lock contention by using Arc clone Signed-off-by: OneZero-Y <aukovyps@163.com>
👥 vLLM Semantic Team NotificationThe following members have been identified for the changed files in this PR and have been automatically assigned: 📁
|
rootfs
approved these changes
Oct 23, 2025
rootfs
merged commit 5ee8a1b
into
vllm-project:feat-candle-refactoring
9 of 11 checks passed
rootfs
pushed a commit
to rootfs/semantic-router.bak
that referenced
this pull request
Oct 23, 2025
…ntention (vllm-project#516) - Remove duplicate UNIFIED_CLASSIFIER global state - Optimize PARALLEL_LORA_ENGINE lock contention by using Arc clone Signed-off-by: OneZero-Y <aukovyps@163.com>
rootfs
pushed a commit
to rootfs/semantic-router.bak
that referenced
this pull request
Oct 23, 2025
…ntention (vllm-project#516) - Remove duplicate UNIFIED_CLASSIFIER global state - Optimize PARALLEL_LORA_ENGINE lock contention by using Arc clone Signed-off-by: OneZero-Y <aukovyps@163.com>
rootfs
pushed a commit
to rootfs/semantic-router.bak
that referenced
this pull request
Oct 23, 2025
…ntention (vllm-project#516) - Remove duplicate UNIFIED_CLASSIFIER global state - Optimize PARALLEL_LORA_ENGINE lock contention by using Arc clone Signed-off-by: OneZero-Y <aukovyps@163.com> Signed-off-by: Huamin Chen <hchen@redhat.com>
rootfs
pushed a commit
that referenced
this pull request
Oct 23, 2025
…ntention (#516) - Remove duplicate UNIFIED_CLASSIFIER global state - Optimize PARALLEL_LORA_ENGINE lock contention by using Arc clone Signed-off-by: OneZero-Y <aukovyps@163.com>
rootfs
added a commit
that referenced
this pull request
Oct 24, 2025
* refactor: Implement modular candle-binding architecture (#254) - Restructure codebase into modular layers (core/, ffi/, model_architectures/, classifiers/) - Add unified error handling and configuration loading systems - Implement dual-path architecture for traditional and LoRA models - Add comprehensive FFI layer with memory safety Maintains backward compatibility while enabling future model integrations. refactor: Implement modular candle-binding architecture - Restructure codebase into modular layers (core/, ffi/, model_architectures/, classifiers/) - Add unified error handling and configuration loading systems - Implement dual-path architecture for traditional and LoRA models - Add comprehensive FFI layer with memory safety Maintains backward compatibility while enabling future model integrations. Signed-off-by: OneZero-Y <aukovyps@163.com> * feat:unit tests for candle refactoring (#296) feat:unit tests for candle refactoring feat:unit tests for candle refactoring Signed-off-by: OneZero-Y <aukovyps@163.com> Signed-off-by: Huamin Chen <hchen@redhat.com> * feat:support for two long-context embedding models (Qwen3-Embedding-0.6B and EmbeddingGemma-300M) (#453) feat:support for two long-context embedding models (Qwen3-Embedding-0.6B and EmbeddingGemma-300M) Signed-off-by: OneZero-Y <aukovyps@163.com> Signed-off-by: Huamin Chen <hchen@redhat.com> * fix:Implement Comprehensive Rayon Parallelization for LoRA Classifiers (#464) Signed-off-by: OneZero-Y <aukovyps@163.com> Signed-off-by: Huamin Chen <hchen@redhat.com> * fix:Improve rust unit test and optimize concurrent tests with rayon (#471) - Add 6 new unit test files - Replace std::thread::spawn with rayon::par_iter Signed-off-by: OneZero-Y <aukovyps@163.com> Signed-off-by: Huamin Chen <hchen@redhat.com> * fix: resolve syntax errors after rebase Signed-off-by: Huamin Chen <hchen@redhat.com> * add additional update Signed-off-by: Huamin Chen <hchen@redhat.com> * Change label count params to c_int (#494) Signed-off-by: carlory <baofa.fan@daocloud.io> * update embedding setting in config (#489) Signed-off-by: Huamin Chen <hchen@redhat.com> * make CUDA and Flash Attention 2 optional features (#511) Signed-off-by: OneZero-Y <aukovyps@163.com> * fix: Fix duplicate UNIFIED_CLASSIFIER definition and optimize lock contention (#516) - Remove duplicate UNIFIED_CLASSIFIER global state - Optimize PARALLEL_LORA_ENGINE lock contention by using Arc clone Signed-off-by: OneZero-Y <aukovyps@163.com> * Merge main to candle refactoring (#523) * Update test description from Math to General (#483) Signed-off-by: carlory <baofa.fan@daocloud.io> * feat: add HuggingChat support (#477) * add chat ui to dashboard and docker compose & refactor dashboard/backend/ Signed-off-by: JaredforReal <w13431838023@gmail.com> * try fix network error Signed-off-by: JaredforReal <w13431838023@gmail.com> * more --------- Signed-off-by: JaredforReal <w13431838023@gmail.com> Co-authored-by: bitliu <bitliu@tencent.com> * project: 2025 Q4 roadmap (#487) * project: q4 roadmap * project: q4 roadmap * project: q4 roadmap * more * more * more * more * feat: add shelleck precommit hook (#488) * feat: add shelleck precommit hook Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat: add shelleck precommit hook Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat: add shelleck precommit hook Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> --------- Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * project: add q4 roadmap news (#495) * fix missing shellcheck in pre-commit image (#497) Signed-off-by: carlory <baofa.fan@daocloud.io> * infra: update tools (#501) Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat(demo): enhance OpenShift demo scripts with improved UX (#478) - Reduce model selection test to 4 categories (2×Model-A, 2×Model-B) - Add new "Classification Examples" option calling curl-examples.sh - Update reasoning examples to avoid cache hits from previous tests - Remove benign examples from PII and Jailbreak tests (show only attacks) - Enhance live-semantic-router-logs.sh with better color visibility: - Fix duplicate "WITH SCORE" text in classification output - Fix CACHE HIT background color extending over timestamp - Distinguish reasoning enabled vs disabled messages - Remove redundant "(standard routing)" text - Add background colors for Model-A/Model-B routing display These improvements make the live demo clearer and more impactful for presentations and demonstrations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Yossi Ovadia <yovadia@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> * fix: fix precommit Argument list too long error (#502) Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat: enforce milvus dial timeout if set (#503) Signed-off-by: cryo <zdtna412@gmail.com> * Add IETF draft publication: Multi-Provider Extensions for Agentic AI Inference APIs (#506) * Initial plan * Add new IETF draft publication for Multi-Provider Extensions for Agentic AI Inference APIs Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Allow semantic cache similarity threshold to be set at the category level (#493) * Initial plan * Add category-level cache settings: enabled and similarity_threshold Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Add comprehensive tests for category-level cache settings Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Update config files and documentation for category-level cache settings - Updated 7 config YAML files (development, production, testing, e2e, and 3 recipes) with commented examples of category-level cache settings - Added comprehensive documentation section explaining category-level cache configuration - Updated semantic cache overview and in-memory cache docs with category-level examples - Added best practices for threshold selection and privacy considerations Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Remove duplicate code in FindSimilar functions Refactored FindSimilar() to delegate to FindSimilarWithThreshold() with default threshold instead of duplicating the entire implementation. This eliminates 226 lines of duplicate code across inmemory_cache.go and milvus_cache.go. Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Update src/semantic-router/pkg/extproc/request_handler.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Revert changes from unsigned commit ae39fe2 Restored the classificationText empty check that was removed in the previous commit. Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> Co-authored-by: Huamin Chen <rootfs@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Allow jailbreak detection and threshold to be configured at the category level (#508) * Initial plan * Add category-level jailbreak detection configuration Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Add documentation for category-level jailbreak settings Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Update documentation for category-level jailbreak detection - Add category-level jailbreak configuration to jailbreak-protection.md - Update category configuration docs with jailbreak_enabled parameter - Add security-focused configuration example - Update global configuration docs with category override notes - Update README to mention fine-grained security control Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Add category-level jailbreak threshold configuration - Add JailbreakThreshold field to Category struct - Add GetJailbreakThresholdForCategory helper method - Create CheckForJailbreakWithThreshold and AnalyzeContentForJailbreakWithThreshold methods - Update performSecurityChecks to use category-specific threshold - Add 5 comprehensive tests for threshold configuration - Update example configs with threshold tuning examples - Update documentation with threshold configuration and tuning guidelines - Add threshold tuning guide with recommendations for different category types Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Allow PII detection threshold to be set at the category level (#510) * Initial plan * Add category-level PII threshold support Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Update documentation with API integration notes Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Fix markdown linting issues Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Fix: The caller information points to the wrapper function instead of the actual call location (#518) Signed-off-by: carlory <baofa.fan@daocloud.io> * feat: Implement hybrid cache that use in-memory index and milvus based doc store (#504) * feat: add HNSW index to inmemory semantic cache and implement hybrid cache that use in-memory index and milvus based doc store Signed-off-by: Huamin Chen <hchen@redhat.com> * chore: run go mod tidy to clean up module dependencies Signed-off-by: Huamin Chen <hchen@redhat.com> * conditionally build candle cuda support Signed-off-by: Huamin Chen <hchen@redhat.com> * rebuild index upon restart Signed-off-by: Huamin Chen <hchen@redhat.com> * precommit fix Signed-off-by: Huamin Chen <hchen@redhat.com> * fix precommit Signed-off-by: Huamin Chen <hchen@redhat.com> * fix precommit Signed-off-by: Huamin Chen <hchen@redhat.com> * fix precommit Signed-off-by: Huamin Chen <hchen@redhat.com> * disable cuda build on ci Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> --------- Signed-off-by: Huamin Chen <hchen@redhat.com> --------- Signed-off-by: carlory <baofa.fan@daocloud.io> Signed-off-by: JaredforReal <w13431838023@gmail.com> Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> Signed-off-by: Yossi Ovadia <yovadia@redhat.com> Signed-off-by: cryo <zdtna412@gmail.com> Signed-off-by: Huamin Chen <hchen@redhat.com> Co-authored-by: 杨朱 · Kiki <baofa.fan@daocloud.io> Co-authored-by: Jared <w13431838023@gmail.com> Co-authored-by: bitliu <bitliu@tencent.com> Co-authored-by: shown <yuluo08290126@gmail.com> Co-authored-by: Yossi Ovadia <yovadia@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: cryo <zdtna412@gmail.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Candle refactoring to main (#524) * Update test description from Math to General (#483) Signed-off-by: carlory <baofa.fan@daocloud.io> * feat: add HuggingChat support (#477) * add chat ui to dashboard and docker compose & refactor dashboard/backend/ Signed-off-by: JaredforReal <w13431838023@gmail.com> * try fix network error Signed-off-by: JaredforReal <w13431838023@gmail.com> * more --------- Signed-off-by: JaredforReal <w13431838023@gmail.com> Co-authored-by: bitliu <bitliu@tencent.com> * project: 2025 Q4 roadmap (#487) * project: q4 roadmap * project: q4 roadmap * project: q4 roadmap * more * more * more * more * feat: add shelleck precommit hook (#488) * feat: add shelleck precommit hook Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat: add shelleck precommit hook Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat: add shelleck precommit hook Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> --------- Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * project: add q4 roadmap news (#495) * fix missing shellcheck in pre-commit image (#497) Signed-off-by: carlory <baofa.fan@daocloud.io> * infra: update tools (#501) Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat(demo): enhance OpenShift demo scripts with improved UX (#478) - Reduce model selection test to 4 categories (2×Model-A, 2×Model-B) - Add new "Classification Examples" option calling curl-examples.sh - Update reasoning examples to avoid cache hits from previous tests - Remove benign examples from PII and Jailbreak tests (show only attacks) - Enhance live-semantic-router-logs.sh with better color visibility: - Fix duplicate "WITH SCORE" text in classification output - Fix CACHE HIT background color extending over timestamp - Distinguish reasoning enabled vs disabled messages - Remove redundant "(standard routing)" text - Add background colors for Model-A/Model-B routing display These improvements make the live demo clearer and more impactful for presentations and demonstrations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Yossi Ovadia <yovadia@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> * fix: fix precommit Argument list too long error (#502) Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat: enforce milvus dial timeout if set (#503) Signed-off-by: cryo <zdtna412@gmail.com> * Add IETF draft publication: Multi-Provider Extensions for Agentic AI Inference APIs (#506) * Initial plan * Add new IETF draft publication for Multi-Provider Extensions for Agentic AI Inference APIs Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Allow semantic cache similarity threshold to be set at the category level (#493) * Initial plan * Add category-level cache settings: enabled and similarity_threshold Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Add comprehensive tests for category-level cache settings Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Update config files and documentation for category-level cache settings - Updated 7 config YAML files (development, production, testing, e2e, and 3 recipes) with commented examples of category-level cache settings - Added comprehensive documentation section explaining category-level cache configuration - Updated semantic cache overview and in-memory cache docs with category-level examples - Added best practices for threshold selection and privacy considerations Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Remove duplicate code in FindSimilar functions Refactored FindSimilar() to delegate to FindSimilarWithThreshold() with default threshold instead of duplicating the entire implementation. This eliminates 226 lines of duplicate code across inmemory_cache.go and milvus_cache.go. Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Update src/semantic-router/pkg/extproc/request_handler.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Revert changes from unsigned commit ae39fe2 Restored the classificationText empty check that was removed in the previous commit. Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> Co-authored-by: Huamin Chen <rootfs@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Allow jailbreak detection and threshold to be configured at the category level (#508) * Initial plan * Add category-level jailbreak detection configuration Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Add documentation for category-level jailbreak settings Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Update documentation for category-level jailbreak detection - Add category-level jailbreak configuration to jailbreak-protection.md - Update category configuration docs with jailbreak_enabled parameter - Add security-focused configuration example - Update global configuration docs with category override notes - Update README to mention fine-grained security control Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Add category-level jailbreak threshold configuration - Add JailbreakThreshold field to Category struct - Add GetJailbreakThresholdForCategory helper method - Create CheckForJailbreakWithThreshold and AnalyzeContentForJailbreakWithThreshold methods - Update performSecurityChecks to use category-specific threshold - Add 5 comprehensive tests for threshold configuration - Update example configs with threshold tuning examples - Update documentation with threshold configuration and tuning guidelines - Add threshold tuning guide with recommendations for different category types Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Allow PII detection threshold to be set at the category level (#510) * Initial plan * Add category-level PII threshold support Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Update documentation with API integration notes Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Fix markdown linting issues Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Fix: The caller information points to the wrapper function instead of the actual call location (#518) Signed-off-by: carlory <baofa.fan@daocloud.io> * feat: Implement hybrid cache that use in-memory index and milvus based doc store (#504) * feat: add HNSW index to inmemory semantic cache and implement hybrid cache that use in-memory index and milvus based doc store Signed-off-by: Huamin Chen <hchen@redhat.com> * chore: run go mod tidy to clean up module dependencies Signed-off-by: Huamin Chen <hchen@redhat.com> * conditionally build candle cuda support Signed-off-by: Huamin Chen <hchen@redhat.com> * rebuild index upon restart Signed-off-by: Huamin Chen <hchen@redhat.com> * precommit fix Signed-off-by: Huamin Chen <hchen@redhat.com> * fix precommit Signed-off-by: Huamin Chen <hchen@redhat.com> * fix precommit Signed-off-by: Huamin Chen <hchen@redhat.com> * fix precommit Signed-off-by: Huamin Chen <hchen@redhat.com> * disable cuda build on ci Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> --------- Signed-off-by: Huamin Chen <hchen@redhat.com> --------- Signed-off-by: carlory <baofa.fan@daocloud.io> Signed-off-by: JaredforReal <w13431838023@gmail.com> Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> Signed-off-by: Yossi Ovadia <yovadia@redhat.com> Signed-off-by: cryo <zdtna412@gmail.com> Signed-off-by: Huamin Chen <hchen@redhat.com> Co-authored-by: 杨朱 · Kiki <baofa.fan@daocloud.io> Co-authored-by: Jared <w13431838023@gmail.com> Co-authored-by: bitliu <bitliu@tencent.com> Co-authored-by: shown <yuluo08290126@gmail.com> Co-authored-by: Yossi Ovadia <yovadia@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: cryo <zdtna412@gmail.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Merge candle refactoring 3 (#525) * Update test description from Math to General (#483) Signed-off-by: carlory <baofa.fan@daocloud.io> * feat: add HuggingChat support (#477) * add chat ui to dashboard and docker compose & refactor dashboard/backend/ Signed-off-by: JaredforReal <w13431838023@gmail.com> * try fix network error Signed-off-by: JaredforReal <w13431838023@gmail.com> * more --------- Signed-off-by: JaredforReal <w13431838023@gmail.com> Co-authored-by: bitliu <bitliu@tencent.com> * project: 2025 Q4 roadmap (#487) * project: q4 roadmap * project: q4 roadmap * project: q4 roadmap * more * more * more * more * feat: add shelleck precommit hook (#488) * feat: add shelleck precommit hook Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat: add shelleck precommit hook Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat: add shelleck precommit hook Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> --------- Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * project: add q4 roadmap news (#495) * fix missing shellcheck in pre-commit image (#497) Signed-off-by: carlory <baofa.fan@daocloud.io> * infra: update tools (#501) Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat(demo): enhance OpenShift demo scripts with improved UX (#478) - Reduce model selection test to 4 categories (2×Model-A, 2×Model-B) - Add new "Classification Examples" option calling curl-examples.sh - Update reasoning examples to avoid cache hits from previous tests - Remove benign examples from PII and Jailbreak tests (show only attacks) - Enhance live-semantic-router-logs.sh with better color visibility: - Fix duplicate "WITH SCORE" text in classification output - Fix CACHE HIT background color extending over timestamp - Distinguish reasoning enabled vs disabled messages - Remove redundant "(standard routing)" text - Add background colors for Model-A/Model-B routing display These improvements make the live demo clearer and more impactful for presentations and demonstrations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Yossi Ovadia <yovadia@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> * fix: fix precommit Argument list too long error (#502) Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> * feat: enforce milvus dial timeout if set (#503) Signed-off-by: cryo <zdtna412@gmail.com> * Add IETF draft publication: Multi-Provider Extensions for Agentic AI Inference APIs (#506) * Initial plan * Add new IETF draft publication for Multi-Provider Extensions for Agentic AI Inference APIs Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Allow semantic cache similarity threshold to be set at the category level (#493) * Initial plan * Add category-level cache settings: enabled and similarity_threshold Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Add comprehensive tests for category-level cache settings Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Update config files and documentation for category-level cache settings - Updated 7 config YAML files (development, production, testing, e2e, and 3 recipes) with commented examples of category-level cache settings - Added comprehensive documentation section explaining category-level cache configuration - Updated semantic cache overview and in-memory cache docs with category-level examples - Added best practices for threshold selection and privacy considerations Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Remove duplicate code in FindSimilar functions Refactored FindSimilar() to delegate to FindSimilarWithThreshold() with default threshold instead of duplicating the entire implementation. This eliminates 226 lines of duplicate code across inmemory_cache.go and milvus_cache.go. Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> * Update src/semantic-router/pkg/extproc/request_handler.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Revert changes from unsigned commit ae39fe2 Restored the classificationText empty check that was removed in the previous commit. Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> Co-authored-by: Huamin Chen <rootfs@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Allow jailbreak detection and threshold to be configured at the category level (#508) * Initial plan * Add category-level jailbreak detection configuration Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Add documentation for category-level jailbreak settings Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Update documentation for category-level jailbreak detection - Add category-level jailbreak configuration to jailbreak-protection.md - Update category configuration docs with jailbreak_enabled parameter - Add security-focused configuration example - Update global configuration docs with category override notes - Update README to mention fine-grained security control Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Add category-level jailbreak threshold configuration - Add JailbreakThreshold field to Category struct - Add GetJailbreakThresholdForCategory helper method - Create CheckForJailbreakWithThreshold and AnalyzeContentForJailbreakWithThreshold methods - Update performSecurityChecks to use category-specific threshold - Add 5 comprehensive tests for threshold configuration - Update example configs with threshold tuning examples - Update documentation with threshold configuration and tuning guidelines - Add threshold tuning guide with recommendations for different category types Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Allow PII detection threshold to be set at the category level (#510) * Initial plan * Add category-level PII threshold support Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Update documentation with API integration notes Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Fix markdown linting issues Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * Fix: The caller information points to the wrapper function instead of the actual call location (#518) Signed-off-by: carlory <baofa.fan@daocloud.io> * feat: Implement hybrid cache that use in-memory index and milvus based doc store (#504) * feat: add HNSW index to inmemory semantic cache and implement hybrid cache that use in-memory index and milvus based doc store Signed-off-by: Huamin Chen <hchen@redhat.com> * chore: run go mod tidy to clean up module dependencies Signed-off-by: Huamin Chen <hchen@redhat.com> * conditionally build candle cuda support Signed-off-by: Huamin Chen <hchen@redhat.com> * rebuild index upon restart Signed-off-by: Huamin Chen <hchen@redhat.com> * precommit fix Signed-off-by: Huamin Chen <hchen@redhat.com> * fix precommit Signed-off-by: Huamin Chen <hchen@redhat.com> * fix precommit Signed-off-by: Huamin Chen <hchen@redhat.com> * fix precommit Signed-off-by: Huamin Chen <hchen@redhat.com> * disable cuda build on ci Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> * review feedback Signed-off-by: Huamin Chen <hchen@redhat.com> --------- Signed-off-by: Huamin Chen <hchen@redhat.com> * merge main to feat branch Signed-off-by: Huamin Chen <hchen@redhat.com> --------- Signed-off-by: carlory <baofa.fan@daocloud.io> Signed-off-by: JaredforReal <w13431838023@gmail.com> Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> Signed-off-by: Yossi Ovadia <yovadia@redhat.com> Signed-off-by: cryo <zdtna412@gmail.com> Signed-off-by: Huamin Chen <hchen@redhat.com> Co-authored-by: 杨朱 · Kiki <baofa.fan@daocloud.io> Co-authored-by: Jared <w13431838023@gmail.com> Co-authored-by: bitliu <bitliu@tencent.com> Co-authored-by: shown <yuluo08290126@gmail.com> Co-authored-by: Yossi Ovadia <yovadia@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: cryo <zdtna412@gmail.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com> * chore: fix unit test (#527) * chore: fix unit test Signed-off-by: Huamin Chen <hchen@redhat.com> * fix go vet Signed-off-by: Huamin Chen <hchen@redhat.com> * fix ci Signed-off-by: Huamin Chen <hchen@redhat.com> * fix ci Signed-off-by: Huamin Chen <hchen@redhat.com> * split test-binding to two stages on ci Signed-off-by: Huamin Chen <hchen@redhat.com> * ignore test failure due to embeddinggemma restriction Signed-off-by: Huamin Chen <hchen@redhat.com> * reorder ci test sequences to avoid missing models Signed-off-by: Huamin Chen <hchen@redhat.com> --------- Signed-off-by: Huamin Chen <hchen@redhat.com> * refactor: Replace lazy_static with OnceLock for zero-cost concurrent reads based on review (#528) * refactor: Replace lazy_static with OnceLock for zero-cost concurrent reads based on review #266 (comment) Signed-off-by: Huamin Chen <hchen@redhat.com> * update tests Signed-off-by: Huamin Chen <hchen@redhat.com> --------- Signed-off-by: Huamin Chen <hchen@redhat.com> * chore: fix lint error (#530) Signed-off-by: Huamin Chen <hchen@redhat.com> * Fix lint error2 (#531) * chore: fix lint error Signed-off-by: Huamin Chen <hchen@redhat.com> * chore: fix lint error Signed-off-by: Huamin Chen <hchen@redhat.com> --------- Signed-off-by: Huamin Chen <hchen@redhat.com> --------- Signed-off-by: OneZero-Y <aukovyps@163.com> Signed-off-by: Huamin Chen <hchen@redhat.com> Signed-off-by: carlory <baofa.fan@daocloud.io> Signed-off-by: JaredforReal <w13431838023@gmail.com> Signed-off-by: yuluo-yx <yuluo08290126@gmail.com> Signed-off-by: Yossi Ovadia <yovadia@redhat.com> Signed-off-by: cryo <zdtna412@gmail.com> Co-authored-by: OneZero-Y <aukovyps@163.com> Co-authored-by: 杨朱 · Kiki <baofa.fan@daocloud.io> Co-authored-by: Jared <w13431838023@gmail.com> Co-authored-by: bitliu <bitliu@tencent.com> Co-authored-by: shown <yuluo08290126@gmail.com> Co-authored-by: Yossi Ovadia <yovadia@redhat.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: cryo <zdtna412@gmail.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: rootfs <7062400+rootfs@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Xunzhuo <48784001+Xunzhuo@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
UNIFIED_CLASSIFIERis defined in bothinit.rsandclassify.rs, resulting in two independent global state instancesclassify_batch_with_loraholds thePARALLEL_LORA_ENGINElock throughout the entire inference process, blocking concurrent requestsWhich issue(s) this PR fixes:
Part of #266