feat: v2.5.0 - #467
Merged
Merged
Conversation
* docs: update wechat (#417) * docs: update wechat (#438) * docs: update wechat (#443) * feat: add LiteLLM as AI gateway provider (#440) * feat: add LiteLLM as AI gateway provider * chore: tighten litellm pin to >=1.80.0,<1.87.0 * fix: address review feedback on LiteLLM provider - Add embedding client initialization in create_client() to match BaseOpenAI pattern, ensuring RAG evaluators work with BaseLiteLLM - Add defensive checks for empty response.choices and None content in send_messages() to prevent IndexError and literal 'None' strings - Skip litellm tests when litellm is not installed (fixes CI test job) - Fix flake8 E302 blank line violation in test file - Add tests for empty choices and None content edge cases --------- Co-authored-by: Aarish Irani <rheagalfire@gmail.com> --------- Co-authored-by: Aarish Alam <arishalam121@gmail.com> Co-authored-by: Aarish Irani <rheagalfire@gmail.com>
feat: DatasetSqlArgs添加engine_args属性
feat: score实时更新
feat: SummaryModel增加type_count
- add relevance, effectiveness, and authority evaluators - add standalone and combined executor-based evaluation scripts - support query-level and result-level classified outputs - improve LLM response parsing and content issue detection - set overall weights to 0.7/0.2/0.1 - add evaluator tests and usage documentation
- add relevance, effectiveness, and authority evaluators - add standalone and combined executor-based evaluation scripts - support query-level and result-level classified outputs - improve LLM response parsing and content issue detection - set overall weights to 0.7/0.2/0.1 - add evaluator tests and usage documentation
- add relevance, effectiveness, and authority evaluators - add standalone and combined executor-based evaluation scripts - support query-level and result-level classified outputs - improve LLM response parsing and content issue detection - set overall weights to 0.7/0.2/0.1 - add evaluator tests and usage documentation
# Conflicts: # dingo/model/llm/llm_search_result_authority.py # examples/retrieval/sdk_eval_authority.py # examples/retrieval/sdk_eval_effectiveness.py # examples/retrieval/sdk_eval_relevancy.py # examples/retrieval/sdk_eval_search_result.py
优化了有效性评测：取消字符长度判断
fix：修复了LLM的jsonl解析格式
Feature/lld ir
# Conflicts: # test/scripts/model/rule/test_rule_common.py
之前 dingo/__init__.py 用 os.environ.setdefault 把 HF_ENDPOINT 默认 指向 hf-mirror.com，会对所有未显式设置该变量的环境生效，导致 CI （美国 runner）被强制走中国镜像、加载 HF 数据集 chupei/format-text 时跨境连接超时，报 LocalEntryNotFoundError 使 Integration Test 失败。 改为在幻觉检测中文文档中指导用户按需 export HF_ENDPOINT，不侵入全局 环境，既满足国内用户走镜像的需求，也不影响能直连官网的环境（如 CI）。 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix: LLMPerspective: UnboundLocalError: cannot access local variable …
RuleHallucinationHHEM.eval 里 result.score 赋值一直是注释状态，导致
EvalDetail.score 恒为 None；而示例脚本用 {getattr(result,'score','N/A'):.3f}
格式化——因 score 属性存在只是值为 None，getattr 默认值不生效，最终
None:.3f 触发 "unsupported format string passed to NoneType.__format__"。
- rule_hallucination_hhem.py: 取消注释 result.score = avg_hallucination_score
（位于 if/else 之前，HALLUCINATION_DETECTED 与 NO_HALLUCINATION 两分支都回填）
- sdk_rule_hhem_detection.py: 6 处格式化改为对 None 安全（有值 .3f，None 显示 N/A）
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dingo-saas 运行 RuleHallucinationHHEM 报 "'HHEMv2ForSequenceClassification' object has no attribute 'all_tied_weights_keys'"。 根因：lmdeploy 硬性要求 transformers>=4.56，被并入 [all]/optional 后拉高了 transformers 版本，而 HHEM 官方 remote code 为旧版 transformers 编写， 4.49+ 的加载流程会访问 all_tied_weights_keys（旧 remote code 未实现）。 - setup.py: lmdeploy 单独成 extra（lmdeploy），不再并入 optional/all， 避免同一环境内拉高 transformers 导致 HHEM 无法加载 - optional.txt: 移除 lmdeploy 行 - hhem_integration.txt: transformers 上限收紧为 <4.49，并注明原因 - base_lmdeploy_apiclient.py: create_client 内惰性 import lmdeploy， 缺失时抛清晰 ImportError 指引 pip install dingo-python[lmdeploy] Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix：HHEM example
# Conflicts: # docs/metrics.md
feat: add perspective check example
feat: tc609
将本地幻觉检测模型从 Vectara HHEM-2.1-Open 换为 lytang/MiniCheck-Flan-T5-Large（保留类名与注册 id 以向后兼容）。 - 标准 T5 模型，去掉 transformers<4.49 版本限制 - LLM-AggreFact 准确率更高（75.0 vs 71.8） - 忠实复刻官方 flan-t5 推理（predict 前缀、单步 decoder、 label token [3,209] softmax、按 chunk 取 max） - 同步更新依赖、示例与中英文幻觉检测指南、自动生成的 metrics.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
feat: RuleHallucinationHHEM 底层模型改用 MiniCheck-Flan-T5-Large
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.