Skip to content

feat: add token usage - #459

Merged
e06084 merged 2 commits into
MigoXLab:devfrom
e06084:dev
Jul 29, 2026
Merged

feat: add token usage#459
e06084 merged 2 commits into
MigoXLab:devfrom
e06084:dev

Conversation

@e06084

@e06084 e06084 commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

主要改动:

  • 新增 TokenUsage,EvalDetail 可携带 usage:dingo/io/output/eval_detail.py:12
  • BaseOpenAI/BaseLiteLLM 从 provider response 的 usage 提取 token,并用 LLMCallResult 兼容旧字符串返回:dingo/model/llm/base_openai.py:79
  • summary.json 新增 token_usage 汇总,按字段组合 + evaluator 聚合:dingo/io/output/summary_model.py:52
  • LocalExecutor 增加内部 token_usage_details 通道,即使 result_save.good=False 导致 good 明细不落盘,也会统计实际 LLM 调用 token:dingo/exec/local.py:125
  • 覆盖了自定义 metric、VLM、search relevance/effectiveness、factcheck、RAG 多调用、claims extractor 等绕过主路径的调用点。
  • 更新文档:docs/config.md:104

输出上,单条 LLM 结果会有 usage;summary.json 会有 token_usage。统计只使用服务商返回的 usage,不做本地 tokenizer 估算;provider 不返回时为空。

当 LLM 服务返回 token usage 时,Dingo 会在对应的 EvalDetail 中写入 usage 字段,并在 summary.json 中按字段组合和 evaluator 汇总到 token_usage。该统计来自模型服务商返回的 usage,不会本地估算;如果兼容 API 不返回 usage,则对应字段为空。

单条结果示例:

{
  "metric": "LLMTextQualityV5",
  "status": false,
  "label": ["QUALITY_GOOD"],
  "reason": ["pass"],
  "usage": {
    "prompt_tokens": 812,
    "completion_tokens": 96,
    "total_tokens": 908,
    "reasoning_tokens": null,
    "cached_tokens": null,
    "model": "gpt-4o-mini",
    "provider": "openai",
    "calls": 1,
    "source": "provider"
  }
}

汇总结果示例:

{
  "token_usage": {
    "content": {
      "LLMTextQualityV5": {
        "prompt_tokens": 81200,
        "completion_tokens": 9600,
        "total_tokens": 90800,
        "reasoning_tokens": 0,
        "cached_tokens": 12000,
        "calls": 100,
        "records": 100,
        "models": {"gpt-4o-mini": 100},
        "providers": {"openai": 100},
        "sources": {"provider": 100}
      }
    }
  }
}

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@e06084
e06084 merged commit d55c0ea into MigoXLab:dev Jul 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants