Skip to content

feat(trace-aliyun): GenAI spans for LLM TTFT, agent ReAct, tools, and failed LLM nodes - #39339

Open
123liuziming wants to merge 4 commits into
langgenius:mainfrom
123liuziming:main
Open

feat(trace-aliyun): GenAI spans for LLM TTFT, agent ReAct, tools, and failed LLM nodes#39339
123liuziming wants to merge 4 commits into
langgenius:mainfrom
123liuziming:main

Conversation

@123liuziming

@123liuziming 123liuziming commented Jul 21, 2026

Copy link
Copy Markdown

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Improve Aliyun Trace (trace-aliyun) so workflow/agent telemetry better matches GenAI semantic conventions and is more useful when model or tool calls fail.

LLM

  • Record gen_ai.response.time_to_first_token (nanoseconds) on workflow LLM spans and message LLM spans when TTFT is available.
  • On LLM invoke failure (when graphon leaves process_data empty), fall back to node inputs / error / error_type so attributes such as gen_ai.request.model, gen_ai.provider.name, gen_ai.prompt, and gen_ai.completion are still populated when possible.
    • Note: structured gen_ai.input.messages still depends on process_data.prompts, which is only written after a successful invoke today.

Agent / ReAct

  • Emit AGENT spans (instead of TASK) for agent-strategy nodes, with gen_ai.operation.name=invoke_agent, agent name, and token usage.
  • Parse agent execution logs in node outputs to build per-round ReAct STEP spans and child call spans, anchoring monotonic log timestamps to the node start time.
  • Classify agent log entries correctly:
    • Thought → LLM
    • CALL …TOOL (previously misclassified as LLM because CALL logs also set metadata.provider)

Tool

  • Align tool span attributes with GenAI conventions:
    • Required: gen_ai.span.kind=TOOL, gen_ai.operation.name=execute_tool
    • Recommended/optional: gen_ai.tool.name, gen_ai.tool.type, gen_ai.tool.description, gen_ai.tool.call.id, gen_ai.tool.call.arguments, gen_ai.tool.call.result
  • Apply across message tool_trace, workflow tool nodes, and agent ReAct CALL spans.
  • Remove legacy tool.name / tool.description / tool.parameters keys.

Tests

  • Unit coverage for TTFT, agent STEP/LLM/TOOL span building, failed-LLM fallback, GenAI tool attribute helpers, and semconv constants.

Screenshots

Before After
Agent CALL logs shown as LLM; failed LLM spans missing model/prompt; tool attrs used legacy tool.* keys Agent CALL → TOOL; failed LLM spans keep model/provider/inputs when available; tool attrs use gen_ai.tool.*

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods

From Cursor

…agent nodes

- Record gen_ai.response.time_to_first_token (nanoseconds) on LLM spans
  for both workflow LLM nodes and message traces
- Emit AGENT-kind spans (instead of TASK) for agent-strategy nodes with
  invoke_agent operation name, agent name and token usage
- Parse the agent execution log in outputs to build per-round ReAct STEP
  spans and their child LLM call spans, anchoring monotonic timestamps
  to the node start time

Change-Id: I178ab61a36389b1698e7dda64836877d9507b51f
Co-developed-by: Cursor <noreply@cursor.com>
@dosubot dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jul 21, 2026
@crazywoola
crazywoola marked this pull request as draft July 21, 2026 06:43
123liuziming and others added 2 commits July 28, 2026 18:28
Agent ReAct tool invocations were misclassified as LLM because CALL logs
also set metadata.provider. Classify CALL labels as TOOL and keep Thought
entries as LLM.

Co-authored-by: Cursor <cursoragent@cursor.com>
…enAI

Fall back to node inputs/error when process_data is empty on invoke failure.
Replace legacy tool.* attributes with gen_ai.tool.* semantic conventions.

Co-authored-by: Cursor <cursoragent@cursor.com>
@123liuziming 123liuziming changed the title [WIP] feat(trace-aliyun): add LLM TTFT attribute and AGENT/ReAct spans for … feat(trace-aliyun): add LLM TTFT attribute and AGENT/ReAct spans for agent nodes Jul 28, 2026
@123liuziming
123liuziming marked this pull request as ready for review July 28, 2026 11:43
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jul 28, 2026
@123liuziming 123liuziming changed the title feat(trace-aliyun): add LLM TTFT attribute and AGENT/ReAct spans for agent nodes feat(trace-aliyun): GenAI spans for LLM TTFT, agent ReAct, tools, and failed LLM nodes Jul 28, 2026
@123liuziming

Copy link
Copy Markdown
Author

@TreyDong @laipz8200 Hey, could you review my PR when you have a chance?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant