fix(analytics): Langfuse cost/token tracking for agentflows + env override#1033
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…n usage and model name for Langfuse/LangSmith in agentflows Cherry-picks FlowiseAI/Flowise@11261d20d with conflict resolution to preserve our fork's custom analytics hooks (callbacks setup, parent trace propagation). Key changes: - onLLMEnd now accepts structured output objects (not just strings) - Extracts usageMetadata + responseMetadata from AIMessage responses - Adds model name and usage data to Langfuse generation.end() calls - Removes langfuseCallbacksActive guard from onLLMStart so generations are always created — ensures cost tracking works for agentflows with parent traces (env override path) - Restores full tag set (Name, chatflow_id, chat_id, chatmessage_id) on both handlerConfig and parentLangfuseTrace.update() - Adds flushAsync() to ensure generation updates reach Langfuse - LLM.ts, Agent.ts, ConditionAgent.ts pass full output to onLLMEnd Fixes: Langfuse cost/token tracking showing $0 for all agentflow traces. Upstream: FlowiseAI#5764, fixes FlowiseAI#4779, FlowiseAI#5015, FlowiseAI#4311
maxtechera
force-pushed
the
fix/langfuse-cost-tracking
branch
from
ddc9e1d to
b657487
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cherry-picks upstream FlowiseAI/Flowise#5764 with additional fixes for our fork's env-override Langfuse integration.
Problem: All agentflow traces show $0 cost in Langfuse. Token usage, model name, and cost are missing because the
langfuseCallbacksActiveguard inonLLMStartskipped generation creation for flows with parent traces — which includes all agentflows and sub-flows called via Execute Flow.Root cause: Known upstream issue (#4779 — 17 thumbsup, #5015, #4311).
Changes
From upstream PR FlowiseAI#5764
onLLMEndnow accepts structured output objects (string | object) with usage + model extractionresponseMetadatamodelandusage(promptTokens/completionTokens/totalTokens) togeneration.end()langfuse.flushAsync()to ensure data reaches Langfuse before request completesonLLMEndchatflow.nameas trace name instead of hardcodedAgentflowFork-specific fixes (on top of upstream)
langfuseCallbacksActiveguard fromonLLMStart— generations are now always created soonLLMEndcan record cost even when a parent trace exists (env override path)onLLMError— errors are now recorded withlevel: ERRORon the generationhandlerConfigandparentLangfuseTrace.update():Name:,chatflow_id:,chat_id:,chatmessage_id:(upstream cherry-pick had reverted these toName:only)Verification matrix
Risk
Low-medium. The generation guard removal means both the manual
AnalyticHandlerpath AND the LangChainCallbackHandlermay create generations — potentially causing duplicate generation entries in Langfuse for the same LLM call. This is cosmetic (both will have correct data) and preferable to missing cost data entirely.Related