feat: bump versions to 1.15.15 - #6962
Conversation
A flow reported only that it started. FlowFinishedEvent, FlowFailedEvent, MethodExecutionFailedEvent, MethodExecutionPausedEvent and FlowPausedEvent all reached the console formatter and stopped there, and FlowInputRequestedEvent, FlowInputReceivedEvent and ConversationTurnFailedEvent had no listener at all - so success rate, failure rate and every HITL pause were unmeasurable. Adds flow:completed, flow:failed, flow:method_failed, flow:paused, flow:hitl_paused, flow:input_requested, flow:input_received and flow:conversation_turn_failed as feature-usage spans, which the existing feature-usage aggregation already reads. Deliberately does not hold the Flow Execution span open to measure duration: flow_executions_daily_target counts those spans at start, so a run that never finishes would disappear from the count entirely. Duration needs its own span. Counts only - flow names, method names, error text and flow state are never recorded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Adds a Flow Completed span carrying flow_name, duration_ms and outcome,
emitted when a flow finishes or fails. Elapsed time comes from a monotonic
stamp taken at flow start and cleared on use.
Kept separate from the Flow Execution span rather than holding that one open:
it is emitted and closed at start and the daily aggregate counts it, so
holding it would drop every run that is killed or crashes from the execution
count. A killed run now simply has no Flow Completed row, and the count is
unaffected.
Elapsed time is an explicit duration_ms attribute rather than the span's own
duration, which the ingestion pipeline stores as a suffixed string
("0.0000184s") that downstream aggregation parses to zero.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Two gaps found while testing the pause/resume path end to end. Resumed runs were invisible. There is no resume event: a restored run re-enters through kickoff(), so it looked identical to a fresh start. flow:resumed is derived from _is_execution_resuming at flow start, which makes flow:paused - flow:resumed the abandonment rate. Flow counts are dominated by CrewAI's own AgentExecutor, which is itself a Flow and runs once per agent execution - it is the top flow in the warehouse by a wide margin. Nothing distinguished it from a user's flows except guessing at the name. Both Flow Execution and Flow Completed now carry origin: "internal" when the flow class is defined under crewai.*, "user" otherwise. Tagging only the new span would have left the existing daily count unsplittable. Both span methods take origin with a default, so their signatures stay backward compatible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Two findings from review, both confirmed against the code. Outcome features counted CrewAI's own flows. The agent executor, memory encoding and memory recall are all Flows and all set suppress_flow_events; they run far more often than anything a user wrote, so flow:completed, flow:failed and flow:method_failed were mostly bookkeeping. Those three are now emitted only for flows the caller wrote. Internal outcomes are still recorded on the Flow Completed span, which carries origin. flow:resumed counted checkpoint restores. _is_execution_resuming is set both by from_pending (a human pause) and by a checkpoint restore that never paused for anyone, so resumes could exceed pauses and the abandonment rate was unusable. Keyed off _pending_feedback_context instead, which only from_pending sets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Three findings from review, all confirmed against the code. Gating on suppress_flow_events was wrong. That flag asks for console quiet and is a public field, so a caller who set it on their own flow silently lost flow:completed, flow:failed and flow:method_failed. Deciding origin from the defining module was also wrong. Flow.from_declaration() returns a Flow typed in crewai.flow.flow, so a caller's declarative flow was reported as one of CrewAI's own - the inversion this split exists to prevent. Both had the same root cause: the discriminator was inferred. Flow now declares is_crewai_internal, set on the agent executor and the memory encoding/recall flows, and one helper serves both origin and the outcome gate. A failed conversational session was reported as completed. Its session closes with FlowFinishedEvent whatever happened, so a failed turn produced flow:conversation_turn_failed and flow:completed together. The turn failure is now recorded on the flow and read back when the session finishes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Flow start, completion, pause and method failure are lifecycle facts, and the lifecycle is reported as spans everywhere else. Reporting them through feature usage put them in a table that aggregates on the feature string alone - it cannot carry origin, duration or outcome, so those signals could never be split between a user's flows and the ones CrewAI runs for itself. Adds Flow Paused and Flow Method Failed spans, and a resumed marker on Flow Execution so a run restored from a pause is not counted as a second fresh start. Removes the duplicate feature rows for completed, failed, method_failed, paused and resumed - every one of those facts is now on a span, with more attached to it than the feature row ever carried. Feature usage keeps only genuine adoption signals: flow:hitl_paused, flow:input_requested, flow:input_received and flow:conversation_turn_failed. Also clears the conversational turn-failure flag on every terminal path. A turn that failed without deferred finalization ends via FlowFailedEvent, and the flag left set there marked the next run on that instance as failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
…ument Adding the resumed marker changed a signature that tests/utilities/test_events.py asserts on exactly, and that assertion was not re-run before pushing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
The test asserted that flow:resumed was absent from feature usage, but that
signal moved onto the Flow Execution span. The assertion could no longer fail,
so a regression that mis-tagged checkpoint restores as resumes would have gone
unnoticed.
Now asserts the resumed attribute, and waits for the handlers: the manual emit
dispatches asynchronously, so the previous shape also read its result before the
listener had run.
Confirmed it discriminates - keying resumed off _is_execution_resuming again
fails it with [('RestoredFlow', True)] == [('RestoredFlow', False)].
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Verified end to end against the live collector and ClickHouse: the pipeline encodes a boolean attribute as the presence of a vBool key, so false arrives as the key simply being absent. That is invisible in the schema and easy to read wrongly - crew_memory is extracted as "the attribute exists" and consequently reports 1 for 99.8% of crews against a field that defaults to False. A string leaves nothing to infer. Confirmed in the warehouse: the emitted span reads resumed = "false". Adds direct coverage for the attributes each flow span records, including both resumed values, and resets the Telemetry singleton in the helper so more than one span method can be exercised per session. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
📝 WalkthroughWalkthroughChangesFlow telemetry
Release metadata alignment
Sequence Diagram(s)sequenceDiagram
participant Flow
participant FlowEventListener
participant Telemetry
Flow->>FlowEventListener: Emit flow start or resume
FlowEventListener->>Telemetry: Record Flow Execution
Flow->>FlowEventListener: Emit pause, input, or failure
FlowEventListener->>Telemetry: Record lifecycle signal
Flow->>FlowEventListener: Emit completion or failure
FlowEventListener->>Telemetry: Record duration and outcome
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 77c68bd. Configure here.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
lib/crewai/tests/telemetry/test_flow_telemetry.py (1)
409-439: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueShare the
StubLLMdefinition between the two Crew-based tests.
test_crewais_own_agent_executor_is_tagged_internalandtest_infrastructure_flows_do_not_pollute_outcome_signalsdeclare identicalStubLLMclasses and build the same agent, task, and crew. Extract a module-levelStubLLMand a small_run_crew()helper.♻️ Proposed refactor
# Module level, near the fixtures: def _run_crew() -> None: """Run a minimal Crew so the internal AgentExecutor flow executes.""" from crewai import Agent, Crew, Task from crewai.llms.base_llm import BaseLLM class StubLLM(BaseLLM): def __init__(self) -> None: super().__init__(model="stub-model") def call(self, messages, **kwargs) -> str: return "Final Answer: done" def supports_function_calling(self) -> bool: return False def supports_stop_words(self) -> bool: return False def get_context_window_size(self) -> int: return 8192 agent = Agent(role="R", goal="G", backstory="B", llm=StubLLM()) task = Task(description="Do it", expected_output="A result", agent=agent) Crew(agents=[agent], tasks=[task]).kickoff()Both tests then call
_run_crew()in place of the duplicated block.Also applies to: 552-590
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/tests/telemetry/test_flow_telemetry.py` around lines 409 - 439, Extract the duplicated StubLLM definition and Crew setup from test_crewais_own_agent_executor_is_tagged_internal and test_infrastructure_flows_do_not_pollute_outcome_signals into a module-level StubLLM and a _run_crew() helper. Keep the helper’s behavior identical, then replace both inline setup blocks with _run_crew() calls.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/edge/en/telemetry.mdx`:
- Line 64: Update the Flow Lifecycle Signals privacy statement to acknowledge
that flow_name is retained and users must keep flow names free of personal
information, while preserving that method names and error messages are not
recorded. Apply equivalent wording in docs/edge/en/telemetry.mdx lines 64-64,
docs/edge/ar/telemetry.mdx lines 64-64, docs/edge/ko/telemetry.mdx lines 62-62,
and docs/edge/pt-BR/telemetry.mdx lines 64-64.
In `@lib/crewai/tests/telemetry/test_flow_telemetry.py`:
- Around line 220-234: Update StubProvider.request_input in
test_requesting_input_reports_both_sides to construct InputResponse with
text="typed answer" instead of the unsupported value argument, preserving the
existing telemetry assertions.
---
Nitpick comments:
In `@lib/crewai/tests/telemetry/test_flow_telemetry.py`:
- Around line 409-439: Extract the duplicated StubLLM definition and Crew setup
from test_crewais_own_agent_executor_is_tagged_internal and
test_infrastructure_flows_do_not_pollute_outcome_signals into a module-level
StubLLM and a _run_crew() helper. Keep the helper’s behavior identical, then
replace both inline setup blocks with _run_crew() calls.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 017a8a1f-9e5d-4ec6-a8aa-183c6aff3948
📒 Files selected for processing (22)
docs/edge/ar/telemetry.mdxdocs/edge/en/telemetry.mdxdocs/edge/ko/telemetry.mdxdocs/edge/pt-BR/telemetry.mdxlib/cli/pyproject.tomllib/cli/src/crewai_cli/__init__.pylib/crewai-core/src/crewai_core/__init__.pylib/crewai-files/src/crewai_files/__init__.pylib/crewai-tools/pyproject.tomllib/crewai-tools/src/crewai_tools/__init__.pylib/crewai/pyproject.tomllib/crewai/src/crewai/__init__.pylib/crewai/src/crewai/events/event_listener.pylib/crewai/src/crewai/experimental/agent_executor.pylib/crewai/src/crewai/flow/runtime/__init__.pylib/crewai/src/crewai/memory/encoding_flow.pylib/crewai/src/crewai/memory/recall_flow.pylib/crewai/src/crewai/telemetry/telemetry.pylib/crewai/tests/telemetry/test_flow_telemetry.pylib/crewai/tests/telemetry/test_telemetry.pylib/crewai/tests/utilities/test_events.pylib/devtools/src/crewai_devtools/__init__.py

Note
Medium Risk
Expands default telemetry collection for flows, a privacy-sensitive area, though method names, errors, and flow state are explicitly excluded. Changes event-listener wiring that runs on every flow execution.
Overview
Previously, flows only reported that they started. Finish, fail, pause, resume, method-failure, and input events reached the console and stopped there. This wires those lifecycle signals into telemetry and tags each flow as internal (agent executor, memory encode/recall) or user-authored so internal bookkeeping no longer swamps daily counts.
New spans and features
Flow Completedwithduration_msandoutcome(completed/failed), emitted separately from start so aborted runs still countFlow PausedandFlow Method Failed(flow name only — no method names or error text)originand a stringresumedflag (keyed off pending feedback, not checkpoint restore)Marks
AgentExecutor,EncodingFlow, andRecallFlowwithis_crewai_internal. Documents the new defaulted Flow Lifecycle Signals row in telemetry docs (EN/AR/KO/PT-BR). Bumps all packages to 1.15.15.Reviewed by Cursor Bugbot for commit 77c68bd. Bugbot is set up for automated code reviews on this repo. Configure here.