Skip to content

feat: bump versions to 1.15.15 - #6962

Merged
joaomdmoura merged 10 commits into
mainfrom
feat/bump-version-1.15.15
Aug 12, 2026
Merged

feat: bump versions to 1.15.15#6962
joaomdmoura merged 10 commits into
mainfrom
feat/bump-version-1.15.15

Conversation

@joaomdmoura

@joaomdmoura joaomdmoura commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Note

Medium Risk
Expands default telemetry collection for flows, a privacy-sensitive area, though method names, errors, and flow state are explicitly excluded. Changes event-listener wiring that runs on every flow execution.

Overview
Previously, flows only reported that they started. Finish, fail, pause, resume, method-failure, and input events reached the console and stopped there. This wires those lifecycle signals into telemetry and tags each flow as internal (agent executor, memory encode/recall) or user-authored so internal bookkeeping no longer swamps daily counts.

New spans and features

  • Flow Completed with duration_ms and outcome (completed/failed), emitted separately from start so aborted runs still count
  • Flow Paused and Flow Method Failed (flow name only — no method names or error text)
  • Feature usage for HITL pause, input requested/received, and conversation-turn failure
  • Start spans now carry origin and a string resumed flag (keyed off pending feedback, not checkpoint restore)

Marks AgentExecutor, EncodingFlow, and RecallFlow with is_crewai_internal. Documents the new defaulted Flow Lifecycle Signals row in telemetry docs (EN/AR/KO/PT-BR). Bumps all packages to 1.15.15.

Reviewed by Cursor Bugbot for commit 77c68bd. Bugbot is set up for automated code reviews on this repo. Configure here.

joaomdmoura and others added 10 commits August 11, 2026 15:51
A flow reported only that it started. FlowFinishedEvent, FlowFailedEvent,
MethodExecutionFailedEvent, MethodExecutionPausedEvent and FlowPausedEvent all
reached the console formatter and stopped there, and FlowInputRequestedEvent,
FlowInputReceivedEvent and ConversationTurnFailedEvent had no listener at all -
so success rate, failure rate and every HITL pause were unmeasurable.

Adds flow:completed, flow:failed, flow:method_failed, flow:paused,
flow:hitl_paused, flow:input_requested, flow:input_received and
flow:conversation_turn_failed as feature-usage spans, which the existing
feature-usage aggregation already reads.

Deliberately does not hold the Flow Execution span open to measure duration:
flow_executions_daily_target counts those spans at start, so a run that never
finishes would disappear from the count entirely. Duration needs its own span.

Counts only - flow names, method names, error text and flow state are never
recorded.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Adds a Flow Completed span carrying flow_name, duration_ms and outcome,
emitted when a flow finishes or fails. Elapsed time comes from a monotonic
stamp taken at flow start and cleared on use.

Kept separate from the Flow Execution span rather than holding that one open:
it is emitted and closed at start and the daily aggregate counts it, so
holding it would drop every run that is killed or crashes from the execution
count. A killed run now simply has no Flow Completed row, and the count is
unaffected.

Elapsed time is an explicit duration_ms attribute rather than the span's own
duration, which the ingestion pipeline stores as a suffixed string
("0.0000184s") that downstream aggregation parses to zero.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Two gaps found while testing the pause/resume path end to end.

Resumed runs were invisible. There is no resume event: a restored run re-enters
through kickoff(), so it looked identical to a fresh start. flow:resumed is
derived from _is_execution_resuming at flow start, which makes
flow:paused - flow:resumed the abandonment rate.

Flow counts are dominated by CrewAI's own AgentExecutor, which is itself a Flow
and runs once per agent execution - it is the top flow in the warehouse by a
wide margin. Nothing distinguished it from a user's flows except guessing at the
name. Both Flow Execution and Flow Completed now carry origin: "internal" when
the flow class is defined under crewai.*, "user" otherwise. Tagging only the new
span would have left the existing daily count unsplittable.

Both span methods take origin with a default, so their signatures stay
backward compatible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Two findings from review, both confirmed against the code.

Outcome features counted CrewAI's own flows. The agent executor, memory
encoding and memory recall are all Flows and all set suppress_flow_events;
they run far more often than anything a user wrote, so flow:completed,
flow:failed and flow:method_failed were mostly bookkeeping. Those three are now
emitted only for flows the caller wrote. Internal outcomes are still recorded
on the Flow Completed span, which carries origin.

flow:resumed counted checkpoint restores. _is_execution_resuming is set both by
from_pending (a human pause) and by a checkpoint restore that never paused for
anyone, so resumes could exceed pauses and the abandonment rate was unusable.
Keyed off _pending_feedback_context instead, which only from_pending sets.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Three findings from review, all confirmed against the code.

Gating on suppress_flow_events was wrong. That flag asks for console quiet and
is a public field, so a caller who set it on their own flow silently lost
flow:completed, flow:failed and flow:method_failed.

Deciding origin from the defining module was also wrong. Flow.from_declaration()
returns a Flow typed in crewai.flow.flow, so a caller's declarative flow was
reported as one of CrewAI's own - the inversion this split exists to prevent.

Both had the same root cause: the discriminator was inferred. Flow now declares
is_crewai_internal, set on the agent executor and the memory encoding/recall
flows, and one helper serves both origin and the outcome gate.

A failed conversational session was reported as completed. Its session closes
with FlowFinishedEvent whatever happened, so a failed turn produced
flow:conversation_turn_failed and flow:completed together. The turn failure is
now recorded on the flow and read back when the session finishes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Flow start, completion, pause and method failure are lifecycle facts, and the
lifecycle is reported as spans everywhere else. Reporting them through
feature usage put them in a table that aggregates on the feature string alone -
it cannot carry origin, duration or outcome, so those signals could never be
split between a user's flows and the ones CrewAI runs for itself.

Adds Flow Paused and Flow Method Failed spans, and a resumed marker on Flow
Execution so a run restored from a pause is not counted as a second fresh
start. Removes the duplicate feature rows for completed, failed, method_failed,
paused and resumed - every one of those facts is now on a span, with more
attached to it than the feature row ever carried.

Feature usage keeps only genuine adoption signals: flow:hitl_paused,
flow:input_requested, flow:input_received and flow:conversation_turn_failed.

Also clears the conversational turn-failure flag on every terminal path. A turn
that failed without deferred finalization ends via FlowFailedEvent, and the flag
left set there marked the next run on that instance as failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
…ument

Adding the resumed marker changed a signature that tests/utilities/test_events.py
asserts on exactly, and that assertion was not re-run before pushing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
The test asserted that flow:resumed was absent from feature usage, but that
signal moved onto the Flow Execution span. The assertion could no longer fail,
so a regression that mis-tagged checkpoint restores as resumes would have gone
unnoticed.

Now asserts the resumed attribute, and waits for the handlers: the manual emit
dispatches asynchronously, so the previous shape also read its result before the
listener had run.

Confirmed it discriminates - keying resumed off _is_execution_resuming again
fails it with [('RestoredFlow', True)] == [('RestoredFlow', False)].

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
Verified end to end against the live collector and ClickHouse: the pipeline
encodes a boolean attribute as the presence of a vBool key, so false arrives as
the key simply being absent. That is invisible in the schema and easy to read
wrongly - crew_memory is extracted as "the attribute exists" and consequently
reports 1 for 99.8% of crews against a field that defaults to False.

A string leaves nothing to infer. Confirmed in the warehouse: the emitted span
reads resumed = "false".

Adds direct coverage for the attributes each flow span records, including both
resumed values, and resets the Telemetry singleton in the helper so more than
one span method can be exercised per session.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ASfWmW3RGy4qAQm6s8U9jH
@mintlify

mintlify Bot commented Aug 12, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
crewai 🟡 Building Aug 12, 2026, 1:02 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Changes

Flow telemetry

Layer / File(s) Summary
Telemetry contracts and runtime state
lib/crewai/src/crewai/telemetry/telemetry.py, lib/crewai/src/crewai/flow/runtime/__init__.py, lib/crewai/src/crewai/experimental/agent_executor.py, lib/crewai/src/crewai/memory/*
Telemetry spans now record flow origin, resume state, completion duration, outcomes, pauses, and method failures. Runtime classes track internal-flow status and telemetry state.
Lifecycle event handling
lib/crewai/src/crewai/events/event_listener.py
Flow events now emit telemetry for starts, resumes, completions, failures, pauses, inputs, conversation failures, and method failures.
Lifecycle telemetry validation
lib/crewai/tests/telemetry/test_flow_telemetry.py, lib/crewai/tests/telemetry/test_telemetry.py, lib/crewai/tests/utilities/test_events.py
Tests cover outcomes, duration, privacy, origin, resume behavior, suppression behavior, and checkpoint restoration.
Telemetry documentation
docs/edge/*/telemetry.mdx
Localized documentation describes flow lifecycle signals and recorded or excluded data.

Release metadata alignment

Layer / File(s) Summary
Package version updates
lib/cli/src/crewai_cli/__init__.py, lib/crewai-core/src/crewai_core/__init__.py, lib/crewai-files/src/crewai_files/__init__.py, lib/crewai-tools/src/crewai_tools/__init__.py, lib/crewai/src/crewai/__init__.py, lib/devtools/src/crewai_devtools/__init__.py
Package versions are updated from 1.15.14 to 1.15.15.
Dependency pin updates
lib/cli/pyproject.toml, lib/crewai-tools/pyproject.toml, lib/crewai/pyproject.toml
Dependency pins are updated to 1.15.15.

Sequence Diagram(s)

sequenceDiagram
  participant Flow
  participant FlowEventListener
  participant Telemetry
  Flow->>FlowEventListener: Emit flow start or resume
  FlowEventListener->>Telemetry: Record Flow Execution
  Flow->>FlowEventListener: Emit pause, input, or failure
  FlowEventListener->>Telemetry: Record lifecycle signal
  Flow->>FlowEventListener: Emit completion or failure
  FlowEventListener->>Telemetry: Record duration and outcome
Loading

Suggested reviewers: lucasgomide, lorenzejay

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 35.23% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the version bump, which is a real part of the changeset, but it does not mention the primary telemetry changes.
Description check ✅ Passed The description clearly explains the flow telemetry changes, privacy behavior, internal-flow tagging, documentation updates, and version bump.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/bump-version-1.15.15

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@joaomdmoura
joaomdmoura enabled auto-merge (squash) August 12, 2026 01:05

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 77c68bd. Configure here.

Comment thread lib/crewai/src/crewai/events/event_listener.py

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
lib/crewai/tests/telemetry/test_flow_telemetry.py (1)

409-439: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Share the StubLLM definition between the two Crew-based tests.

test_crewais_own_agent_executor_is_tagged_internal and test_infrastructure_flows_do_not_pollute_outcome_signals declare identical StubLLM classes and build the same agent, task, and crew. Extract a module-level StubLLM and a small _run_crew() helper.

♻️ Proposed refactor
# Module level, near the fixtures:
def _run_crew() -> None:
    """Run a minimal Crew so the internal AgentExecutor flow executes."""
    from crewai import Agent, Crew, Task
    from crewai.llms.base_llm import BaseLLM

    class StubLLM(BaseLLM):
        def __init__(self) -> None:
            super().__init__(model="stub-model")

        def call(self, messages, **kwargs) -> str:
            return "Final Answer: done"

        def supports_function_calling(self) -> bool:
            return False

        def supports_stop_words(self) -> bool:
            return False

        def get_context_window_size(self) -> int:
            return 8192

    agent = Agent(role="R", goal="G", backstory="B", llm=StubLLM())
    task = Task(description="Do it", expected_output="A result", agent=agent)
    Crew(agents=[agent], tasks=[task]).kickoff()

Both tests then call _run_crew() in place of the duplicated block.

Also applies to: 552-590

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/crewai/tests/telemetry/test_flow_telemetry.py` around lines 409 - 439,
Extract the duplicated StubLLM definition and Crew setup from
test_crewais_own_agent_executor_is_tagged_internal and
test_infrastructure_flows_do_not_pollute_outcome_signals into a module-level
StubLLM and a _run_crew() helper. Keep the helper’s behavior identical, then
replace both inline setup blocks with _run_crew() calls.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/edge/en/telemetry.mdx`:
- Line 64: Update the Flow Lifecycle Signals privacy statement to acknowledge
that flow_name is retained and users must keep flow names free of personal
information, while preserving that method names and error messages are not
recorded. Apply equivalent wording in docs/edge/en/telemetry.mdx lines 64-64,
docs/edge/ar/telemetry.mdx lines 64-64, docs/edge/ko/telemetry.mdx lines 62-62,
and docs/edge/pt-BR/telemetry.mdx lines 64-64.

In `@lib/crewai/tests/telemetry/test_flow_telemetry.py`:
- Around line 220-234: Update StubProvider.request_input in
test_requesting_input_reports_both_sides to construct InputResponse with
text="typed answer" instead of the unsupported value argument, preserving the
existing telemetry assertions.

---

Nitpick comments:
In `@lib/crewai/tests/telemetry/test_flow_telemetry.py`:
- Around line 409-439: Extract the duplicated StubLLM definition and Crew setup
from test_crewais_own_agent_executor_is_tagged_internal and
test_infrastructure_flows_do_not_pollute_outcome_signals into a module-level
StubLLM and a _run_crew() helper. Keep the helper’s behavior identical, then
replace both inline setup blocks with _run_crew() calls.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 017a8a1f-9e5d-4ec6-a8aa-183c6aff3948

📥 Commits

Reviewing files that changed from the base of the PR and between 7642e61 and 77c68bd.

📒 Files selected for processing (22)
  • docs/edge/ar/telemetry.mdx
  • docs/edge/en/telemetry.mdx
  • docs/edge/ko/telemetry.mdx
  • docs/edge/pt-BR/telemetry.mdx
  • lib/cli/pyproject.toml
  • lib/cli/src/crewai_cli/__init__.py
  • lib/crewai-core/src/crewai_core/__init__.py
  • lib/crewai-files/src/crewai_files/__init__.py
  • lib/crewai-tools/pyproject.toml
  • lib/crewai-tools/src/crewai_tools/__init__.py
  • lib/crewai/pyproject.toml
  • lib/crewai/src/crewai/__init__.py
  • lib/crewai/src/crewai/events/event_listener.py
  • lib/crewai/src/crewai/experimental/agent_executor.py
  • lib/crewai/src/crewai/flow/runtime/__init__.py
  • lib/crewai/src/crewai/memory/encoding_flow.py
  • lib/crewai/src/crewai/memory/recall_flow.py
  • lib/crewai/src/crewai/telemetry/telemetry.py
  • lib/crewai/tests/telemetry/test_flow_telemetry.py
  • lib/crewai/tests/telemetry/test_telemetry.py
  • lib/crewai/tests/utilities/test_events.py
  • lib/devtools/src/crewai_devtools/__init__.py

Comment thread docs/edge/en/telemetry.mdx
Comment thread lib/crewai/tests/telemetry/test_flow_telemetry.py

@alex-clawd alex-clawd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@joaomdmoura
joaomdmoura merged commit 7d2437c into main Aug 12, 2026
62 checks passed
@joaomdmoura
joaomdmoura deleted the feat/bump-version-1.15.15 branch August 12, 2026 01:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants