Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Use Parseable's dashboard import flow with either a downloaded JSON file or the
| --- | --- | --- | ---: | ---: | ---: | --- | --- |
| Agent Cost Analysis | Analyze GenAI agent spend, token consumption, and cost distribution across models, services, tools, and sessions. | SQL | 30 | 6 | 1 | [JSON](agent-cost-analysis/agent-cost-analysis-sql.json) | [Raw](https://raw.githubusercontent.com/parseablehq/dashboards/main/agent-cost-analysis/agent-cost-analysis-sql.json) |
| Agent Observability | Track GenAI agent invocations, outcomes, latency, tools, tokens, cost, and request-level behavior. | SQL | 77 | 9 | 1 | [JSON](agent-observability/agent-observability-sql.json) | [Raw](https://raw.githubusercontent.com/parseablehq/dashboards/main/agent-observability/agent-observability-sql.json) |
| CrewAI Observability | Monitor CrewAI crews, flows, agents, tasks, LLM calls, token usage, tools, latency, and errors from OpenInference traces. | SQL | 38 | 6 | 1 | [JSON](crewai-observability/crewai-observability-sql.json) | [Raw](https://raw.githubusercontent.com/parseablehq/dashboards/main/crewai-observability/crewai-observability-sql.json) |
| LiteLLM Proxy Observability | Monitor LiteLLM proxy traffic, reliability, latency, token usage, spend, model behavior, deployment health, and request traces. | Mixed | 42 | 6 | 2 | [JSON](litellm-proxy-observability/litellm-proxy-observability-mixed.json) | [Raw](https://raw.githubusercontent.com/parseablehq/dashboards/main/litellm-proxy-observability/litellm-proxy-observability-mixed.json) |
| LiteLLM SDK Observability | Monitor LiteLLM SDK calls, reliability, latency, TTFT, token usage, spend, models, providers, correlated logs, traces, and metric health. | SQL | 40 | 8 | 8 | [JSON](litellm-sdk-observability/litellm-sdk-observability-sql.json) | [Raw](https://raw.githubusercontent.com/parseablehq/dashboards/main/litellm-sdk-observability/litellm-sdk-observability-sql.json) |
| Modal Observability | Monitor Modal application logs, traces, function and container capacity, CPU, memory, GPU, input events, latency, errors, and correlated telemetry. | Mixed | 40 | 8 | 3 | [JSON](modal-observability/modal-observability-mixed.json) | [Raw](https://raw.githubusercontent.com/parseablehq/dashboards/main/modal-observability/modal-observability-mixed.json) |
Expand Down
45 changes: 45 additions & 0 deletions crewai-observability/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# CrewAI Observability

Monitor CrewAI crews, flows, agents, tasks, LLM calls, token usage, tools, latency, and errors from OpenInference traces. It uses SQL panels to analyze OpenTelemetry spans directly from the selected Parseable trace dataset.

## What the dashboard shows

- **Overview:** total traces, crew runs, crew success rate, error spans, average and P95 crew duration, LLM calls, and tool calls.
- **Runs & Performance:** scenario throughput, workload mix, latency by scenario, and recent tagged runs.
- **Agents & Tasks:** agent execution counts, outcomes, error rate, P95 duration, and per-agent performance.
- **LLM & Tokens:** input, output, and total token consumption; model usage; LLM latency; and LLM outcomes.
- **Tools:** tool error rate, latency, failures, usage by name, performance, and call volume over time.
- **Errors & Trace Detail:** span error rate, failed traces, errors by span kind, recent errors, and trace-level summaries.

## Data used

| Signal | Default dataset | Query language | Purpose |
| ------ | --------------- | -------------- | -------------------------------------------------------------------------------------------------- |
| Traces | `crewai-traces` | SQL | OpenInference spans for CrewAI flows, crews, agents, tools, LLM calls, latency, tokens, and errors |

The template expects traces produced by `openinference-instrumentation-crewai` and exported to Parseable through OTLP/HTTP. Important fields include:

- `openinference.span.kind`
- `crew_id`, `task_id`, `task_name`, and `flow_id`
- `llm.model_name` and `llm.token_count.*`
- `tool.name`
- `span_trace_id`, `span_parent_span_id`, `span_duration_ns`, and `span_status_code`

Scenario-specific panels use the optional `load_test.*` attributes from the accompanying fixture workload. Core CrewAI, agent, LLM, tool, token, latency, and error panels work without those attributes.

## Filters

1 dashboard variable lets users select the telemetry source:

- Dataset selector: Trace Dataset

## Dashboard contents

- **38 tiles** across **6 collapsible sections**
- SQL panels over OpenInference and OpenTelemetry trace fields
- Dashboard preview images in [`assets/`](assets/)
- Importable template: [`crewai-observability-sql.json`](crewai-observability-sql.json)

## Import

Download the JSON file and use Parseable's dashboard import flow. During import or after creation, map **Trace Dataset** to the dataset receiving CrewAI OpenInference spans. The default is `crewai-traces`.
Binary file added crewai-observability/assets/AgentsAndTasks.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added crewai-observability/assets/AllSections.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added crewai-observability/assets/LLMAndTokens.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added crewai-observability/assets/Overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added crewai-observability/assets/Tools.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading