Analytics: Analytics Dashboard APIs#888
Merged
Merged
Conversation
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR adds monthly analytics aggregation for the organization, exposing two new API endpoints that compute request counts, costs, and evaluation metrics across LLM calls, chains, and evaluation runs, grouped by month, modality, and provider with optional filtering and time-window selection. ChangesMonthly Analytics Feature
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…jectTech4DevAI/kaapi-backend into feat/analytics-dashboard-api
Ayush8923
commented
May 26, 2026
Comment on lines
+127
to
+151
| stmt = ( | ||
| select( | ||
| month_col, | ||
| modality_col, | ||
| provider_col, | ||
| LlmCall.model, | ||
| count_col, | ||
| input_tokens_col, | ||
| output_tokens_col, | ||
| ) | ||
| .where( | ||
| LlmCall.deleted_at.is_(None), | ||
| LlmCall.organization_id == organization_id, | ||
| ) | ||
| .group_by(month_col, modality_col, provider_col, LlmCall.model) | ||
| ) | ||
| if from_month is not None: | ||
| stmt = stmt.where(LlmCall.inserted_at >= from_month) | ||
| if end_date is not None: | ||
| stmt = stmt.where(LlmCall.inserted_at < end_date) | ||
| if project_id is not None: | ||
| stmt = stmt.where(LlmCall.project_id == project_id) | ||
| if provider_filter is not None: | ||
| stmt = stmt.where(LlmCall.provider == provider_filter) | ||
|
|
Collaborator
Author
There was a problem hiding this comment.
Please review these types of queries thoroughly, as I am fetching the data using queries like these.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/app/api/docs/analytics/monthly_chart.md`:
- Around line 111-113: The fenced example HTTP request blocks in
monthly_chart.md lack language identifiers; update each triple-backtick fence
that wraps the GET examples (e.g., the blocks showing "GET
/api/analytics/monthly/chart?metric=...") to use ```http so the markdown linter
recognizes them as HTTP examples—apply this change to the blocks at the shown
ranges (including the examples for metric=requests, metric=eval_cost,
metric=cost, and the other occurrences at 117-119, 123-125, 129-131).
In `@backend/app/api/docs/analytics/monthly.md`:
- Around line 131-133: The fenced code blocks showing HTTP request examples
(e.g., the lines starting "GET
/api/analytics/monthly?metric=cost&from_month=2026-01-01&to_month=2026-05-01",
"GET /api/analytics/monthly?metric=requests&modality=T-FS-T&provider=openai",
and "GET
/api/analytics/monthly?metric=eval_cost&modality=STT&from_month=2026-01-01") are
missing a language tag; update each triple-backtick fence in
backend/app/api/docs/analytics/monthly.md that wraps these GET examples to
include the language identifier http (i.e., change ``` to ```http) so markdown
linting passes.
In `@backend/app/api/routes/analytics.py`:
- Around line 160-167: After snapping dates with _snap_to_first_of_month, add an
explicit validation that if both from_month and to_month are present (or
effective_from_month computed) and the from_month > to_month (i.e., inverted
window) you return a 400 error instead of proceeding to
aggregate_monthly_metrics; locate the logic around from_month, to_month,
effective_from_month in the analytics route handler and raise/return a
BadRequest (HTTP 400) with a clear message like "from_month must be <= to_month"
before calling aggregate_monthly_metrics.
- Around line 103-145: Add explicit return type annotations to both endpoint
handlers: update get_monthly_analytics and the other analytics endpoint defined
later (the function around the 200-252 block) to declare return types like ->
APIResponse[Any] (or a more specific APIResponse[YourResponseModel] if a
response model exists), and ensure APIResponse and Any (or the specific model)
are imported; this satisfies the typing guideline by annotating the function
return values with APIResponse[...] for both endpoints.
In `@backend/app/services/analytics/aggregation.py`:
- Around line 164-170: The aggregation loop is performing a DB lookup per
grouped row via estimate_model_cost; instead, preload the model/pricing metadata
once and compute costs in-memory. Before the loop that calls
estimate_model_cost, query and build a dict cache keyed by (row.provider,
row.model) containing the required model_config/pricing info, then change the
loop to fetch from that cache and call a new helper or an updated
estimate_model_cost that accepts the preloaded config (refer to
estimate_model_cost and the aggregation loop variable names) so no DB access
happens inside the loop.
- Around line 269-273: The provider_expr currently uses mc_lookup.c.provider
first which can misattribute providers; change the coalesce order so
cv_provider_normalized (the normalized provider from ConfigVersion) is the first
choice, then fall back to sa.cast(mc_lookup.c.provider, sa.String), then
"unknown"; ensure you still nullify empty strings via
sa.func.nullif(cv_provider_normalized, "") and keep use of sa.func.coalesce,
sa.cast, mc_lookup.c.provider, cv_provider_normalized, and provider_expr when
making the replacement.
- Around line 163-167: The membership check against KNOWN_PROVIDERS happens
before normalizing provider names so rows like row.provider == "openai-native"
skip cost lookup; normalize the provider string first (e.g., map or strip
"-native" to "openai") before the if-check that uses KNOWN_PROVIDERS and pass
the normalized provider into estimate_model_cost (call sites: the if using
input_tokens/output_tokens and the estimate_model_cost invocation). Update any
local variable (e.g., create normalized_provider) and use that variable in both
the membership check and the provider argument to estimate_model_cost.
- Line 65: The function _llm_modality_case currently has an untyped parameter
`call`; add a proper type annotation that accepts either the LlmCall ORM class
or an aliased variant (e.g., typing.Union[LlmCall, sa.orm.util.AliasedClass] or
the project's preferred alias type) so callers using LlmCall and
aliased(LlmCall) type-check correctly; update the signature to annotate `call`
accordingly and run type checks to ensure compatibility with code that passes
aliased(LlmCall).
In `@backend/app/tests/api/routes/test_analytics.py`:
- Around line 136-140: Annotate the _ok helper with explicit type hints: add a
parameter annotation (e.g., response: Response) and a return annotation (e.g.,
-> Dict[str, Any] or -> Any) and import the referenced types; specifically
update the _ok function in test_analytics.py (function name _ok) to accept
response: Response and return a typed dict (or Any) and add the needed imports
(from requests import Response or from typing import Any, Dict) at the top of
the file.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 65fd5fe5-1f4e-4d94-a2ca-47bdb6064d84
📒 Files selected for processing (10)
backend/app/api/docs/analytics/monthly.mdbackend/app/api/docs/analytics/monthly_chart.mdbackend/app/api/main.pybackend/app/api/routes/analytics.pybackend/app/crud/model_config.pybackend/app/models/__init__.pybackend/app/models/analytics.pybackend/app/services/analytics/__init__.pybackend/app/services/analytics/aggregation.pybackend/app/tests/api/routes/test_analytics.py
vprashrex
approved these changes
Jun 1, 2026
AkhileshNegi
approved these changes
Jun 2, 2026
…kend into feat/analytics-dashboard-api
OpenAPI changes 🟢 2 non-breaking changesTip Safe to merge from an API-contract perspective. Full changelog ·
|
| Method | Path | Change | |
|---|---|---|---|
| 🟢 | GET |
/api/v1/analytics/monthly |
endpoint added |
| 🟢 | GET |
/api/v1/analytics/monthly/chart |
endpoint added |
main ↔ 257af80a · generated by oasdiff
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue: #851
Summary
GET /analytics/monthlyandGET /analytics/monthly/chart) that compute usage metrics live fromllm_call,llm_chain, andevaluation_run.from_month/to_month, modality, provider, andproject_id. Auto-scopes to the caller's current project (or whole org when none is selected) and requiresREQUIRE_ORGANIZATIONpermission.Checklist
Before submitting a pull request, please ensure that you mark these task.
fastapi run --reload app/main.pyordocker compose upin the repository root and test.