fix: scope evaluation results by user identity

[yyiilluu]

Summary

• Add user_id to agent success evaluation result identity and retrieval paths.
• Prevent cross-user collisions when users share session or request identifiers.
• Keep evaluation-only interaction models available to agent-success prechecks.

Changes

• Thread user identity through agent success evaluation result schemas, storage lookups, and overview grouping.
• Update SQLite storage behavior and tests for user-scoped result identity.
• Add regression coverage for evaluation-only session prechecks.
• Refresh server docs for the result identity contract.

Test Plan

• uv run ruff check on staged Python files
• uv run ruff format on staged Python files
• uv run pyright on staged Python files
• uv run pytest tests/server/services/agent_success_evaluation/test_agent_success_evaluation_services.py -q --no-cov

Summary by CodeRabbit

• New Features

  • Evaluation results are now scoped per user, allowing sessions to be safely reused across different users without ID collisions.
  • The system now uniquely identifies evaluation data using user and session pairs instead of session IDs alone, improving multi-user support and data isolation.

• Database

  • Added schema migration to support user-scoped evaluation result storage.

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR adds user_id as a scoping dimension to agent success evaluation identity, changing the key from session_id to (user_id, session_id) across data models, storage interfaces, SQLite schema (with migration), evaluation service pipeline, group runner, regen jobs, the grade_on_demand API endpoint, and the evaluation overview aggregation service.

Changes

user_id scoping for evaluation identity

Layer / File(s)	Summary
Data model contracts reflexio/models/api_schema/domain/entities.py, reflexio/models/api_schema/internal_schema.py, reflexio/models/api_schema/ui/entities.py, reflexio/models/api_schema/ui/converters.py, reflexio/server/services/agent_success_evaluation/agent_success_evaluation_utils.py, reflexio/server/services/agent_success_evaluation/agent_success_evaluation_service.py, reflexio/server/README.md	Adds user_id field to AgentSuccessEvaluationResult, SessionCitation, EvaluationResultView, AgentSuccessGenerationServiceConfig, and AgentSuccessEvaluationRequest; wires it through the UI converter; updates README key documentation.
Storage base interfaces + bulk user/session fetch reflexio/server/services/storage/storage_base/_playbook.py, reflexio/server/services/storage/storage_base/_requests.py, reflexio/server/services/storage/storage_base/_extras.py	Updates abstract get_agent_success_evaluation_result_ids and delete_agent_success_evaluation_results_for_session signatures to require user_id; adds get_first_requests_by_user_session_pairs method; populates user_id="" in base SessionCitation construction.
SQLite storage: schema, migration, and DML reflexio/server/services/storage/sqlite_storage/_base.py, reflexio/server/services/storage/sqlite_storage/_playbook.py, reflexio/server/services/storage/sqlite_storage/_extras.py, reflexio/server/services/storage/sqlite_storage/_requests.py	Adds user_id column to agent_success_evaluation_result DDL; introduces _migrate_eval_result_user_id startup migration that adds the column and rebuilds the compound identity index; updates INSERT, SELECT, and DELETE SQL to filter/write user_id; adds optimized get_first_requests_by_user_session_pairs using ROW_NUMBER() window function.
Evaluation service pipeline reflexio/server/services/agent_success_evaluation/agent_success_evaluation_service.py, reflexio/server/services/agent_success_evaluation/agent_success_evaluator.py	_load_generation_service_config populates user_id from the request; _build_evaluation_result passes user_id into the result; new _collect_scoped_interactions_for_precheck scopes session interaction models using source filter helpers.
Group runner, regen jobs, and API endpoint reflexio/server/services/agent_success_evaluation/group_evaluation_runner.py, reflexio/server/services/agent_success_evaluation/regen_jobs.py, reflexio/server/api.py	Group runner passes user_id into prior-result lookups and evaluation requests; regen jobs bulk-fetch by (user_id, session_id) pairs; grade_on_demand scopes _find_fresh_result_id and previous_result_ids by user_id.
Evaluation overview service and rule attribution reflexio/server/services/evaluation_overview/service.py, reflexio/server/services/evaluation_overview/rule_attribution.py	Introduces ResultKey/SessionIdentity type aliases; rewrites run, _build_attribution, _load_citations, _build_source_set_comparison, and _build_first_request_sources to key by (user_id, session_id) tuples; updates compute_net_sessions to iterate SessionIdentity keys.
Tests tests/server/api_endpoints/test_grade_on_demand_integration.py, tests/server/services/agent_success_evaluation/test_*.py, tests/server/services/evaluation_overview/test_*.py, tests/server/services/storage/test_session_window_contract.py	Updates test helpers, mocks, and fixture factories to supply and assert on user_id; adds test_eval_result_identity_is_user_scoped, test_precheck_uses_provided_session_models_for_evaluation_only, and related contract tests.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant grade_on_demand
  participant storage
  participant AgentSuccessEvaluationService
  participant AgentSuccessEvaluator

  Client->>grade_on_demand: POST /api/evaluations/grade_on_demand {user_id, session_id, ...}
  grade_on_demand->>storage: get_agent_success_evaluation_result_ids(user_id, session_id, ...)
  storage-->>grade_on_demand: previous_result_ids
  grade_on_demand->>AgentSuccessEvaluationService: run(request{user_id, session_id, ...})
  AgentSuccessEvaluationService->>AgentSuccessEvaluationService: _load_generation_service_config → AgentSuccessGenerationServiceConfig{user_id}
  AgentSuccessEvaluationService->>AgentSuccessEvaluator: evaluate(service_config{user_id})
  AgentSuccessEvaluator->>AgentSuccessEvaluator: _build_evaluation_result → AgentSuccessEvaluationResult{user_id}
  AgentSuccessEvaluator->>storage: save_agent_success_evaluation_results([result{user_id}])
  AgentSuccessEvaluator-->>grade_on_demand: result saved
  grade_on_demand->>storage: get_agent_success_evaluation_result_ids(user_id, session_id, ...) [find fresh]
  storage-->>grade_on_demand: fresh_result_id
  grade_on_demand-->>Client: evaluation result

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• ReflexioAI/reflexio#101: Refactors the agent-success generation service extractor/evaluator configuration flow in the same agent_success_evaluation_service.py file that this PR extends with user_id and _collect_scoped_interactions_for_precheck.
• ReflexioAI/reflexio#168: Added bulk "first request" storage read and evaluation overview aggregation work that this PR directly extends by keying on (user_id, session_id) pairs instead of session_id alone.

Poem

🐇 Hopping through sessions, one user at a time,
No more mixed-up results—now each row's aligned!
(user_id, session_id) makes identity clear,
The bunny's migration runs fresh every year.
With tuple keys scoped, no collision in sight—
Multi-tenant eval grading, done just right! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 69.12% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The pull request title 'fix: scope evaluation results by user identity' accurately reflects the main change: adding user identity scoping to prevent cross-user collisions in evaluation results.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/eval-results-user-identity

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}