feat(evals): C# call-retrieval eval measuring the Roslyn hybrid frontend win

[vitali87]

What

Adds evals/csharp_retrieval.py — the file-level call-localization eval for C#, completing the multi-language retrieval matrix (Go/Rust/Java/TS/JS/PHP/Lua/C/C++/Scala already exist; the C# call oracle half, run_csharp_call_oracle, was already in place but had no consumer).

For each first-party C# symbol, the eval grades which files call it: cgr's CALLS and INSTANTIATES edges reduced to (caller_file, callee_simple_name) against Roslyn invocation and object-creation sites over the same first-party name universe. INSTANTIATES counts because new T() on a type with no explicit constructor has no ctor node to CALL (found the hard way: Polly's ResiliencePipelineBuilder has 65 creation sites and no explicit ctor — RED test in commit history).

Why

This is the eval that finally measures the #738 Roslyn hybrid frontend's win, which the L1 structure eval could not show (structure was already ~0.999 under tree-sitter).

Results on Polly

Mode	Precision	Recall	F1
tree-sitter (default)	1.0000	0.7660	0.8675
CSHARP_FRONTEND=hybrid	1.0000	0.8912	0.9425

The hybrid frontend's location-keyed call facts recover 472 additional call edges (overloads, extension methods, cross-project dispatch) at zero precision cost. Residual misses are dominated by BCL simple-name collisions inherent to the name-restricted universe (list.Add graded against a first-party Add — the same documented tradeoff as every other language retrieval) plus call sites in projects outside Polly.slnx (samples/, bench/), where the frontend deliberately degrades to heuristics.

Validation

RED → GREEN in commit history (module-missing RED, then the implicit-ctor RED). Pinned by codebase_rag/tests/test_csharp_retrieval_eval.py (oracle fixture, cgr-matches-oracle fixture, implicit-ctor creation, PRF scoring). README documents the eval and the measured win. Full suite: 5236 passed, 10 skipped.

[gemini-code-assist]

Code Review

This pull request introduces multi-language retrieval evaluation for C#, adding a new evaluation script, tests, constants, and documentation. The review feedback correctly identifies a critical bug where the constant CSHARP_ORACLE_MISSING is referenced in evals/csharp_retrieval.py but is not defined in evals/logs.py, which would result in an AttributeError if the dotnet toolchain is missing.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[greptile-apps]

Greptile Summary

This PR adds a C# call-retrieval eval that compares cgr call edges with the Roslyn oracle. The main changes are:

• New evals.csharp_retrieval CLI for scoring C# CALLS and INSTANTIATES edges.
• Roslyn oracle callable universe expanded to include class names for ctorless object creation sites.
• Tests covering oracle extraction, cgr parity, implicit constructor creation, target-typed new() symmetry, and PRF scoring.
• README, constants, logs, and lockfile metadata updated for the new eval.

Confidence Score: 5/5

Safe to merge with minimal risk.

The changes follow the existing multi-language retrieval eval pattern and include tests for the C#-specific edge cases covered by the PR.

No files require special attention.

T-Rex Logs

What T-Rex did

• A proof was produced for the posted P1 finding and linked to the reviewer comment detailing the finding.
• The fallback proof logs document the initial failure path that occurred before the environment was reconciled.
• The setup proof log shows the dependency sync completed successfully with exit code 0 and without installing pymgclient.
• The main proof log shows pytest collected 6 tests, skipped 5, and errored 1 with AttributeError: module 'codebase_rag' has no attribute 'cli' (exit code 1).

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
evals/csharp_retrieval.py	Introduces the C# call-retrieval CLI and scoring path, including CALLS and INSTANTIATES edge reduction against the Roslyn oracle.
evals/oracles/csharp_oracle.py	Expands the declared callable universe to include class names so object creation of ctorless C# types is graded.
codebase_rag/tests/test_csharp_retrieval_eval.py	Adds C# retrieval eval coverage for oracle extraction, cgr parity, implicit constructor instantiation, target-typed new symmetry, and PRF scoring.
evals/README.md	Documents the new C# multi-language retrieval eval, usage, dotnet requirement, and reported Polly measurements.
evals/constants.py	Adds C# retrieval filenames, diff prefix, label/title, and edge formatting constants consistent with existing retrieval evals.
evals/logs.py	Adds log message templates for the C# retrieval oracle and cgr graph phases.
uv.lock	Updates the editable package version metadata in the lockfile.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant CLI as evals.csharp_retrieval
participant Oracle as Roslyn call oracle
participant CGR as cgr graph capture
participant Score as PRF scorer
CLI->>Oracle: oracle_csharp_call_edges(target)
Oracle-->>CLI: first-party call edges + declared names
CLI->>CGR: cgr_csharp_call_edges(target, project, declared)
CGR-->>CLI: CALLS/INSTANTIATES file-name edges
CLI->>Score: score_csharp_retrieval(cgr, oracle)
Score-->>CLI: rows + missing/extra diff
CLI->>CLI: write outputs and render report

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant CLI as evals.csharp_retrieval
participant Oracle as Roslyn call oracle
participant CGR as cgr graph capture
participant Score as PRF scorer
CLI->>Oracle: oracle_csharp_call_edges(target)
Oracle-->>CLI: first-party call edges + declared names
CLI->>CGR: cgr_csharp_call_edges(target, project, declared)
CGR-->>CLI: CALLS/INSTANTIATES file-name edges
CLI->>Score: score_csharp_retrieval(cgr, oracle)
Score-->>CLI: rows + missing/extra diff
CLI->>CLI: write outputs and render report

Loading

Comments Outside Diff (1)

1. General comment

   Focused C retrieval eval test file fails during autouse fixture setup

   • Bug
     • The PR's pinned validation file codebase_rag/tests/test_csharp_retrieval_eval.py cannot complete successfully. After a successful frozen test dependency sync excluding pymgclient, running .venv/bin/python -m pytest codebase_rag/tests/test_csharp_retrieval_eval.py -q -vv collects 6 tests, skips 5, and errors on test_score_csharp_retrieval_prf before the test body runs.
   • Cause
     • The autouse fixture _disable_stack_autostart in codebase_rag/tests/conftest.py calls patch("codebase_rag.cli._maybe_start_stack"), but importing/resolving that dotted path fails because the codebase_rag package object has no cli attribute in this environment/package layout.
   • Fix
     • Update the fixture patch target to the actual module path that defines _maybe_start_stack, or ensure codebase_rag.cli is importable/attached when tests run. Then rerun the focused C# retrieval eval file to verify the oracle, CGR-match, implicit-constructor, and PRF assertions execute.

   _{Ran code and verified through T-Rex}

_{Reviews (4): Last reviewed commit: "test(evals): pin symmetric omission of C..." | Re-trigger Greptile}

[vitali87]

@greptile review

[vitali87]

@greptile review

[vitali87]

@greptile review

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud