LLM Evaluation Sequence in Software Engineering Tasks

LLM Evaluation Pipeline for Software Engineering Tasks

To conduct this study, we used the SWE-bench benchmark as the evaluation basis, rather than reproducing the benchmarking process from scratch. The researchers responsible for the project maintain a public SWE-bench leaderboard, updated regularly as new models are released. In this study, we use that leaderboard as a reference: SWE-bench Leaderboard.

---

Phase 1: Baseline Collection and Artifact Materialization

Step 01 — Repository Selection

For the benchmark evaluation, we selected the RAG (Retrieval-Augmented Generation) GPT-4 configuration from April 2, 2024 (20240402), as described in the SWE-bench experiments repository:

🔗 shawnlewis/swebench-experiments

From this repository and its associated forks, we retrieved the raw prediction data generated by the LLMs, as well as the artifacts produced during the benchmark evaluation. The shawnlewis/swebench-experiments repository is a fork of the official SWE-bench/experiments repository and provides access to the evaluation data and results associated with the SWE-bench experiments.

Within the evaluation directory, the repository is organized into subdirectories corresponding to the different benchmark variants, including lite/, test/, verified/, and multimodal/. In these directories, evaluation configurations are organized into folders named by date and by model (or evaluation configuration). For this study, we selected the 20240402_rag_gpt4 configuration within the verified/ directory.

The 20240402_rag_gpt4 directory contains the files and data needed to retrieve the model predictions and evaluate the benchmark results:

• all_preds.jsonl — contains the predictions generated for the SWE-bench instances;
• metadata.yaml — provides the metadata associated with the evaluation.

The accompanying README file identifies this configuration as the SWE-bench RAG baseline, in which BM25 is used to retrieve relevant code snippets before they are passed to the language model to generate a patch.

The all_preds.jsonl file consists of JSON objects, with one prediction object per line. The source code generated by the LLM is stored in the model_patch field, which contains the exact patch (or diff string) proposed by the model as a solution to the corresponding GitHub issue.

In the SWE-bench setting, the model receives a software issue along with the corresponding codebase and must generate a patch that resolves the problem. Therefore, the model_patch field provides the model-generated code changes in diff format, not the full source code of the modified files.

Step 02 — Patch Extraction

⬆️ Back to top

This step extracts the code changes generated by the LLM during the SWE-bench Verified benchmark, for subsequent static analysis with SonarQube.

The all_preds.jsonl file was renamed to all_preds-Verified-rag-gpt4.jsonl. It does not contain complete, directly executable source files; instead, it contains the patches generated by the model in the model_patch field, represented in Unified Diff format. A patch describes the changes proposed by the model, including the affected files, the added and removed lines, and the corresponding line coordinates. Therefore, the contents of the model_patch field cannot be submitted directly to SonarQube as a complete source file for static analysis.

To compute software engineering metrics — such as McCabe's Cyclomatic Complexity, Halstead Metrics, and Cognitive Complexity — the source code must first be reconstructed into a syntactically valid state. Static analysis tools rely on parsing language-recognized source files and, depending on the tool, build internal representations — such as an Abstract Syntax Tree (AST) — to identify program structures and compute the corresponding metrics. Raw diff content, containing markers such as +, -, and @@, is not a valid source file and therefore cannot be analyzed as the resulting program.

Consequently, the preprocessing step requires reconstructing the repository state after applying the LLM-generated changes. The mandatory processing flow consists of extracting the patches, applying them to the corresponding SWE-bench repositories, and generating the modified source files, which are then submitted to the static analysis tools.

The Python script 1extracao01.py performs the first step of this flow: it processes the patches contained in all_preds-Verified-rag-gpt4.jsonl and extracts the applicable patches to the /patches_extraidos directory. These extracted patches are then used in the next step to reconstruct the modified repository state.

📄 Execution script: 1extracao01.py
📄 Generated artifact (output): mapeamento_commits.json

Step 03 — Obtaining the Base Commit

⬆️ Back to top

The Verified dataset evaluated in this study, corresponding to the predictions in all_preds-Verified-rag-gpt4.jsonl, is associated with the SWE-bench/SWE-bench_Verified dataset, hosted on Hugging Face.

To retrieve the base_commit associated with each SWE-bench instance, the Python datasets library was used to access the corresponding records. The 2mapear_base_commit.py script was developed to map each instance_id to its respective base_commit. The resulting mapping is used to reconstruct the repository state that existed before the issue was resolved.

The repository reconstruction and analysis workflow consists of the following steps:

1. Patch Extraction — extract each model_patch from the JSONL file and save it as an individual .patch file.
2. Base Repository Checkout — use the instance_id (e.g., django_django-16116) to identify the corresponding repository and check out the exact base_commit, representing the repository state before the proposed fix is applied.
3. Patch Application — apply the AI-generated .patch file to the corresponding base repository, using a mechanism such as git apply. This produces the modified repository state containing the changes proposed by the LLM.
4. Analysis Isolation — configure the sonar-project.properties file using the sonar.inclusions parameter to restrict the analysis to the files modified by the patch, preventing the results from being dominated by pre-existing code in the repository.

Since the directory containing the extracted patches already exists — with each patch named following the <instance_id>.patch convention (e.g., django_django-16116.patch) — the subsequent automation can be run directly from that directory.

The 2mapear_base_commit.py script automates this process: it reads the extracted patches from the /patches_extraidos directory, derives the instance_id from each filename, retrieves the corresponding instance information from the Hugging Face dataset, and generates the mapeamento_commits.json file. This file serves as the master index for the subsequent repository reconstruction and static analysis step.

📄 Execution script: 2mapear_base_commit.py
📄 Generated artifact (output): Creation of the folder/downloads patches = "./patches_extraidos"

---

Phase 2: Batch Static Processing

Step 04 — Orchestration and Static Analysis with SonarQube

⬆️ Back to top

To process the 500 instances and submit them to SonarQube, it was necessary to orchestrate the steps of repository cloning, base commit checkout, patch application, and SonarScanner invocation. Before running the automation script (DT02-Sonar.py), two mandatory technical prerequisites related to the SonarQube infrastructure were identified:

1. SonarScanner CLI — the sonar-scanner command-line tool was installed on the machine and added to the environment variables (PATH). The script depends on calling this executable directly to start the analysis of each instance.
2. Authentication token — by default, SonarQube does not allow anonymous submissions. Authentication was performed through the web interface (http://localhost:9000), under My Account > Security, where a User Token was generated. This token is injected dynamically by the script to authorize the automatic creation of projects and the upload of results. To run the procedure in another environment, a new token must be generated and the corresponding variable in the script must be updated.

The execution script reads the mapping file (mapeamento_commits.json), provisions the repositories locally — creating the /workspace_repos directory when necessary — and triggers the scan via SonarScanner.

In line with the methodology established in Step 03, the -Dsonar.inclusions parameter is set to restrict the analysis scope exclusively to the files modified by the patch, preventing the results from being dominated by pre-existing code. Additionally, -Dsonar.exclusions directives are applied to test files, ensuring that the Cyclomatic and Cognitive Complexity metrics are not artificially diluted or inflated by any modified test code. The goal of this combined configuration is to isolate the impact of the modifications strictly to the production code affected by the AI.

The analysis results are processed and stored by SonarQube in PostgreSQL. Once the run completes, the 500 projects are individually available in the local SonarQube dashboard, allowing the metrics of each instance to be queried and extracted independently.

📄 Execution script: DT02-Sonar.py
📄 Output artifact: folder "./workspace_repos"
📄 Output artifact: folder "./cache_repos_mirror"
📄 Output artifact: progresso_sonar.json

⏱️ Performance note: in the first version of the DT02-Sonar.py script, analyzing the 500 instances took approximately 125 hours of continuous processing. This time may vary depending on the hardware used and the network connection speed (internet access).

Step 05 — Exporting SonarQube Metrics

⬆️ Back to top

After the analyses of the 500 instances are complete, the DT03-Export.py script consolidates and exports the collected metrics.

The script uses the SonarQube Web API to query the results stored in the local instance. It first paginates through the projects to retrieve the project keys corresponding to the 500 analyzed instances. It then issues individual requests to the api/measures/component endpoint, retrieving the metrics associated with each project.

The extracted data is consolidated into tabular structures and exported in two formats:

• CSV — the primary file, intended for analysis and further data processing;
• JSON — a backup file that preserves the data returned by the API in a hierarchical structure, facilitating auditing and any reprocessing of the extraction.

The resulting files are stored on the G: drive, keeping a structured copy of the metrics produced by SonarQube and allowing the results of the 500 instances to be used later in the statistical analysis and correlation with the experiment's other data.

Generated files:

• metricas_completas_swebench.csv
• metricas_completas_swebench.json
• erros_extracao_completa.log

📄 Execution script: DT03-Export.py

---

Phase 3: LLM-as-a-Judge Execution

Step 06 — Prompt Preparation for LLM-as-a-Judge Evaluation

⬆️ Back to top

After the static analysis performed by SonarQube, the semantic evaluation of the LLM-generated patches begins. The concatenar04-claude.py script prepares the data and builds the prompts that will later be used by the LLM-as-a-Judge.

The script integrates, in a single flow, the original problem recorded in the SWE-bench Verified dataset (problem_statement) and the patch generated by the LLM (model_patch), through the following steps:

1. Dataset loading — the script loads the official princeton-nlp/SWE-bench_Verified dataset, available on Hugging Face, using the datasets library and the test split.
2. Instance indexing — the dataset records are organized into a dictionary indexed by the instance_id field. For each instance, the respective problem_statement is stored, allowing direct retrieval during processing.
3. Reading predictions — the all_preds-Verified-rag-gpt4.jsonl file is opened and processed line by line; from each record, the instance_id and the LLM-generated model_patch are extracted.
4. Problem–patch association — for each instance_id, the script retrieves the corresponding problem_statement from the dataset. The original problem and the generated patch are then inserted into the prompt template defined in the script itself.
5. Evaluation payload generation — for each valid instance, a record is created containing the instance_id, the system_prompt, and the user_prompt. These records are written to evaluation_payload.jsonl, which is the structured input for the subsequent LLM-as-a-Judge evaluation step.

The system_prompt establishes the criteria used in the evaluation, including semantic correctness, side effects, and solution completeness. The judge must also produce a decision (resolve_issue) and a confidence level (score_confianca) between 1 and 5.

The script also validates the match between the predictions and the dataset: instances whose instance_id is not found are skipped and counted in the final report, along with the total number of lines processed and the number of payloads actually generated. Thus, concatenar04-claude.py does not run the LLM evaluation itself — its role is to prepare and structure, reproducibly, the set of inputs that will be submitted to the evaluator model in the next step.

📄 Execution script: concatenar04-claude.py
📄 Output artifact: evaluation_payload.jsonl

Methodological transparency note: the system_prompt text used in the LLM-as-a-Judge evaluation was refined with the assistance of the Claude language model (Anthropic), used as a tool to support the wording and clarity of the evaluation criteria. The definition of the judgment criteria, the validation of the final prompt, and the interpretation of the results remained the responsibility of the researchers.

Step 07 — Semantic Evaluation with LLM-as-a-Judge

⬆️ Back to top

I. Persistence and Resume Structure

run3_evaluation.py was designed for incremental processing using JSONL (JSON Lines) files.

The evaluation_payload.jsonl file, produced in the previous step, is processed line by line. After each successful evaluation, the result is immediately serialized and written to evaluation_results.jsonl, followed by outfile.flush(). In case of failure, the instance is also recorded in the file as a structured error record.

At the start of execution, the script checks the results already persisted and identifies the instances completed successfully, avoiding their reprocessing. Instances associated with errors remain eligible for retry.

This strategy reduces lost work in long-running executions and allows the process to resume after interruptions.

II. Authentication and Model Configuration

Execution requires a Google Gemini API key, provided through the GEMINI_API_KEY environment variable. The script uses load_dotenv() to load the variables and explicitly checks for the credential, aborting execution if it is not available.

The key is not stored in the source code. Configuration examples:

# PowerShell
$env:GEMINI_API_KEY="your_key_here"

:: CMD
set GEMINI_API_KEY=your_key_here

The evaluation uses the gemini-2.5-flash model with temperature 0.0, aiming to reduce response variability across instances.

III. Structured Output Generation

The call_llm_judge() function sends the system_prompt and user_prompt prepared in the previous step to the Gemini API. The system_prompt defines the model's role as a quality judge and the evaluation criteria, while the user_prompt provides the instance-specific context.

The response_mime_type="application/json" setting requests a structured JSON response. The return value is validated and parsed with json.loads(). Empty or invalid responses are treated as failures and passed to the retry mechanism.

The retry mechanism allows up to five attempts, using exponential backoff between calls. In the current implementation, retries are triggered broadly on any exception and are not restricted to specific API error types.

IV. Evaluation Execution

The input and output paths are derived from the script's own location. For each instance in evaluation_payload.jsonl, run3_evaluation.py retrieves the prompts, calls the evaluator model, and associates the response with the respective instance_id.

After processing, the result is persisted to evaluation_results.jsonl. If all attempts fail, the instance is kept in the file with its instance_id and the error description, allowing later identification and reprocessing.

V. Nature of the Evaluation

This step performs the semantic evaluation of the patches using an LLM-as-a-Judge. Unlike the static analysis performed by SonarQube — which assesses structural properties of the code — the judge receives the original problem and the generated patch and verifies whether the change semantically satisfies the requirement.

The criteria defined in the system_prompt cover semantic correctness, side effects, and completeness, producing a verdict (resolve_issue) and a confidence level (score_confianca).

Thus, the evaluation_results.jsonl file is the output of this step, containing the LLM's structured judgments associated with the respective instance_id.

📄 Execution script: run3_evaluation.py
📄 Generated artifact (output): evaluation_results.jsonl

---

Phase 4: Multidimensional Statistical Analysis

Step 08 — Translating Semantic Analysis into Statistical Data

⬆️ Back to top

Semantic Analysis

After the semantic evaluation by the LLM-as-a-Judge, the results were persisted to evaluation_results.jsonl. Each record contains the raciocinio_analitico (analytical reasoning), the resolve_issue verdict, and the score_confianca confidence level.

The raciocinio_analitico is produced by analyzing the patch against the issue, considering three criteria:

• Semantic correctness — checks whether the patch implements the required logic and addresses the root cause of the problem.
• Side effects — checks for regressions, contract violations, invariant violations, and unexpected behavior.
• Completeness — checks the handling of the cases explicitly described in the issue and of critical edge cases.

Semantic Metric Extraction

A second processing step uses the raciocinio_analitico as input to extract three ordinal quality metrics:

Metric	Description
Fix Depth Index (M_profundidade)	Assesses whether the solution addresses the root cause or only the symptom.
Architectural Integrity Grade (M_integridade)	Assesses preservation of system integrity and the occurrence of regressions or side effects.
Defensive Coverage Index (M_defesa)	Assesses the completeness of the solution and its ability to handle edge cases.

Each metric uses a scale from 1 to 3. Classification follows strict inference rules: the evaluator must use exclusively the information present in the raciocinio_analitico. When there is insufficient evidence for a higher rating, the corresponding fallback value defined in the criterion applies.

The processing also produces a short justification (just_*) for each metric, enabling audit by sampling and verification that the judgment process adheres to the defined criteria.

Consistency Metric

In addition to the three ordinal metrics above, the script computes M_coerencia locally, without an additional API call. This metric assesses the internal consistency of the original evaluator's output, flagging any contradiction between the issue-resolution verdict (resolve_issue) and the confidence level assigned to it (score_confianca). It is used for auditing purposes and is not part of the quality metrics extracted from the raciocinio_analitico.

M_coerencia is binary (0 or 1), as defined by the calcular_m_coerencia(resolve_issue, score_confianca) function:

Value	Classification	Condition
0	Inconsistent	The issue is marked as resolved (resolve_issue is True) and the assigned confidence level is low (score_confianca <= 2).
1	Consistent	All other cases.

Extraction Prompt

The prompt used for extraction enforces strict inference and structured output, as shown below:

[START OF PROMPT]

You act as a software engineering evaluator. Based exclusively on the
`raciocinio_analitico` text, extract three ordinal quality metrics.

Inference rule: do not infer information that is not explicitly present
in the text. When a criterion is omitted, you must use the corresponding
fallback value.

M_profundidade — Fix Depth Index
* 1: symptom treatment, workaround, or failure to fix the root cause.
* 2: functional fix for the specific case, with no evidence of root cause treatment.
* 3: explicit evidence of root cause correction or treatment.

M_integridade — Architectural Integrity Grade
* 1: regressions, contract or invariant violations, or architectural flaws.
* 2: structural degradation without functional breakage, such as redundancies or unnecessary logic.
* 3: no negative side effects, or no evidence of problems mentioned.

M_defesa — Defensive Coverage Index
* 1: explicitly incomplete solution or one unable to handle relevant scenarios.
* 2: handles the specific case in the issue, with no evidence of additional coverage.
* 3: explicit handling of critical implicit cases or adoption of additional robustness mechanisms.

Return exclusively a valid JSON object:

{
  "M_profundidade": <integer from 1 to 3>,
  "just_profundidade": "<quote or summary of at most 15 words>",
  "M_integridade": <integer from 1 to 3>,
  "just_integridade": "<quote or summary of at most 15 words>",
  "M_defesa": <integer from 1 to 3>,
  "just_defesa": "<quote or summary of at most 15 words>"
}

Input:
Analytical Reasoning: `{raciocinio_analitico}`

[END OF PROMPT]

Output Structure

The final JSONL file contains two distinct data blocks:

• LLM_Judge_Metrics — stores the ordinal metrics (M_profundidade, M_integridade, M_defesa) and their respective short justifications;
• Metadata_Metrics — stores the M_coerencia result along with the original fields (resolve_issue_original and score_confianca_original).

Metric Processing

The processing script iterates over the 500 instances in evaluation_results.jsonl, inserts the raciocinio_analitico into the prompt, and sends the request to the model. The implementation uses:

• System Instructions — evaluation rules and output format isolated from the individual inputs;
• Structured output — response_mime_type="application/json" to request JSON responses directly, eliminating the need for regular-expression processing;
• Resilience — try/except blocks and progressive backoff to handle failures and API rate limits during processing;
• Local computation — the consistency metric (M_coerencia) is computed locally by the script, without a new API call.

The result of this processing is the basis for consolidating the semantic metrics and for the multidimensional analysis of the LLM-generated patches.

📄 Execution script: processar_metricas.py
📄 Generated artifact (output): metricas_estruturais_extraidas.jsonl

Step 09: Metric Processing and Consolidation

⬆️ Back to top

This script acts as an integration (ETL) pipeline, consolidating the metrics from three distinct evaluation dimensions — structural, semantic, and functional — into a standardized tabular matrix optimized for multivariate statistical analysis.

Execution Flow (Pipeline)

1. Structural Data Ingestion (SonarQube)

• Loads static code analysis metrics (e.g., cognitive complexity, bugs, code smells).
• Input: metricas_completas_swebench.csv

2. Semantic Data Extraction (LLM Evaluation)

• Parses the generated inferences, isolating the numeric scores associated with the heuristic metrics M_profundidade, M_integridade, and M_defesa.
• Input: metricas_estruturais_extraidas.jsonl

3. Functional Data Mapping (pass@1)

• Processes the functional test validation artifact. Applies boolean encoding to the resolution status: 1 (successfully resolved) and 0 (failed). The reference file (results.json) was retrieved from: https://github.com/shawnlewis/swebench-experiments/blob/main/evaluation/verified/20240402_rag_gpt4/results/results.json. It was renamed to "results-20240402_rag_gpt4.json".
• Input: JSON file of execution results.

4. Merge and Sanitization

• Performs an inner join across the three datasets using instance_id as the primary key.
• Cleans the dataset: retains only the numeric features of interest to the project and discards incomplete or inconsistent records (NA removal).

5. Export (Output)

• Performs the final casting to standardize numeric types (integers and floating points) and persists the unified dataset.
• Output: matriz_consolidada.csv

📄 Execution script: consolidar_valor.py
📄 Generated artifact (output): matriz_consolidada.csv

Step 10 - Normality (Shapiro-Wilk) and Statistical Tests

⬆️ Back to top

This script applies the statistical tests to the consolidated data matrix. Its role in the pipeline is to assess the distribution of the data and compute the correlation of the independent metrics (structural and semantic) with the binary dependent variable indicating the functional success of the patch (pass@1).

1. Sample Imbalance Validation and Logging:

• The code computes the exact success rate by dividing the resolved instances by the total evaluated (n_resolvidas / n_total).
• As a control mechanism, the script prints a console warning ([Nota de limitação]) reporting the class imbalance (success vs. failure). This step informs the parameterization of the future predictive modeling, with direct reference to Section 4.6 of the paper.

2. Feature Selection and Filtering:

• The script sanitizes the dataframe to keep only the variables of interest for the statistical test.
• Inclusions: Isolates the three semantic metrics (M_profundidade, M_integridade, and M_defesa) and the structural metrics sqale_index and violations (used technically as proxies in the experiment).
• Exclusions: Removes the code_smells and bugs columns from the data structure before the mathematical processing.

3. Statistical Test Execution:

• Shapiro-Wilk Test: The algorithm applies the Shapiro test to assess the normality of the distribution of each selected metric.
• Correlation and p-value Analysis: The script cross-references each metric individually with the pass@1 identifier. The routine's goal is to compute the correlation and extract the p-value ((p\text{-value})) of each variable, structuring the numeric base that will be exported or plotted for the final analysis of the experiment (identification of (p < 0.05)).

📄 Execution script: CalcNormal-Spearma.py
📄 Generated artifact (output): matriz_correlação_spearman.csv, matriz_p_values_spearman.csv, and figura hotmap.png

Step 11: Multidimensional Analysis and Predictive Modeling

⬆️ Back to top

This script acts as a confirmatory validator. Its role in the pipeline is to implement a Binary Logistic Regression model to estimate the probability of functional success of the AI-generated patch (pass@1), based on an optimized subset of independent variables.

1. Feature Transformation and Engineering:

• Applies the log1p mathematical transformation to the vectors of the continuous variables (cognitive_complexity and ncloc). This step is necessary to compress the data scale, reduce skewness (the impact of outliers), and mitigate multicollinearity effects before the regression.

2. Variable Selection and Model Definition:

• The algorithm parameterizes the predictive model with exactly 4 features: M_profundidade, M_defesa, cognitive_complexity_log, and ncloc_log.
• Programmatically drops variables that exhibit high structural redundancy or that induce quasi-complete separation of the data (e.g., M_integridade), ensuring convergence of the Maximum Likelihood estimator.

3. Statistical Evaluation and Logistic Fit:

• Computes the Variance Inflation Factor (VIF) for the predictor variables to mathematically validate the absence of severe collinearity.
• Instantiates and fits the logistic regression estimator (via the sm.Logit statistical module) and computes the Odds Ratio matrix.

Standard Output (Console Output)

During execution, the script compiles and displays four analytical blocks on standard output:

• Sample Validation (EPV): Reports the exact proportion of resolutions (pass@1) in the dataset and computes the Events Per Variable (EPV) indicator. This calculation technically assesses whether the number of minority-class instances supports the number of variables included in the model, flagging reliability limits.
• Collinearity Diagnostics (VIF Matrix): Renders a VIF table for each predictor, classifying the values into predefined thresholds: OK (<5), Warning (5–10), or Problematic (≥10). The algorithm appends an automatic textual assessment to this output.
• Statistical Summary (resultado.summary()): Prints the full dump of the regression model, showing the fitted coefficients ($\beta$), standard errors, z statistics, and p-values associated with each model variable.
• Odds Ratio Matrix: Displays the final inference on the weight of each variable in resolving the issue, tabulating:
• Odds Ratio: The exponential of the computed coefficient ($e^\beta$);
• Confidence Interval (95%): The lower (2.5%) and upper (97.5%) probabilistic bounds;
• P-value: The isolated significance level of each feature.

📄 Execution script: Final2-3Analis-semant-py