Início

The evaluation of Large Language Models (LLMs) on code generation tasks has been conducted predominantly through benchmarks such as HumanEval, MBPP, and SWE-bench, which measure exclusively the functional correctness of generated artifacts. Such instruments are unable to capture structural dimensions of software quality — such as cyclomatic complexity and coupling — nor qualitative maintainability attributes, such as readability, resulting in an incomplete understanding of the models’ actual capabilities in production contexts. To address this limitation, this work proposes a multidimensional evaluation framework that extends functional analysis by integrating deterministic structural metrics, obtained via SonarQube, and a semantic evaluation of robustness and architectural adequacy, conducted through a parameterized LLMas-a-Judge method. The central question investigated is whether a correlation exists between performance on functional benchmarks, internal structural quality, and the readability of generated code. The study draws on data from the execution of the SWE-bench Verified benchmark by the GPT-4 model [Lewis 2024], comprising 500 test instances, whose generated code is submitted to semantic analysis via LLM-as-a-Judge, totaling 500 executions. Contributions include a reproducible methodological pipeline and empirical evidence on the multidimensional relationships among functionality, structural rigor, and interpretive maintainability of code synthesized by artificial intelligence.