Could the concepts in the scientific paper about PEARL help find hallucinated and bloated code that still compiles?

[LevyBytes]

Could the concepts in the scientific paper about PEARL help find hallucinated and bloated code that still compiles?

A hypothesis

I came across PEARL: Path-Entity Aligned Relational Learning with Contextual Subgraphs for Inductive Knowledge Graph Completion, through a YouTube video which I felt was interesting and has some parallels with the spatial intelligence I'm working on adding to my harness Kaizen.

My hypothesis is that those references could provide more than just relationships. I suspect the concepts could be applied to detect patterns that don't belong in a codebase's code domains.

I asked Codex (GPT 6 Astra Ultra) to investigate. I asked, if there's a positive finding, show the maths. Treat the paper as evidence, and don't make up data or experimental results to support the idea.

So, potentially useful slop begins below. Something to examine before deciding whether it belongs in my Kaizen harness.

---

The response below is from OpenAI Codex, edited into this discussion format. The cited paper was read; the proposed code-analysis method has not been implemented or experimentally validated. (yet)

Model response

Yes—as a plausible method for identifying suspicious code and prioritizing review. The paper does not establish that it detects code hallucinations or proves code is inefficient. Those applications would require additional representations, verification, and experiments.

PEARL combines graph context with relational paths. Its experiments concern knowledge-graph completion on WN18RR, FB15k-237, and NELL-995. Reading its experiments and appendices, and searching for code-anomaly terminology, revealed no evaluation of code correctness, manual-grounded analysis, or code optimization. Paper, section 5 and Appendix B

The proposed connection is to assess a code operation through relationships between what it does, what its API requires, and what the surrounding program guarantees.

The mathematics available from the paper

PEARL's scoring and training equations, with notation shortened, are:

$$f(q)=w^\top[m_h\Vert m_t\Vert e_r\Vert g\Vert p]+b$$ $$\mathcal L_{\mathrm{rank}} = \frac{1}{N}\sum_{i=1}^{N} \max\!\left(0,\gamma+f(q_i^-)-f(q_i^+)\right)$$

Here $m_h,m_t$ represent endpoints, $e_r$ the relation, $g$ the surrounding graph, and $p$ the contextualized path evidence. Training favors positive triples over negative ones by margin $\gamma$. Paper, equations 31 and 33; path construction in equations 16–26

Proposed adaptation: code and documentation in a shared graph

This is an unvalidated adaptation, not a demonstrated result from the paper.

Represent parsed code, documented requirements, and independently checked reference implementations in a shared graph:

Evidence source	Relationships to represent
Technical manuals	API preconditions, permitted states, effects, version constraints
Program analysis	Calls, data flow, control flow, ordering, branch conditions
Verified reference implementations	How particular requirements are satisfied

Every relationship should retain its source location. A proposed query would be:

$$q=(\text{code operation},\ \text{satisfies},\ \text{documented requirement})$$

The model would rank how strongly the surrounding evidence supports that claim. The claim being evaluated, and any inverse that reveals it, must be withheld from the input graph.

This addresses the "compiles but is wrong" case: the learning target becomes satisfaction of a behavioral requirement. Compilation would be an admission condition for the examples, not the correctness label.

Training would require independently established valid examples and compiling examples with verified violations. Manuals and good examples alone do not establish that every unfamiliar implementation is wrong.

Why the score is not a probability of hallucination

The ranking equation has an important mathematical limitation. Replacing every score with

$$f'(q)=f(q)+c$$

leaves the loss unchanged, because

$$f'(q^-)-f'(q^+)=f(q^-)-f(q^+).$$

But applying a sigmoid to $f'(q)$ changes its numerical value. The ranking objective alone therefore cannot justify interpreting a score as "probability of hallucination." This is a direct algebraic deduction, not an experimental finding.

A proposed way to use the scores is to compare each claim with held-out, verified valid claims of comparable requirement type and context:

$$\rho(q)= \frac{ 1+\sum_{i=1}^{n}\mathbf{1}\!\left[f(q_i^{\mathrm{valid}})\le f(q)\right] }{n+1}$$

Here $n$ is the number of reference claims, and $\mathbf{1}$ is 1 when its condition is true and 0 otherwise.

Small $\rho(q)$ means the claim has unusually low support relative to those references. This is a proposed empirical relative rank, not a validated confidence measure. Thresholds and false-positive rates would need independent evaluation.

Missing documentation should produce an "insufficient evidence" outcome. A suspected violation still requires checking the documented requirement against the actual program. An unusual pattern does not establish a defect or identify who generated the code.

Bloat needs a separate equivalence-and-cost check

For bloat or suboptimality, add a separate mathematical criterion. Let:

• $C$ be the candidate implementation.
• $C'$ be an independently checked alternative.
• $D$ specify required behavior, effects, error handling, operating conditions, and other constraints.
• $c_j$ measure an agreed cost, such as runtime, allocations, memory, or a precisely defined structural measure.

For this proposal, a sufficient criterion for an inefficiency finding is:

$$\begin{aligned} C' &\equiv_D C,\\\ c_j(C') &\le c_j(C) &&\text{for every selected cost }j,\\\ c_j(C') &< c_j(C) &&\text{for at least one selected cost }j. \end{aligned}$$

This establishes that $C'$ dominates $C$ on the measured dimensions under the specified conditions. It does not establish global optimality.

This criterion is a proposed addition, not a PEARL result. Pattern recognition could retrieve promising alternatives and identify suspicious repetition; equivalence checks and cost evidence would substantiate the finding.

Agreement on finite tests establishes agreement on those inputs, not unrestricted equivalence. A shorter reference implementation is not enough: its behavior and relevant constraints must match. If equivalence cannot be established, the proposed replacement remains a candidate for investigation.

What would need verification

Any prototype should meet these requirements:

• Verify extracted manual requirements before treating them as authority. An LLM's plausible interpretation remains a hypothesis.
• Preserve semantics when augmenting code for training. PEARL's contextual node sampling cannot simply be transferred to deleting program conditions. Paper, equations 27–29
• Evaluate against real reviewed compiling defects and valid unfamiliar code, including separate repositories and API versions.
• Compare with existing deterministic analyzers and measure both useful detections and false alarms.
• Require actual equivalence and cost evidence for any inefficiency finding.

The positive finding is mathematical suitability for an evidence-linked anomaly-ranking component. Whether it adds useful detection capability remains unverified. The separate equivalence-and-cost criterion supplies a basis for defensible bloat findings.

No detector training, code benchmarks, reproduction of PEARL, or performance measurements were performed for this review.