Skip to content

Orphaned heading remains in answer after CODE_RECOMMENDATIONS splice #6

@will-fawcett

Description

@will-fawcett

Background

PR #5 fixed bug #2 from the post-v0.0.1 review: `_assess_factuality_issue` now correctly removes the entire `<CODE_RECOMMENDATIONS>...</CODE_RECOMMENDATIONS>` block from the `answer` field. Verified end-to-end during the v0.0.1 smoke test against the live EVE API.

Issue

The LLM commonly precedes the `<CODE_RECOMMENDATIONS>` block with a markdown heading introducing it, e.g. "### Recommendations". When the block is spliced out, the heading is left dangling with no content underneath:

```
... main assessment body ...



Additional Notes

...
```

(Real example, captured from a smoke run of `assess_factuality_issue` against a GEE NDVI script.)

Severity

Cosmetic — not a correctness issue.

  • The answer is still complete, correct, and useful.
  • `code_recommendations` is populated correctly as a separate field.
  • A renderer that displays the answer markdown will show an empty heading line, which looks slightly odd but doesn't lose information.

Why this isn't covered by the existing fix

The fix in PR #5 splices out exactly `<CODE_RECOMMENDATIONS>...</CODE_RECOMMENDATIONS>` (tags inclusive). It doesn't look at surrounding context. Stripping a preceding heading requires heuristic reasoning about "what counts as a heading that introduces the block" — there's no clean structural signal because:

  • The heading level varies (`###`, `####`, `Code Recommendations`, etc.).
  • The heading text varies ("Recommendations", "Suggested Updates", "Code Changes", etc.).
  • Sometimes there's also a horizontal rule (`---`) above and/or below.
  • Occasionally the LLM omits the heading altogether, in which case stripping would remove a legitimate sibling heading.

Possible approaches

  1. Strip an empty trailing heading after splicing. After removing the `<CODE_RECOMMENDATIONS>` block, scan backwards for a line matching `^#{1,6}\s*$` (or one followed only by whitespace and `---`) and remove it. Conservative and structural — only acts on lines that are actually empty headings post-splice.
  2. Update the prompt to instruct the LLM not to put a heading immediately before `<CODE_RECOMMENDATIONS>`. Cheaper to ship, but LLMs are inconsistent at following negative instructions.
  3. Both. Tighten the prompt and clean up post-splice as a defensive layer.

I'd lean toward approach 1 — it's safe, doesn't depend on prompt compliance, and is easy to test.

Acceptance criteria

  • After `_assess_factuality_issue` strips the recommendations block, the resulting `answer` contains no `^#{1,6}\s*$` lines that weren't there before.
  • A regression test covers the case where the LLM precedes the block with a heading.
  • Existing tests still pass.

Out of scope

  • Changing the structure of `code_recommendations` (already valid JSON object).
  • Adding any other markdown post-processing beyond the empty-heading case.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions