Skip to content

bug(score): grounding checker counts URLs, env vars, and HTTP paths as broken file references #164

@stackbilt-admin

Description

@stackbilt-admin

Problem

charter score on aegis-daemon reports 177 broken path references. The top entries reveal the path resolver is not filtering non-file strings:

  • AEGIS_LOCAL_URL=http:/localhost:11434 — env var assignment pulled from a code block
  • C:/Users/kover/Documents/aegis-daemon — Windows absolute path in a markdown table
  • /chat, /api/runs, /webhooks/voice/complete — HTTP route paths in a table

None of these are file paths, but the grounding checker treats any string that looks like a path as a reference to validate.

False positive categories

  1. URL strings — anything containing :// or starting with http should be excluded
  2. Env var assignmentsKEY=value lines in code blocks should not have the value extracted as a path
  3. HTTP routes — strings starting with /api/, /webhooks/, /chat etc. in markdown tables are URL paths, not filesystem paths
  4. Windows absolute pathsC:/... paths in documentation tables are reference documentation, not resolvable local paths
  5. Env var names — bare ANTHROPIC_API_KEY tokens are not file paths

Impact

These false positives inflate the broken reference count from a likely real count of ~5–10 to 177, dragging the Grounding score from what should be ~18/20 to 7/20. This makes the score misleading and undermines trust in the metric.

Fix

Before running path resolution:

  1. Strip any token that matches https?:// (URL)
  2. Strip any token that matches ^[A-Z_]+= (env var assignment)
  3. Strip any token that starts with /api/, /webhooks/, /health, /chat (HTTP route heuristic, or: only validate paths that resolve to a plausible relative file under the repo root)
  4. Strip Windows absolute paths ([A-Z]:/)
  5. Only validate paths containing a file extension (.ts, .md, .json, etc.) OR paths that start with ./ or ../ (explicit relative references)

Alternatively: require file references to appear in a specific ADF notation (e.g., ref:path/to/file) rather than doing heuristic path extraction from all text.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions