Skip to content

bug: example/demo penalty (-2) still leaves examples at high confidence in top ranks #128

Description

@aryamthecodebreaker

What happened?

Example/demo paths are only penalized by 2 points (EXAMPLE_CODE_PENALTY), so they still land in the top ranks at high confidence on real implementation tasks.

Against expressjs/express (fresh remote fetch) with:

npx -y @aryam/fixmap@0.7.3 plan --repo https://github.com/expressjs/express \
  --issue "req.query is empty when using a custom query parser" --format json

Observed:

rank path score confidence note
1 lib/application.js 22 high good (compileQueryParser)
2 lib/response.js 20 high questionable for query parsing
3 examples/mvc/index.js 18 high reason includes “example or demo code deprioritized”
4 lib/utils.js 18 high also defines compileQueryParser
5 lib/request.js 16 high actual request query surface
6–8 more examples/… 14–12 high/medium same weak penalty

So the deprioritization fires (reason string present) but does not keep examples out of the agent’s top window. lib/request.js — the natural fix site for req.query — sits below an example.

Source: packages/core/src/rank.ts (EXAMPLE_CODE_PENALTY = 2, AUXILIARY_CODE_DIRS).

How can we reproduce it?

Run the express command above; confirm examples/ entries in the top 8 with high confidence despite the deprioritized reason.

What did you expect instead?

Examples should drop below first-party lib/ (or at least below high confidence) for implementation tasks unless the task text targets demos/examples. A −2 flat penalty is too weak when content overlap can add 10–20 points.

FixMap version or commit

v0.7.3

Where did this happen?

CLI

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingevaluationBenchmarks, datasets, and ranking quality

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions