In-line PR comments sometimes request redundant changes

[KalleV]

I’m fairly often seeing this type of PR feedback where the change already being applied is added as the PR feedback comment:

Another seemingly consistent way to reproduce this seems to be when adding try / catch around a code block that did not have it before.

[okotek]

Do you have a public PR you can share with this behavior?

[KalleV]

I can try to set up a reproduction of this issue but I can see the prompts are clearly trying to avoid this already:

pr-agent/pr_agent/settings/pr_reviewer_prompts.toml

Line 8 in 36ab12c

- Make sure not to provide suggestions repeating modifications already implemented in the new PR code (the '+' lines).

Do you think tweaking that instruction or its placement would make a difference? Something like:

- It is essential to avoid suggesting changes that mirror modifications already introduced in the new PR code (the '+' lines). Ensure that your feedback brings new insights and does not reiterate the changes already incorporated in the PR.

[okotek]

Cc @mrT23

[mrT23]

@KalleV It's tempting to use longer and longer prompts to try and fix cases that didn't work 100% well.
Usually, there are better ways to go.

You can try your modified prompt on the bad cases you saw. Did it improve ?

To try improve the prompting for this specific issue, I would focus on a different aspect -further explain to the model what are the '+' lines. Maybe be a specific example, simulating a hunk in the prompt, would help
(similar to

pr-agent/pr_agent/settings/pr_code_suggestions_prompts.toml

Line 46 in 36ab12c

Example input:

)

(If you prefer, share a public example that gave bad results, and I will try an improved prompt)

[KalleV]

I wasn't able to create a minimum public reproduction yet but I came across a PR consistently requesting a try/catch. Not sure if the PR size makes a big difference but according to https://llmtokencounter.com/, the system prompt + user prompt is about 4801 tokens 19205 characters 5004 words.

I couldn't consistently prove small tweaks such as:

+ - It is essential to avoid suggesting changes that mirror modifications already introduced in the new PR code (the '+' lines).
- - Make sure not to provide suggestions repeating modifications already implemented in the new PR code (the '+' lines).

Or rearranging the system instructions led to a consistently better response but it was fun to try!

[mrT23]

ok.

notice that the '/improve' tool works differently in terms of patch and prompt design, and I think it will happen less there, so you can you it instead

[mrT23]

I am closing this issue. if you encounter this again in the '/improve' command, feel free to open another

[KalleV]

Sounds good, thank you!