Repository navigation
Question: do you test for unintended side effects? #4606
Unanswered
gokuljayaprakash8
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi, I ran a live test where an LLM agent with an issue-creation tool was given an injected note in third-party text. A check of the intended issue passed 10/10 while an unauthorized extra issue existed. I found it by reading the repository, not the agent's log: https://github.com/gokuljayaprakash8/kalpang-observer/blob/main/EB006_RESULTS.md
I haven't tested PR-Agent. It runs /review, /ask and /improve on comment events with write permissions and reads untrusted PR text, so I'm curious whether you check what the repository shows beyond the intended comment or label. I'm running this check on a few open-source GitHub agents and can run it on a sandbox copy of yours and share the result if that's useful.
All reactions