LLM Misbehavior

[Fcpacheco]

This isn’t directly tied to BMAD, but I believe it could help us improve the method by better understanding a core weakness in how current LLMs behave under reinforcement learning (RL).

The article and accompanying YouTube video highlight an important finding: when an LLM is optimized using RL with reward signals, it can start skipping steps or cutting corners just to maximize reward. If we try to correct this by punishing that behavior, it often reduces the model’s overall task-solving ability instead of improving it.

I’ve seen this firsthand. Even when I provide clear instructions about things like test coverage, linting, or fixing specific warnings, the model often skips over them and still responds as if the task is complete. Sometimes, I have to switch models entirely just to get the same task done correctly, which adds a lot of unnecessary iteration and time.

While the article introduces a new model architecture to address this issue, I think the insight applies broadly to any LLM trained with RL-based reward systems.

It’s a good reminder that we may need structured safeguards or guardrails when building task workflows — especially in agentic settings — to reduce reward hacking behavior and keep model outputs aligned with our expectations.

Would love to hear your thoughts on this — especially if you’ve seen similar behavior or have ideas for better guardrail design!

https://www.youtube.com/watch?v=Xx4Tpsk_fnM
https://cdn.openai.com/pdf/34f2ada6-870f-4c26-9790-fd8def56387f/CoT_Monitoring.pdf

[bmadcode]

I can't wait to watch this later - Computerphile is one of my fav channels on YT - and what you are describing is an important topic for all to really understand how to work with LLMs, not just for the BMad method. Understanding this and maximizing techniques for working with the LLMs as they change and evolve is the new important skill to master. Prompt Engineering as a profession sounded silly a few years ago to most - now most professionals in industry agree and are seeing the power of understanding advanced techniques

BTW - CoT, ToT and other forms of working with LLMs are great to learn. I the latest version I added a file called Advanced Elicitation - I only added 9 to keep the list short - but this list will force the LLM to rethink what it just told you, consider alternatives, perform CoT, ToT - many of the other items in the list, some might sound funny (which they do produce fun results) but are all based on real techniques and theories.

Thank you @Fcpacheco for sharing this - great information for all to understand all of this at a deeper level!