[Rant] Best optimization step: change random seed

[carasuca]

Please share thoughts here if any: does it make sense for an iterative research project to apply a seed change, report improvement and call it a day?
To me that is unsustainable, what if you plan continue the research - do you keep that seed value as the "best known" hyper-parameter?!

[ishita-0301]

I agree. A single seed improvement can just be noise. I'd only accept it if it stays better across multiple seeds or repeated runs. Also worth having runtime guardrails for long autonomous loops so agents don't get stuck or chase misleading wins. I've been following https://github.com/FailproofAI/failproofai for ideas around that.