Every time devtools::document() is run, test helpers are run as well.
I put together a small package to illustrate the issue, located at nbenn/roxythat. This was the best kind of reproducible example I could come up with. Sorry for the hassle.
This might not be a bug. However I have a situation at hand where I only want the helper files to be run before tests. If this is not how test helpers are supposed to be used, I can figure something else out.