Yaml hotreloading needs an integration test

[slarticodefast]

It broke in #5874 and was fixed in #5907 and we did not notice for a while.
The test would somehow have to modify some local files to see if the file watcher notices the changes.

[Tayrtahn]

Interestingly enough, there is one.

Looks like it forces the reload to happen though, instead of using the file watcher. So we should either expand or change it to cover that too.