Currently if an app fails, the reloader alert message is pretty aggressive. If you close it, it will re-appear almost immediately. We should only show the `alert` once per run.