Fix command line crashing

[jan-janssen]

Certain queuing systems like flux list pyiron jobs as failed, this is fixed now.

[pmrv]

Does this mean on certain queuing systems the log file is not properly written or we are running in a different directory?

[jan-janssen]

Everything that happens in pyiron_base/jobs/job/wrapper.py uses its own logger. So in this case no pyiron.log file exists. When we submit jobs to the queue it internally calls python -m pyiron_base.cli wrapper -p {{working_directory}} -j {{job_id}} so these calls do not write log files as they use the wrapper module. Consequently, when we try to delete the log file but not log file exists the job crashes and returns an exit status 1 rather than 0. At least the flux queuing system lists these jobs as failed, while pyiron lists them as finished correctly. With this pull request this is fixed.

[pmrv]

The patch is perfectly good, I'm just trying to understand why this happens only with flux and not pysqa. From #1120 I guess that it also just calls python -m pyiron_base.cli wrapper -p {{working_directory}} -j {{job_id}}. I'll try and see if I can at least make everything use the same logging file.

[pmrv]

So I found that wrapper does its own logger instantiation (of the same name) here as opposed the global one, which doesn't add a FileHandler for pyiron.log. I'm guessing this is because we don't want to have a pyiron.log in the job folder (at least on our default SLURM we capture stdout/stderr [from the default StreamHandler] into error.log instead). Then again pyiron_base.jobs.job.wrapper does import pyiron_base.state.logger via pyiron_base.state, which should then install also the FileHandler for pyiron.log. So the only influence of pyiron_base.jobs.job.wrapper.setup_logger should be to change the default logging levels.

The only way I can then see the logging file not being present is if the python program crashes very early, before either state or the wrapper are imported, probably because of some other import error. Is it possible that you only had those errors with flux because some environment variable or other was not forwarded and that made it fail so early? (Guessing, I have no idea how flux works.)

Either way, we should maybe discuss what is the intended logging that we want and simplify the situation accordingly.