Send email when last job is done vs. last submitted job is done

[renozao]

Currently an email is sent when the last submitted job is done, but, this one could finish before any/all of the other submitted jobs.
It would be nice to rather get an email when the job of a same submission batch last finishes.
This could be implemented as an other option to the mail.* configuration variables , e.g., 'last_done'.

[berndbischl]

There is a reason we did not do this yet.
How do you figure out on the slave that current job is indeed the last one?
We would need to query the DB for this, right?
As we need to check whether all other jobs are done.

We did not do this, as less load on the DB always seems preferable
(and see the dozen other issues and improvements we added to the package where slave DB load was a problem).

@mllg What do you thing? This would be a getJobsDone call at the end of each slave job.
The default would be to have this mailing feature turned off. Is this worth the effort / a good idea?

[renozao]

Yes I understand the implications on the DB, and you know better if this could be too much
Alternative ways of doing it, without loading the DB:

• not ideal, but a dependent job that is launched only when all other jobs are done (error or success) and just sends a summary email. This is not ideal since the job may be stuck in queue but it should be easy to implement (at least on systems that allow for such jobs)
• use a temporary shared directory where each separate process creates empty files (touch) with pattern nodename_pid_<N>, where N is the number of jobs performed by this given process (error or success). At the end of a job, the worker list the files and sum up Ns to get the number of finished jobs. There is a possible non-blocking race condition, which may make no emails be sent, but this could be limited if listing files twice, once before (= N_1) and once after touching the file (= N_2), with a random short sleep time in between. If N_1 >= N_job - 1 or N_2 >= N_job, then an email is sent.