Extension of the Worker Class

[jan-janssen]

No description provided.

[liamhuber]

Would be nice to offer some flexibility in how patient the user wants to be

[pmrv]

Could this be implemented in terms of Project.wait_for_jobs? If not then the arguments @liamhuber suggested should match that function. I'm fine with having this extra function for convenience, but we should avoid duplicated logic or interfaces.

[jan-janssen]

Could this be implemented in terms of Project.wait_for_jobs? If not then the arguments @liamhuber suggested should match that function. I'm fine with having this extra function for convenience, but we should avoid duplicated logic or interfaces.

The difference is basically that it searches for jobs based on the master_id rather than the path, which allows you to have multiple workers inside the same project and it sets the status of the worker to collect once all jobs are finished to stop the worker. So while I foresee in the long run the worker jobs just like pyirontable being meta jobs the user never actively works with, at the current stage I prefer to have a separate function.

[liamhuber]

I see the merit of having such wait code directly on the worker master, but Marvin's point about not duplicating things is also a very good one. Can we have our cake and eat it too?

What if we extract the waiting code, such that the source, times, etc. are all taken as arguments, then exploit this extracted behaviour at both the worker and project levels. The additional behaviour that the worker sets itself to collect should be easy enough to append after.

I'm sitting at 30k ft on this, so forgive me if I miss some impossible architecture obstacle. Seems like it should work though..

[jan-janssen]

I'm sitting at 30k ft on this, so forgive me if I miss some impossible architecture obstacle. Seems like it should work though..

While the functions have a similar effect for the user, their technical implementation is different e.g. filter by master id, write log file and only finish after time out, for the case the tasks for the worker are submitted from a different notebook. So I currently do not see an option to combine them. Internally they both use the job_table() to get the data from the database but the filtering and the reaction are different.