Unknown job_id errors

[khusmann]

Hello,

I'm running dask on a PBSCluster, and was getting "Unknown job_id" errors when workers attach to the scheduler. It doesn't seem to be causing any problems (jobs still process ok), but wanted to see how I could go about getting rid of these messages. I turned on debugging and get this:

DEBUG:job id for new worker: 11202854.server.address.domain
DEBUG:11202854.server.address.domain is a new job or restarting worker
ERROR:Unknown job_id: 11202854.server.address.domain for worker tcp://ip-address:37826

(actual addresses replaced by "server.address.domain", actual ip address replaced by ip-address)

What's happening here, and how can I stop the "Unknown job_id" errors? Thanks!

Note: I think it might have to do with the regex of the job_id? Earlier in the log I get:

DEBUG:started job: 11202854

So I think perhaps it's looking for "11202854.server.address.domain" when it should be looking for just "11202854"

[guillaumeeb]

Hi and thanks for sharing your problem!

Dask-jobqueue tries to keep track of submitted jobs in order to avoid scaling more than necessary. So it keeps an internal dic of pending, running, ended jobs.

In your case, the job which connect to the scheduler does not match any of the pending ones. So Your analysis is correct.

Normaly, the expression at this line https://github.com/dask/dask-jobqueue/blob/master/dask_jobqueue/pbs.py#L98 should keep only the numerical part of the jobid. I suspect it does not in your case, could you verify this?

[khusmann]

That was it! Just submitted a pull request. Thanks!