Deadlock Issue when submitting new jobs to genie

[amitsharma1981]

In Genie, the Janitor Thread periodically runs to check jobs which don't have update time changed in the last configured time. At the same time, when a new job is submitted after the initial init in the db, genie tries to run an update query changing status to RUNNING after launching the job.

The above two threads end up in a deadlock, causing the job to remain in INIT state in db, but the process gets launched successfully.

We should retry this update call to see if that fixes the issue.

[amitsharma1981]

Deployed and verified the fix in our production environment. The job was successfully submitted in-spite of hitting a deadlock in production and there has been no 5xx errors since deployment last week. Will re-open if the problem happens again.