Deadlock when fetching experiments simultaneously.

[fmohr]

I frequently get an error when starting various jobs at the same time, but sometimes also when starting just a small number (64 jobs):

1205 (HY000): Lock wait timeout exceeded; try restarting transaction
 raised when executing sql statement.

The jobs won't start. I suppose it's a problem at the moment of fetching jobs from the table. Even restarting the mysql server does not help immediately. Sometimes, after several dozens of these errors, he eventually grabs the experiments and starts through.

There is no stack trace, and I have currently no clue why exactly it is happening. But it is a rather severe problem, because valuable cluster compute time is burnt only in waiting for the job to start.

I would suggest to rethink the way how the experiments are fetched. You might want to checkout the Java version in AILibs, in which we do a combined SELECT/INSERT statement to immediately reserve a job when it is being fetched (in the same query, without need of transaction).

[alexandertornede]

Do you have a minimal code example to reproduce this?

Also: I suppose this is about the MySQL version?

[tornede]

This issue should be solved with PR #76