Replace _LOCK file mechanism with query for running jobs?

[GoogleCodeExporter]

Instead of using a _LOCK file on HDFS / S3 to indicate a CL is running, perhaps 
better to look for running jobs by querying the cluster.

Original issue reported on code.google.com by srowen@myrrix.com on 29 Nov 2012 at 1:13

[GoogleCodeExporter]

Now, the CL will query the cluster for any running M/R jobs from the CL, and 
refuse to proceed until they go away. This substantially precludes two CLs from 
running at the same time, though there is still a small chance that two may run 
if started simultaneously. The caller should of course not run multiple 
instances. This behavior however makes it simple to restart a failed CL and 
have it reliably wait for any outstanding steps to finish before continuing.

The _LOCK file no longer exists, which was a poor mechanism anyway.

Original comment by srowen@myrrix.com on 30 Nov 2012 at 7:14

• Changed state: Fixed