Sporadic NoSuchJobException in sort tests

[hannes-ucsc]

http://jenkins.cgcloud.info/job/toil-pull-requests/1017/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsSingle/

[hannes-ucsc]

Another one: http://jenkins.cgcloud.info/job/toil-pull-requests/1164/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsSingle/

[cket]

AWS with mesos: http://jenkins.cgcloud.info/job/toil-pull-requests/1205/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsMesos/

[hannes-ucsc]

I've added a Lucene indexer to Jenkins so we can do full text search on build results:

http://jenkins.cgcloud.info/search/?q=NoSuchJobException

[hannes-ucsc]

http://jenkins.cgcloud.info/job/toil-branches/branch/master/167/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsMesos/

[hannes-ucsc]

http://jenkins.cgcloud.info/job/toil-pull-requests/1445/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsMesos/

[joelarmstrong]

Hmm, I was sort of hoping the fix to #808 would fix this too, but after looking at it, it looks like it's either a race condition (exists check followed right after by a load) or stale reads from SDB. I tried to aggravate the race locally by sleeping in between but no dice.

edit: Now that I think about it, you could just do a single load and catch the NoSuchJobException rather than checking existence separately, but that seems like sweeping the problem under the rug.

[hannes-ucsc]

I think this is a stale read after delete. We've been in touch with AWS support about this. They didn't outright reject the idea of SDB not being fully consistent and recommended that we upgrade to DynamoDB.

I wonder is why this happens in the same code location every time.

[hannes-ucsc]

… meaning that if it is a stale read, the time window should be somewhat flexible and therefore cause different problems in different places.

[hannes-ucsc]

http://jenkins.cgcloud.info/job/toil-pull-requests/1478/testReport/junit/src.toil.test.sort.sortTest/SortTest/testAwsSingle/

[hannes-ucsc]

I've been trying to brute-force reproduce this by creating and deleting jobs in quick succession. I tried 16k jobs. No dice. We should entertain the possibility of there being a the possibility of the job being deleted concurrently, maybe by the worker. Then again, why does this only happen with AWS? If there is a race between leader and worker, this should manifest itself in other stores as well/

[hannes-ucsc]

Punting for next sprint.

[cket]

Given that this is most likely a SDB issue, I like @joelarmstrong's solution. We can fix this particular symptom to curb the problem until we switch to DynamoDB, if/when that happens.