SortSam ShardedInput tests failing.

[jamesemery]

@cmnbroad and I have both observed that the SortSamSparkIntegrationTest.testSortBAMsSharded tests fail locally on our machines despite the tests apparently working on travis. The tests fail because the comparator detects the files are out of their reported sort order. When I went digging into the failing tests it appears that the files are getting correctly sorted and written out correctly into 2 shards with proper names (filename-0000 and filename-0001). After reading the sharded directory as input, it appears that the two files are read out of order. That is to say that calling readsRDD.collect() clearly places all of the filename-0001 reads before the filename-0000 reads.

After digging around it appears the problem might lie in Disq somewhere as it appears everything is working as expected until the abstractSamSource.getReads() line is encountered in HtsjdkReadsRddStorage. I suspect something is going awry with the filesystem mechanism for ordering the input files on our Macs that travis is sidestepping.

Out of curiosity @tomwhite I thought that the sharded output wrote headerless bam chunks, but that appears not to be the case at all? Was I wrong in that assumption or did that change when we switched to Disq.

[lbergelson]

@jamesemery This seems to be striking everywhere now. Any further thoughts?

[lbergelson]

Further investigation:

This seems to be cause by the fact that when reading part's files Hadoop does not order them by filename. Instead they are ordered by whatever order the file system happens to return them in which is explicitly NOT sorted.

The root cause can be seen in FileInputFormat.listStatus() which calls singleThreadedListStatus in one of it's branches to get a list of input files. This delegates to whatever the file system does. The filesystem returns things in random order apparently.

We can fix this by sorting the results of listStatus.