Issues/54: on the multifile problem in spark-fits

[JulienPeloton]

What has changed?

This PR brings two major improvements:

• Solve the multifile problem that was seen in On the multifile problem in spark-fits #54
• Reduce a potential large overhead when reading many files, due to schema checks.

How this has been tested?

Unit test suite passes + additional integration tests performed. Seem all good, though I need to keep an eye on this

Is there anything left?

Yes for 20,000+ input files, the job explodes by sending many errors in the same times (probably related to each other):

java.lang.AssertionError:
        HDU number 0 does not exist!

java.lang.ArithmeticException: / by zero

java.util.NoSuchElementException: key not found: BITPIX

org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1151505977-134.158.75.222-1469858775214:blk_1075726089_1986406

.... (and then loop over those four)

I need to investigate a bit more.

[JulienPeloton]

For the record, the multifile problem was solved with minimal change: instead of looping over files and unioning RDD, I now give to Spark the full list of files and he fills the partition alone... Just magic.

[codecov-io]

Codecov Report

Merging #55 into master will decrease coverage by 0.21%.
The diff coverage is 90.9%.

@@            Coverage Diff            @@
##           master     #55      +/-   ##
=========================================
- Coverage   89.52%   89.3%   -0.22%     
=========================================
  Files           9       9              
  Lines         487     477      -10     
  Branches       87      88       +1     
=========================================
- Hits          436     426      -10     
  Misses         51      51

Impacted Files	Coverage Δ
...strolabsoftware/sparkfits/FitsSourceRelation.scala	97.36% <90.9%> (-0.31%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dff168a...6789c37. Read the comment docs.

[JulienPeloton]

Probably the last error arises because of ulimit -n being too small (1024). Need to investigate.

[JulienPeloton]

OK - the problem with >> 10,000 seems deeper than expected.
I will merge this PR since it solves a large part of the problem and open a separate one to this remaining issue.