fix cornercase / functionality / default tests

[chu11]

It has been a long time since I have tested "default", "cornercase" and "functionality" tests (I normally just test specific hadoop/spark versions). it appears various test errors have creeped in over the years as little things in newer versions have changed.

e.g. newer sparks assume hadoop 3+, leading to different output expectations, but "defaults" and some other tests still assume hadoop 2.X output.