Flatten operator for Optional columns.

[imarios]

val ds: TypedDataset[(Int,Option[Int])] = TypedDataset.create(Seq((1,Option(1))))
ds.flatten('_2): TypedDataset[(Int,Int)]

[codecov-io]

Codecov Report

Merging #307 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #307   +/-   ##
=======================================
  Coverage   94.79%   94.79%           
=======================================
  Files          52       52           
  Lines         961      961           
  Branches        9        9           
=======================================
  Hits          911      911           
  Misses         50       50

Impacted Files	Coverage Δ
...ataset/src/main/scala/frameless/TypedDataset.scala	100% <ø> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c763176...a0e8c2e. Read the comment docs.

[OlivierBlanvillain]

Looks really cool! I'm I right saying that this is not in vanilla? A few comments:

• If not it should probably be better documented, at least with an example.

• The name make it sound more general that what it is. .flatten should be .flatMap(identity).

[imarios]

@OlivierBlanvillain Yea, I think in Vanilla you can use flatMap(_.a) on a Dataset, with the problem of having to serialize all the columns. Another way to do that with DataFrames is filter($"a".isNotNull), which is essentially the dynamically typed version of it. Both examples worth documenting, so let me do that.

Do you have any alternative naming suggestions?

[OlivierBlanvillain]

I seen, then maybe

• flatMapOption
• flatMapNotNull
• flattenOption
• flattenNotNull

?

[imarios]

@OlivierBlanvillain I think flattenOption feels more appropriate.

[OlivierBlanvillain]

LGTM! Could you maybe add a minimal example to the scaladoc of flattenOption?

For discoverability, what do you think about adding a "new APIs" and "missing APIs" sections to the readme? This way someone familiar with Spark could pickup frameless in no time by looking at this diff.

[imarios]

@OlivierBlanvillain done! If there are no more suggestions, let me squash and merge.

[OlivierBlanvillain]

LGTM!