Write mode "Ignore" is incorrectly treated as "Overwrite"

[wajda]

Shouldn't we skip a write event completely if the target exists and the mode is "Ignore"?

[wajda]

Since Spline listener is triggered only after the result has been written to the destination, we cannot really check if the destination existed before writing on not.
Luckily along with the executed physical plan Spark 2.3+ also provides associated metrics. including numOutputRows one. What can we do is to look at the number of rows produced by all the terminal nodes, and if equals to zero then it could only mean the plan wasn't actually executed, so we can skip the lineage as well.

[wajda]

Correcting myself. We are interested in numFiles on a write node only. Zero changed files means that result was ignored.