Question: when to use Barrier Execution Mode ?

[ttpro1995]

I see mention Barrier Execution Mode of in mmlspark lightgbm documentation . The documentation does not clear for me.

I would like to know in what specific use case to use Barrier Execution Mode (is there a error that require to turn on Barrier Execution Mode ?), and when it is not necessary.

[imatiach-msft]

@ttpro1995 great question. There is some doc on barrier execution mode here:
https://github.com/Azure/mmlspark/blob/master/docs/lightgbm.md#barrier-execution-mode
It's a new feature in spark that was introduced in 2.4. Here is some doc on it:
https://www.waitingforcode.com/apache-spark/apache-spark-2.4.0-features-barrier-execution-mode/read
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-barrier-execution-mode.html
https://jira.apache.org/jira/browse/SPARK-24374
Since I've added I have found a lot of issues with it though. It doesn't automatically coalesce the number of partitions to the number of workers, which I was hoping it would take care of because that's one of the parts of the code with the most issues. It also does not support dynamic allocation, it just errors out, which is a long standing item that I need to fix in the current implementation.
Currently, I would not use it be default unless you are getting some errors during lightgbm training, in which case it may be one thing to try out.

[ttpro1995]

please make documentation in "Barrier Execution Mode" clear.

It is seem unclear for me.