Unable to train ComputationGraph with SharedTrainingMaster

[ChrisYohann]

Hello everyone, I'm facing some issues with training a ComputationGraph in Spark using the gradient sharing implementation.

I'm using Spark 2.4.4, on a YARN cluster. The executors threads are still in a TIMED_WAITING state, and my model is never fit, the weights are never updated.

I don't have the problem using ParameterAveraging method, and I don't think it's a network issue. In fact, using the debugger on a Spark Executor, I think the problem is here :

https://github.com/eclipse/deeplearning4j/blob/58aa5a3a9bbd2c1cbe041c81a57182e4c5d62606/deeplearning4j/deeplearning4j-scaleout/spark/dl4j-spark-parameterserver/src/main/java/org/deeplearning4j/spark/parameterserver/pw/SharedTrainingWrapper.java#L105

The variable iteratorMDS is not initialized, unlike the variable iteratorDS, and later, when fitting on a MultiDataSet under the run method :

                while((iteratorDS != null && iteratorDS.hasNext()) || (iteratorMDS != null && iteratorMDS.hasNext())) {
                    //Loop as a guard against concurrent modifications and RCs

                    if (wrapper != null) {
                        if (iteratorDS != null)
                            wrapper.fit(iteratorDS);
                        else
                            wrapper.fit(iteratorMDS);
                    } else {
                        // if wrapper is null, we're fitting standalone model then
                        if (iteratorDS != null) {
                            if (model instanceof ComputationGraph) {
                                ((ComputationGraph) originalModel).fit(iteratorDS);
                            } else if (model instanceof MultiLayerNetwork) {
                                ((MultiLayerNetwork) originalModel).fit(iteratorDS);
                            }
                        } else {
                            if (model instanceof ComputationGraph) {
                                ((ComputationGraph) originalModel).fit(iteratorMDS);
                            } else if (model instanceof MultiLayerNetwork) {
                                ((MultiLayerNetwork) originalModel).fit(iteratorMDS);
                            }
                        }
                    }
  

My condition is never met since iteratorDS is empty and iteratorMDS is null.

I use a MultiDataSet since I'm training a ComputationGraph. As I expect, It works when I use a MultiLayerNetwork.

Version Information

• Deeplearning4j version : 1.0.0-beta6
• Platform information : Amazon EMR & Mac OS 10.15.3 (Tested on both)

[AlexDBlack]

Thanks for the report - and for pointing out the location of the problem (you were right about the cause).

A fix has been implemented here: KonduitAI#318
It will be merged back to eclipse/deeplearning4j soon (next week at the latest), and will be in snapshots soon after that - https://deeplearning4j.konduit.ai/config/config-snapshots
We'll be doing a release soon (2-3 weeks hopefully) as well.