Multi array label Softmax Cross Entropy loss function

[jSaso]

Description

We want to have multi array labels/prediction. Currently API can handle only one label (first element in NDArray). Obvious examples are Multi digit number recognition, where we can predict multiple digits from provided input. Example of this: https://github.com/eclipse/deeplearning4j-examples/blob/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/convolution/captcharecognition/MultiDigitNumberRecognition.java
Also there are many cases when we want multiple array labels, especially when dealing with just numbers (outputs) and predictions.
There is small thread about this (python):
https://datascience.stackexchange.com/questions/23614/keras-multiple-softmax-in-last-layer-possible

Documentation for this is very hidden or non existent - the same is with examples. Everyone is using just single array label, multiple classes (out neurons):
[0.2, 0.3, 0.6,...0.12] -> 0.6

We want to have multiple array labels, multiple classes (out neurons):
[0.2, 0.3, 0.6,...0.12] -> 0.6
[0.4, 0.2, 0.7,...0.88] -> 0.88
[0.11, 0.77, 0.55,...0.33] -> 0.77
:
:

Will this change the current api?

Hopefully only new loss function (class) can be implemented, which will calculate multi array labels (Split tensor to parts, then compute softmax separately per part and concatenate tensor parts at end) - SoftmaxCrossEntropyLossMulti.

References

• deeplearning4j framework has this implementation (https://deeplearning4j.org/) - see their example above

[frankfliu]

@jSaso
Thanks very much for reaching out to us.

Our currently API already support multiple input/multiple label/multiple output use case.
I agree it's not obvious by just looking at the API. And we don't have examples to show case this usage. We will add examples/document to make it clear.

Here is some quick pointer:

1. Loss API uses NDList, NDList is a list of NDArray, not a single NDArray, so it can handle multiple data and labels.
2. Batch API uses NDList type for data and labels, it's not NDArray, NDList can take multiple data and multiple labels.
3. Translator, when pre-processing and post processing data, you can see its also using NDList, which allows user handle multiple input and outputs.

NDList class not only allows users process NDArray by its order, you can actually use named NDArray, so you don't need worry about the order of the inputs. Here is code that handles multiple inputs: https://github.com/awslabs/djl/blob/master/mxnet/mxnet-model-zoo/src/main/java/ai/djl/mxnet/zoo/nlp/qa/BertQATranslator.java#L53-L62

[jSaso]

Thanks @frankfliu
I saw that API uses NDList. Thats why I thoot that multi array labels - multi class are supported. This is not the case in current API.
It would be nice to to have this supported - implement softmax loss function which support not just single label array, but multiple labels array.
Currently I don't have time to open PR about this feature, but it would be nice to have that feature in DJL API.

[zachgk]

@jSaso Could you use the CompositeLoss to combine multiple SoftmaxCrossEntropyLosses together?

[zachgk]

I created an example showing how to implement a multi array label softmax cross entropy loss function at https://github.com/awslabs/djl/blob/master/examples/src/main/java/ai/djl/examples/training/TrainCaptcha.java. Please take a look and let me know if there are any further problems or suggestions.