LogSoftmax in the output but not in the description/code (page 532)

[labdmitriy]

Hi Sebastian,

There is an output of created RNN model which includes log softmax as the last layer on the page 532:

(softmax): LogSoftmax(dim=1)

But based on the code of the model and on the following steps we do not need this layer because we use nn.CrossEntropyLoss() where the input is expected to contain raw, unnormalized scores for each class.
Is it correct?

Thank you.

[rasbt]

Good catch. I remember seeing it and flagging it in several places. It's unfortunate that one of it slipped through! You are right, CrossEntropyLoss works with logits and the LogSoftmax shouldn't be there. However, it also doesn't break the network:

E.g., consider the example data:

import torch.nn.functional as F


logits = torch.tensor([
    [-1.1, 0.1],
    [-0.4, 2.1],
    [3.1, -1.1]]
)

y_target = torch.tensor([1, 0, 1])

Then

>>> ce = torch.nn.CrossEntropyLoss()
>>> ce(logits, y_target)
tensor(2.3524)

and

>>> log_softmax = torch.log(F.softmax(logits, dim=1))
>>> ce(log_softmax, y_target)
tensor(2.3524)

Because

>>> log_softmax == torch.log(F.softmax(log_softmax, dim=1))
tensor([[True, True],
        [True, True],
        [True, True]])

The LogSoftmax is unnecessary and redundant though. It's basically applied twice, @haydenliu.

PS: Thanks for all the other comments. Please keep posting! I am trying to catch up in the next couple of days.

[rasbt]

Oh, now I see. It just shows up in the description/output on the next slide but it's not actually used. Phew 😅

[labdmitriy]

Hi Sebastian,

Yes I mean it is not used anywhere in the code and in the following steps :)
In any case, thank you for a very useful remark about log softmax.

[rasbt]

Added it to the errata.