Comments from a squirrel

[jhong93]

L119: Change to AdamW or Adam (less need to tune LR)

L85: no relus after final layer

L95: check docs to see if you are applying softmax twice by accident from nn.CrossEntropy

cross_entropy and cross_entropy_with_logits (check L119)
https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html

csknow/learn_bot/learn_bot/bot.py

Line 118 in f1551f4

loss_fn = nn.CrossEntropyLoss()

[David-Durst]

L119: changed. do I need to set any parameters? Or are the defaults all good?

L85: fixed

L95: I was, fixed

[David-Durst]

I notice that Adam gets the same test set avg loss after epoch 1 as SGD gets after epoch 5, but Adam never improves during epochs 2-5. I guess my loss is so small (0.04) that there's nothing more to learn?