Query over the forward function of all models

[ankitpatnala]

BreizhCrops/breizhcrops/models/PETransformerModel.py

Lines 40 to 52 in 6de796e

	def forward(self,x):
	x = self.inlinear(x)
	x = self.pe(x)
	x = x.transpose(0, 1) # N x T x D -> T x N x D
	x = self.transformerencoder(x)
	x = x.transpose(0, 1) # T x N x D -> N x T x D
	x = x.max(1)[0]
	x = self.relu(x)
	logits = self.outlinear(x)
	logprobabilities = F.log_softmax(logits, dim=-1)
	return logprobabilities

I wanted to implement your models on my data and I found at line 50 of script ```PETransformerModel.py" log_softmax function has been implemented.

Secondly, when I saw your examples/train.py , I found criterion used is CrossEntropyLoss

BreizhCrops/examples/train.py

Lines 36 to 40 in 6de796e

	criterion = torch.nn.CrossEntropyLoss(reduction="mean")
	log = list()
	for epoch in range(args.epochs):
	train_loss = train_epoch(model, optimizer, criterion, traindataloader, device)

BreizhCrops/examples/train.py

Lines 170 to 182 in 6de796e

	def train_epoch(model, optimizer, criterion, dataloader, device):
	model.train()
	losses = list()
	with tqdm(enumerate(dataloader), total=len(dataloader), leave=True) as iterator:
	for idx, batch in iterator:
	optimizer.zero_grad()
	x, y_true, _ = batch
	loss = criterion(model.forward(x.to(device)), y_true.to(device))
	loss.backward()
	optimizer.step()
	iterator.set_description(f"train loss={loss:.2f}")
	losses.append(loss)
	return torch.stack(losses)

So, the loss is calculated on log_softmax which is not recommended by pytorch as nn.CrossEntropy Loss function already does that in its subroutine. (https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html).
I found that it has been done in other models too.

I am not sure whether it was done intentionally or an implementation bug or a mistake in my understanding. Can you explain me before I run my computation.