Loss exploding after few steps #5
Comments
|
Managed to correct above by changing hyperparameters,but after few steps i get |
|
I would advise using Adam for optimization. As for the latter error you reported, it is likely to be due to a singleton batch (i.e., a batch with only one sample). It may be due to the DataLoader, try setting the drop_last flag to True. D |
|
Thanks,corrected it,however i can see that reconstructed images have negative values in the tensor,is it desirable? sample slice of x_r *255 = [ 19.986425 33.786083 109.08704 ]] [[ 49.809772 -32.651962 -1.5576267] [[ -68.59759 64.74513 51.421898 ] If yes than how to get back RGB image from them? |
|
Negative values are not undesirable per se, as long as it is a concious choice. D |
|
input images are RGB images with standard 0-255 range being fed through dataloader pytorch torchvision.transforms.Grayscale(num_output_channels=1)` |
|
I would advise standardizing input images. D |
|
that makes sense,however standardization wouldnt guarantee negative output right,? |
|
Also i checked my input data is between 0 and 1 already, Pytorch default backend 1 for images are Pillow, and when you use ToTensor() 2 class, PyTorch automatically converts all images into [0,1]. So shouldn't the reconstruction output be standardized before calculating reconstruction loss or perhaps use another activation function? |
|
I would advise using a linear activation function for the reconstruction and providing the groundtruth image to the same loss in the same range as the input. |
|
Are you talking about the x_r from decoder,should i use sigmoid on top of last layer in decoder output,, i am trying to train the model to learn the reconstruction of the image itself,so GT will be the same image |
|
|
|
The sigmoid is not mandatory. I would advise not to use it. The rest of the code seems fine. Would an [0-1] input deliver those reconstructions? D |
|
Sorry didnt understand that part,my input is already [0,1],its the reconstruction which has negative values |
|
Does the reconstruction loss go down? |
|
How have you done the reconstruction of image,as in what was your input range in cifar images for instance and what was the activation function of decoder and range of reconstructed output? |
|
It has high values ~2500,since i am using sigmoid the learning is pretty slow,i suspect the gradients are very less due to higher output before sigmoid.After 100 epochs on dataset of size 1400 with batch size 64 the loss only decrease by approx 50 units. Can you guide me as to what have you done |
|
|
I see ,but how come the values are negative in my case,i mean weights arent negative,my input [0,1] no non linearity applied on top of your code,what is going wrong here? |
|
I saw you are using 128x128 images. The number of downsampling in the model is tuned on 32x32 images. You might then have a huge linear layer before the bottleneck of the autoencoder, with many parameters slowing down learning. As for the negative values, it is really weird. I would try optimizing the reconstruction loss only (plain autoencoder) and see if the problem fades. |
|
Thanks,I ll let you know the progress,let me lighten up the model a bit.. :) |
|
Keep in mind that, if I am right, the best way to lighten up the model would be to add downsample and upsample blocks. |
|
Yeah sure will do,but how do I deal with reconstruction with negative values,using sigmoid is the last option I would prefer.Any help? Did you encounter negative values while reconstruction ,if yes how did you deal with it,(apart from using activation),is standardizing the x_r right at inference time/validation time? |
`from functools import reduce
from operator import mul
from typing import Tuple
import numpy as np
import torch
import torchvision
import torch.nn as nn
from models.loss_functions.lsaloss import LSALoss
from models.base import BaseModule
from models.blocks_2d import DownsampleBlock
from models.blocks_2d import ResidualBlock
from models.blocks_2d import UpsampleBlock
from models.estimator_1D import Estimator1D
import cv2
class Encoder(BaseModule):
"""
CIFAR10 model encoder.
"""
def init(self, input_shape, code_length):
# type: (Tuple[int, int, int], int) -> None
"""
Class constructor:
class Decoder(BaseModule):
"""
CIFAR10 model decoder.
"""
def init(self, code_length, deepest_shape, output_shape):
# type: (int, Tuple[int, int, int], Tuple[int, int, int]) -> None
"""
Class constructor.
class LSACIFAR10(BaseModule):
"""
LSA model for CIFAR10 one-class classification.
"""
def init(self, input_shape, code_length, cpd_channels):
# type: (Tuple[int, int, int], int, int) -> None
"""
Class constructor.
def load_dataset(data_path="/home/jbmai/Downloads/Defect Images-20190705T133320Z-001"):
# data_path = 'data/train/'
torchvision.transforms.Grayscale(num_output_channels=1)
net = LSACIFAR10(input_shape=[3,128,128],code_length = 32,cpd_channels =100)
lossFunction = LSALoss(cpd_channels=100)
optimizer = torch.optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
try:
except Exception as e:
print (e)
for epoch in range(1000): # loop over the dataset multiple times
print('Finished Training')`
Output :
<class 'torch.Tensor'>
[1, 1] loss: 727109273.600
<class 'torch.Tensor'>
[2, 1] loss: 2495627954514337382531072.000
Hi can you help me rectify the issue.
The text was updated successfully, but these errors were encountered: