Parameter setting

[kenrubiooo]

Hi @xinntao,

I am trying to train with upscale of x2. However, I am still encountering this error

RuntimeError: The size of tensor a (256) must match the size of tensor b (128) at non-singleton dimension 3

1. What should be the paths in the dataroot_LQ in the .yml file? Are they supposed to be the generated directories after running the generate_mod_LR_bic.py code? I am always pointing to the ./srtoolbox/dataset/DIV2K/mod/DIV2K800_sub/LR/x2 and to the ./srtoolbox/dataset/DIV2K/mod/DIV2K100/LR/x2 directories for both training and validation datasets.

2. What should I place in the GT_size in the .yml file? Is it the step or the crop_sz parameter in the extract_subimgs_single.py?

Thank you!

[kenrubiooo]

Aside from the .yml file, do we need to change some parameters or lines in the RRDBNet_arch.py? I am trying to root out the cause of the errors being encountered. Thank you!

[xinntao]

1. I think you are right.
   I attach a config file of X2 model of my own (change the txt to yml).
   train_SRResNetX2.txt

2. Make sure your generated images are OK. And check whether they have the same number of images.
   For me, I use the following parameters. (Remembe to use differnet size for GT and LQ.)

GT:
crop_sz = 480
step = 240
thres_sz = 48

LQ:
crop_sz = 240
step = 120
thres_sz = 24

3. For RRDBNet,
   You need to comment two lines in RRDBNet_arch.py

https://github.com/xinntao/BasicSR/blob/f9139136821a25c036a5beb835d2f5e1658fff56/codes/models/modules/RRDBNet_arch.py#L57
and
https://github.com/xinntao/BasicSR/blob/f9139136821a25c036a5beb835d2f5e1658fff56/codes/models/modules/RRDBNet_arch.py#L69

[kenrubiooo]

1. I think you are right.
   I attach a config file of X2 model of my own (change the txt to yml).
   [train_SRResNetX2.txt](https://github.com/xinntao/BasicSR/files/3382645/train_SRResNetX2.txt)

2. Make sure your generated images are OK. And check whether they have the same number of images.
   For me, I use the following parameters. (Remembe to use differnet size for GT and LQ.)

GT:
crop_sz = 480
step = 240
thres_sz = 48

LQ:
crop_sz = 240
step = 120
thres_sz = 24

1. For RRDBNet,
   You need to comment two lines in RRDBNet_arch.py

https://github.com/xinntao/BasicSR/blob/f9139136821a25c036a5beb835d2f5e1658fff56/codes/models/modules/RRDBNet_arch.py#L57

and

https://github.com/xinntao/BasicSR/blob/f9139136821a25c036a5beb835d2f5e1658fff56/codes/models/modules/RRDBNet_arch.py#L69

This is much appreciated, @xinntao! I will update you on this training. I will comment and close if this goes well.

[kenrubiooo]

Hi @xinntao,

Here is what I did:

1. I cropped the DIV2K800 images using the extract_subimgs_single.py using these values:

    n_thread = 20
    crop_sz = 480
    step = 240
    thres_sz = 48

2. Then, I generated the HR, LR, and Bic folders for the DIV2K800_sub using the generate_mod_LR_bic.py

3. After that, I also created the lmdb files for the HR and LR images from the generated HR and LR folders.

4. I also did steps 1-2 for the DIV2K100 for the validation part in the .yml file.

Here is the train_ESRGANx2.yml file that I have. I am still having a CUDA out of memory error even though I have it running on a Titan RTX.

#### general settings
name: 003_RRDB_ESRGANx2_DIV2K
use_tb_logger: true
model: srgan
distortion: sr
scale: 2
gpu_ids: [1]

#### datasets
datasets:
  train:
    name: DIV2K
    mode: LQGT
    dataroot_GT: /home/ubuntu/srtoolbox/dataset/DIV2K/lmdb/DIV2K800_GTsub/HR.lmdb
    dataroot_LQ: /home/ubuntu/srtoolbox/dataset/DIV2K/lmdb/DIV2K800_GTsub/LR.lmdb

    use_shuffle: true
    n_workers: 6  # per GPU
    batch_size: 4
    GT_size: 480
    use_flip: true
    use_rot: true
    color: RGB
  val:
    name: DIV2K
    mode: LQGT
    dataroot_GT: /home/ubuntu/srtoolbox/dataset/DIV2K/mod/DIV2K100_sub/HR/x2
    dataroot_LQ: /home/ubuntu/srtoolbox/dataset/DIV2K/mod/DIV2K100_sub/LR/x2

#### network structures
network_G:
  which_model_G: RRDBNet
  in_nc: 3
  out_nc: 3
  nf: 64
  nb: 23
network_D:
  which_model_D: discriminator_vgg_128
  in_nc: 3
  nf: 64

#### path
path:
  pretrain_model_G: /home/wavemanila/srtoolbox/codes/esrgan/models/RRDB_ESRGAN_x4.pth
  strict_load: false
  resume_state: ~

#### training settings: learning rate scheme, loss
train:
  lr_G: !!float 1e-4
  weight_decay_G: 0
  beta1_G: 0.9
  beta2_G: 0.99
  lr_D: !!float 1e-4
  weight_decay_D: 0
  beta1_D: 0.9
  beta2_D: 0.99
  lr_scheme: MultiStepLR

  niter: 400000
  warmup_iter: -1  # no warm up
  lr_steps: [50000, 100000, 200000, 300000]
  lr_gamma: 0.5

  pixel_criterion: l1
  pixel_weight: !!float 1e-2
  feature_criterion: l1
  feature_weight: 1
  gan_type: ragan  # gan | ragan
  gan_weight: !!float 5e-3

  D_update_ratio: 1
  D_init_iters: 0

  manual_seed: 10
  val_freq: !!float 5e3

#### logger
logger:
  print_freq: 100
  save_checkpoint_freq: !!float 5e3

Your help is much appreciated. Thank you!

[yl-to]

Hi @xinntao,

Here is what I did:

1. I cropped the DIV2K800 images using the extract_subimgs_single.py using these values:

    n_thread = 20
    crop_sz = 480
    step = 240
    thres_sz = 48

1. Then, I generated the HR, LR, and Bic folders for the DIV2K800_sub using the generate_mod_LR_bic.py
2. After that, I also created the lmdb files for the HR and LR images from the generated HR and LR folders.
3. I also did steps 1-2 for the DIV2K100 for the validation part in the .yml file.

Here is the train_ESRGANx2.yml file that I have. I am still having a CUDA out of memory error even though I have it running on a Titan RTX.

#### general settings
name: 003_RRDB_ESRGANx2_DIV2K
use_tb_logger: true
model: srgan
distortion: sr
scale: 2
gpu_ids: [1]

#### datasets
datasets:
  train:
    name: DIV2K
    mode: LQGT
    dataroot_GT: /home/ubuntu/srtoolbox/dataset/DIV2K/lmdb/DIV2K800_GTsub/HR.lmdb
    dataroot_LQ: /home/ubuntu/srtoolbox/dataset/DIV2K/lmdb/DIV2K800_GTsub/LR.lmdb

    use_shuffle: true
    n_workers: 6  # per GPU
    batch_size: 4
    GT_size: 480
    use_flip: true
    use_rot: true
    color: RGB
  val:
    name: DIV2K
    mode: LQGT
    dataroot_GT: /home/ubuntu/srtoolbox/dataset/DIV2K/mod/DIV2K100_sub/HR/x2
    dataroot_LQ: /home/ubuntu/srtoolbox/dataset/DIV2K/mod/DIV2K100_sub/LR/x2

#### network structures
network_G:
  which_model_G: RRDBNet
  in_nc: 3
  out_nc: 3
  nf: 64
  nb: 23
network_D:
  which_model_D: discriminator_vgg_128
  in_nc: 3
  nf: 64

#### path
path:
  pretrain_model_G: /home/wavemanila/srtoolbox/codes/esrgan/models/RRDB_ESRGAN_x4.pth
  strict_load: false
  resume_state: ~

#### training settings: learning rate scheme, loss
train:
  lr_G: !!float 1e-4
  weight_decay_G: 0
  beta1_G: 0.9
  beta2_G: 0.99
  lr_D: !!float 1e-4
  weight_decay_D: 0
  beta1_D: 0.9
  beta2_D: 0.99
  lr_scheme: MultiStepLR

  niter: 400000
  warmup_iter: -1  # no warm up
  lr_steps: [50000, 100000, 200000, 300000]
  lr_gamma: 0.5

  pixel_criterion: l1
  pixel_weight: !!float 1e-2
  feature_criterion: l1
  feature_weight: 1
  gan_type: ragan  # gan | ragan
  gan_weight: !!float 5e-3

  D_update_ratio: 1
  D_init_iters: 0

  manual_seed: 10
  val_freq: !!float 5e3

#### logger
logger:
  print_freq: 100
  save_checkpoint_freq: !!float 5e3

Your help is much appreciated. Thank you!

Also got the cuda out of memory problem.

[xinntao]

You can use a smaller training patch size. Change GT_size: 480 to GT_size: 128
Note that when we make lmdb files, the crop size is not the training patch size (GT_size)

[kenrubiooo]

Hi, @xinntao,

What do you mean by this?
Note that when we make lmdb files, the crop size is not the training patch size (GT_size)

I think you mean, extract_subimgs_single.py since it is where the crop_sz parameter is found.
What should we put in the crop_sz parameter in the .yml if the crop size is not the training patch size (GT_size)?

Also for the validation part in the .yml file, should the HR and LR folders/directories be enough or should we still use the .lmdb files path?

[xinntao]

1. the crop_sz in extract_subimgs_single.py does not have relations with GT_size in the .yml file.
   It is OK to not crop sub-images by extract_subimgs_single.py. I think I may remove it because it raises too many misunderstandings.
   Actually, we first crop images to sub-images (for example 480*480). And then the data loader will further randomly crop them to GT_size (like 128*128).
2. For validation, you do not need to create lmdb files. Indeed, for training, you also do not need to create lmdb files.
   But for faster IO speed during the training, I recommend to use lmdb files.

[b4go3s]

hi @xinntao

if i want to change GT_size: 128 (128 x 128) in train_ESRGAN.yml file to 96 x 96 or 192 x 192 but error RuntimeError: size mismatch, m1: [16 x 18432], m2: [8192 x 100]

can you help me please, what should i do...

[xinntao]

@b4go3s
Do all the patches have the size larger than GT_size? I think this error may arise from that some patches have a smaller size than GT_size.

[b4go3s]

dear @xinntao
i change crop_sz = 760 step = 380 in extract_subimgs_single.py file & LR image size become 190, so i change GT_size: 192 in train_ESRGAN.yml file. so this if i run error.. so i must make it more larger size, crop_sz = 760 & LR image size = 190

because i calculate
extract_subimgs_single.py
crop_sz = 480
120 x 4 = 480

generate_mod_LR_bic.m
mod_scale = 4 #modulo scale
so LR image become 120 x 120
and GT_size: 128
this working if i run
can you give me example which one i must change, please...