Problem when training: Somehow training_orig_images become colored

[xyy-Iv]

Firstly, thank you for your tremendous work on colorization. It gives me a lot of new ideas.

Here is my problem, I have used your training code for my own training process. I have noticed that it generate scaled images like 128*128 or 256*256 before training process and it takes a lot of time. But when I train on my own dataset, the BlackAndWhiteTransform() transformation is not working. The input image is become colorful. So the whole training is become meaningless.

And I did add BlackAndWhiteTransform() to all my progressive GANs.

x_tfms = [BlackAndWhiteTransform()]
scheds.extend(GANTrainSchedule.generate_schedules(szs=[64, 64],
                                                  bss=[64, 64],
                                                  path=IMAGENET,
                                                  x_tfms=x_tfms,
                                                  extra_aug_tfms=extra_aug_tfms,
                                                  keep_pcts=[1.0,1.0],
                                                  save_base_name=proj_id,
                                                  c_lrs=c_lrs,
                                                  g_lrs=g_lrs,
                                                  lrs_unfreeze_factor=lrs_unfreeze_factor,
                                                  gen_freeze_tos=gen_freeze_tos))

I don't know what the problem is, please help me. Thank you very much.

[jantic]

I've actually run into issues with Tensorboard doing weird caching of images in the browser. Have you tried moving the train_gen_images scrubber tool (the orange thing)?

[xyy-Iv]

Yeah, of course.
I thought it would be some display errors, but in fact, it is not -- The train_gen_images trained so well that it must be some weird changes in input images.
Also refresh the webpage didn't get any better.

And maybe there is something wrong when the tmp images are first generated. After the first time it would be normal black and white images. I don't know where the bug is.

This is the second time I trained:

[jantic]

That looks like it's working then.... in that second image you posted here.

[xyy-Iv]

Yeah, I found that the tmp image files should be generated first and then go to the training process. Otherwise, the input would become colorful.

Don't know what the problem is.

[jantic]

The x_tfms are dynamic, so it shouldn't have anything to do with the tmp folder. If you can replicate it as a legit issue I'll certainly try to hunt it down.

[jantic]

Going to close this for now.

[chakri-muthyala]

@xyy-Iv How many training images you've taken, I'm taking ~3k images of anime, is it enough?
@jantic Please suggest!

[jantic]

@ChakriMuthyala It turns out that very soon there will be an update to DeOldify that should help you out quite a bit. The short of it is that you will be able to take the network having been pertained on Imagenet for colorization (which has over a million images), and then fine tune it with the anime images. I haven't tried this specifically yet but I'm almost certain it should work great. The fine tuning would first consist of pertaining with non-gan training generator only, then pertaining the critic to do binary classification on generated vs real images. Finally you do the (no)GAN training to top it off, which will be described in detail in the readme.