Training own dataset

[junyang-cynopsis]

Hi I want to train my own dataset.

I received the following error
The directory name is invalid: 'weights\rrdn-C4-D3-G64-G064-T10-x2\2019-05-27_23:41'

Any reasons why?

[junyang-cynopsis]

Okay I got it, apparently windows cant create folders with semicolon in its filename, which appears in the timestamp. I removed the timestamp(or semicolon) and it works now

[junyang-cynopsis]

I have an issue now, the code doesnt save my hdf5 weights after end of epochs training, any reasons why?

[junyang-cynopsis]

Okay I got it too

So we need to set the monitor metrics to be specifically one of the following keys, cant use the default "val_generator_PSNR". ie example use "val_generator_PSNR_Y"

{'train_d_real_loss': 0.060769938, 'train_d_real_acc': 1.0, 'train_d_fake_loss': 0.027885916, 'train_d_fake_acc': 1.0, 'train_loss': 0.18021318, 'train_generator_loss': 0.1242274, 'train_discriminator_loss': 4.2881703, 'train_feature_extractor_loss': 2.6231012, 'train_generator_PSNR_Y': 17.87545}

{'val_loss': 0.14819694967009128, 'val_generator_loss': 0.1413433525711298, 'val_discriminator_loss': 0.6808462247252465, 'val_feature_extractor_loss': 2.6356906309723853, 'val_generator_PSNR_Y': 17.35666718482971}

[cfrancesco]

Okay I got it, apparently windows cant create folders with semicolon in its filename, which appears in the timestamp. I removed the timestamp(or semicolon) and it works now

Cool, I was not aware of this issue, thank you for pointing it out. I will look into a fix.

Okay I got it too

So we need to set the monitor metrics to be specifically one of the following keys, cant use the default "val_generator_PSNR". ie example use "val_generator_PSNR_Y"}

There needs to be at least one monitored metric amongst the available metrics. If not, it should only save the weights during the fallback saves. I will double check the behavior.
I am strongly considering to remove a weight saving logic in favor of a "save every epoch, mark if it is best on some metric" kind of logic, especially due to the difficulties while evaluating the perceptual quality (when GANS and VGG loss are introduced).
Thank you for your effort!