the program runs error

[lijhong]

when I running the program ,there are some errors like this:
[2018-05-20 20:13:24 @logger.py:43] Trigger callback: Total counts of trainable weights: 10582152.
[2018-05-20 20:13:24 @weights_viewer.py:60] Total size of trainable weights: 0G 10M 94K 136B (Assuming32-bit data type.)
|##############################---------------------| 58.00%, 36937/26403 sec. train epoch 1, iter 2900/5000, loss 0.891349, 0.08 batches/sec.Traceback (most recent call last):
File "train.py", line 112, in
trainer.train()
File "/home/anaconda3/lib/python3.6/site-packages/neuralgym/train/trainer.py", line 133, in train
cb.run(sess, step)
File "/home/anaconda3/lib/python3.6/site-packages/neuralgym/callbacks/secondary_trainer.py", line 26, in run
self.train()
File "/home/anaconda3/lib/python3.6/site-packages/neuralgym/train/trainer.py", line 143, in train
assert not np.isnan(loss_value)
AssertionError

I can't solve it. is it machine problem? I hope you give me some ideas. thanks.

[lipanpeng]

@lijhong Some losses become NaN. Maybe you can adjust the parameter. I also have this problem before which is solved by adjusting the size of mask.

[lijhong]

@lipanpeng ok, I try it.thanks . I want to konw how much your mask size is .my image size is 64*64

[lipanpeng]

@lijhong The size of my mask is 56x56. The images I use are of size 256x256.

[lijhong]

i see.i mask size is 8x8 .but loss decrease slowly

[utunga]

FWIW I'm also getting this error. Attempting to train with a mask size of 192x192. I will go for a smaller mask but would far prefer to try and avoid this entirely can we clip the value of the loss or something?

[Tiuuuuu]

How can I set the size of mask?

[lipanpeng]

邮件已收到！

[Tiuuuuu]

IN inpaint.yml to set the size of mask?
Right?