loss progress

[Sh0lim]

Hi Matteo,

I was wondering how was the loss progress during your training? For example, after 3 epochs loss didnt change at all for me. It is around 8.1 and not dropping.
I was wondering how much loss was for you when you finished training, and how much we can expect to be after 49 epochs?

All the best.

[matteo-dunnhofer]

Hello @Sh0lim

unfortunately I don't have the logs of the experiments anymore so I hope to remember right. I remember that at the beginning I had to have patience, after a few epochs the loss started decreasing. But you could try to lower the learning rate by a factor of 10 earlier.
At epoch 49 the loss should be around 2.7...something.

[Sh0lim]

Yes, patience. :)
Anyway, that was useful. Thank you.

[donghn]

@Sh0lim Have you finished your training, yet? Can you tell me about your result. I got a similar problem.

[Sh0lim]

Hi @donghn,
i had problem with learning rate. I modify code a bit and there was bug. I was reducing learning rate by 10 every step, so learning rate was almost 0.
When I correct it, loss was doping down normally. Loss after 3 epoch was around 4.5, after 7 epochs was around 3. I started with learning rate 0.001. After 10 epochs i divide learning rate by 10. training is still running, I let you know what was the loss at the end. But I have old gpu, so I will not go to the end. Ill stop it after 20 epochs.

All the best.

[donghn]

@Sh0lim Yes. Thank so much. !!

[donghn]

@Sh0lim, @dontfollowmeimcrazy Hi you. Can you show me. How to run this program on GPU? I have installed tensorflow_GPU. So, when I run this program I checked my GPU and I found that the GPU was working for this training. But, It's so slow. It take over 12 hours for 1 epoch. thanks

[Sh0lim]

try with this. gpu_memory_fraction goes up to 1.

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=gpu_memory_fraction)

sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options, allow_soft_placement=True, log_device_placement=False))

with sess.as_default():
...

Best,

M.

[briansune]

@Sh0lim may i ask about the details of your workable settings? learning rate test_step and batch_size etc.
This will really helpful, i also encounter some loss stuck issues.

[matteo-dunnhofer]

Hi guys

if you happen to find better hyper parameters just submit a PR. That would be appreciated! 👍