Any tips to reduce training time

[cricket1]

• I am using p2.x.large machine on aws .It has 12gb gpu memory

• For batch size > 1 I see high gpu utility and gpu memory usage .Thus ending up with out of memory issues

• With batch size = 1 ,an epoch takes 4 hrs ,thus 3000 epochs = 500 days :)

• Is there a pretrained model which I can use for training

• Is such High usage normal.I see that each image is 10kb and mask is 2.5kb.I am using chainer for the first time .Am i doing something wrong .

[bobye]

Make sure you have cudnn enabled for chainer installation. I am able to run with batchsize = 3 after I fixed the cudnn issue (for 12gb GPU).

[shiba24]

Hi, thank you for the comments, @bobye and @cricket1 .
Large memory usage is sometimes inevitable for this kind of neural network.
Just one tip: chainer recently implemented Forget function (http://docs.chainer.org/en/stable/reference/functions.html#forget), and using this definitely we can reduce memory usage, make batchsize larger! (Sorry i do not have enough time to implement this now, though...)

[cricket1]

@bobye @shiba24 thnks for the input will try both

[xscjun]

could somebody share a pretrained model ?

[bobye]

@xscjun I guess this project is still in progress, actually not complete. You are welcome to contribute.

[shiba24]

In a few weeks I will try to reduce memory usage by implementing forget function as long as we can increase GPU utils.

[GhadeerMohamad]

Hello, Tnx for the great work, and is there anyway to save checkpoints while training the model since i haven't found the trained model after i finished training.
thank you