Training speed

[bugtig]

Hello, thank you for your work.
With the default settings on a 1080 and TF 1.0, i'm getting about 13 secs per size 16 batch, which would mean 1 epoch takes about 3 days, which is clearly off. Do you any ideas what may be causing the slowdown?

[abisee]

That speed sounds about right. RNNs are very slow for long sequences, unfortunately.

In the "Experiments" section of the paper we note that we find it expedient to start training with highly-truncated sequences, then increase max_enc_steps and max_dec_steps once the loss curve has flattened out. For example, you could start with max sequence lengths only 50 (you could even try 20 or 10), and gradually work up to max_enc_steps=400 and max_dec_steps=100.

Edit: This is now in the README.

[bugtig]

Thank you!

[StevenLOL]

Indeed , the code uses cpu 0 ...

https://github.com/abisee/pointer-generator/blob/master/run_summarization.py#L109

[abisee]

@StevenLOL It uses GPU for the main computations: https://github.com/abisee/pointer-generator/blob/master/model.py#L294

You can see which ops are performed on which device by looking at the "graph" in Tensorboard. You can change any of these to fit your needs.

[pltrdy]

@StevenLOL @bugtig same here. As referenced by tianjianjiang we also discussed the fact that Nvidia 1080 is completely useless if the device is set to CPU. It is really close to 0% load on GPU which is problematic.