Best parameter for larger VRAM.

[adamshamsudeen]

I am using a Tesla P100 with 16GB VRAM for training, with the default settings only 4GB of the VRAM is consumed. Which parameter in the settings will help me use the full potential of the GPU.
. My bleu score is always 0.
I increased the batch size to 1024 still, it remains the same.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.30 Driver Version: 390.30 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P100-PCIE... Off | 00000000:00:04.0 Off | 0 |
| N/A 40C P0 37W / 250W | 4551MiB / 16280MiB | 5% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1930 G /usr/lib/xorg/Xorg 82MiB |
| 0 25720 C python3 4459MiB |
+-----------------------------------------------------------------------------+

loaded eval model parameters from /home/adamshamsudeen/nmt-chatbot/model/translate.ckpt-51000, time 0.06s
eval dev: perplexity 13.00, time 0s, Thu Mar 15 13:49:54 2018.
eval test: perplexity 13.00, time 0s, Thu Mar 15 13:49:55 2018.
global step 51100 lr 0.001 step-time 2.49s wps 4.34K ppl 4.43 gN 10.65 bleu 0.00
global step 51200 lr 0.001 step-time 2.52s wps 4.36K ppl 4.45 gN 10.24 bleu 0.00
global step 51300 lr 0.001 step-time 2.50s wps 4.34K ppl 4.46 gN 9.46 bleu 0.00
global step 51400 lr 0.001 step-time 2.52s wps 4.33K ppl 4.44 gN 9.61 bleu 0.0

Thanks

[daniel-kukiela]

You should probably do not touch batch_size.
Instead of that you can increase num_layers and num_units. The other thing you want to play with is vocab_size (100k+ using BBE/WPM tokenizer and even higher for standard tokenizer).

[adamshamsudeen]

Great, will try it out. Btw why is my bleu score constantly at zero?

[daniel-kukiela]

Because it is not straight conversion from one language to another. We are using NMT to translate from english to english, and the more sentences in corpus, the more response will be way different than in training pair (even if still correct), and BLEU is calculated by comparison NMT output and response (sentence from to file).

[adamshamsudeen]

So that means we have to manually evaluate a model by feeding in questions.
Thank you for your time 👍