train crashed.

[10183308]

when I train the model, it crashed.

F0718 15:20:34.923629 16833 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
./train_voc_reduced.sh: line 7: 16833 Aborted (core dumped) python tools/train_net.py --gpu 0 --solver models/pascalvoc/VGG16-REDUCED/solver.prototxt --imdb voc_2007_trainval --weights data/ImageNet_models/VGG_ILSVRC_16_layers_fc_reduced.caffemodel --batchsize 64 --iters 4000

root$ nvidia-smi
Tue Jul 18 15:29:09 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66 Driver Version: 375.66 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 Off | 0000:06:00.0 Off | 0 |
| N/A 69C P0 57W / 149W | 0MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K80 Off | 0000:07:00.0 Off | 0 |
| N/A 51C P0 71W / 149W | 0MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 Tesla K80 Off | 0000:83:00.0 Off | 0 |
| N/A 62C P0 58W / 149W | 0MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 Tesla K80 Off | 0000:84:00.0 Off | 0 |
| N/A 48C P0 72W / 149W | 0MiB / 11439MiB | 99% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

[taokong]

batchsize is too large