Is it Done?

[chamcham115]

I0627 03:28:07.408565 10388 solver.cpp:244] Train net output #20: per_class_accuracy = 0.037037
I0627 03:28:07.408572 10388 sgd_solver.cpp:106] Iteration 74980, lr = 5e-06
I0627 03:28:14.539170 10388 solver.cpp:454] Snapshotting to binary proto file /home/chamcham/Dev/ENet/weights/snapshots_encoder/enet_iter_75000.caffemodel
I0627 03:28:14.571485 10388 sgd_solver.cpp:273] Snapshotting solver state to binary proto file /home/chamcham/Dev/ENet/weights/snapshots_encoder/enet_iter_75000.solverstate
I0627 03:28:14.706820 10388 solver.cpp:317] Iteration 75000, loss = 0.411978
I0627 03:28:14.706852 10388 solver.cpp:322] Optimization Done.
I0627 03:28:14.706856 10388 caffe.cpp:254] Optimization Done.
*** Aborted at 1498501694 (unix time) try "date -d @1498501694" if you are using GNU date ***
PC: @ 0x7fba3dfb813c (unknown)
*** SIGSEGV (@0x0) received by PID 10388 (TID 0x7fba406eda40) from PID 0; stack trace: ***
@ 0x7fba3df6bcb0 (unknown)
@ 0x7fba3dfb813c (unknown)
@ 0x7fba3fef8a87 caffe::Blob<>::~Blob()
@ 0x7fba3ff90ecc caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer()
@ 0x7fba3ffca7d4 caffe::DenseImageDataLayer<>::~DenseImageDataLayer()
@ 0x7fba3ffca909 caffe::DenseImageDataLayer<>::~DenseImageDataLayer()
@ 0x40e226 std::vector<>::~vector()
@ 0x40ec10 caffe::Net<>::~Net()
@ 0x40edc9 caffe::Net<>::~Net()
@ 0x7fba3ff12690 caffe::AdamSolver<>::~AdamSolver()
@ 0x40d2fe boost::detail::sp_counted_base::release()
@ 0x40b0a7 train()
@ 0x4089ac main
@ 0x7fba3df56f45 (unknown)
@ 0x4092b3 (unknown)
@ 0x0 (unknown)
Segmentation fault (core dumped)

[ChidanandKumarKS]

Currently iam using GEforce GTX Titan Z...Iam not able to train 100 iteration till now since it is throwing error=cudaSuccess(2 Vs 0) out of memory.
Kindly suggest and do the needful

Regards
Kumar

[TimoSaemann]

This error should not occur. Since the necessary snapshots are saved before the error occurs, your training has been successfully completed.