UCF C3D train fails with 'Unknown Layer Type : NdConvolution' error

[timanish]

Hi!

I am using CPU_ONLY build of this project. My frames/videos etc are all in correct location. While running train_ucf101.sh (c3d_ucf101 example), I am getting the following error

I0329 20:02:23.644191   388 layer_factory.hpp:77] Creating layer data
I0329 20:02:23.644265   388 net.cpp:106] Creating Layer data
I0329 20:02:23.644301   388 net.cpp:411] data -> data
I0329 20:02:23.644348   388 net.cpp:411] data -> label
I0329 20:02:23.644840   388 video_data_layers.cpp:39] Opening file examples/c3d_ucf101/c3d_ucf101_train_split2.txt
I0329 20:02:23.644953   388 video_data_layers.cpp:59] Shuffling data
I0329 20:02:23.645027   388 video_data_layers.cpp:64] A total of 22 video chunks.
I0329 20:02:23.662021   388 video_data_layers.cpp:106] output data size: 30,3,16,112,112
I0329 20:02:23.722612   388 net.cpp:150] Setting up data
I0329 20:02:23.722738   388 net.cpp:157] Top shape: 30 3 16 112 112 (18063360)
I0329 20:02:23.722796   388 net.cpp:157] Top shape: 30 (30)
I0329 20:02:23.722846   388 net.cpp:165] Memory required for data: 72253560
I0329 20:02:23.722899   388 layer_factory.hpp:77] Creating layer conv1a
F0329 20:02:23.722992   388 layer_factory.hpp:81] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: NdConvolution (known types: AbsVal, Accuracy, ArgMax, BNLL, BatchNorm, BatchReindex, Bias, Concat, ContrastiveLoss, Convolution, Data, Deconvolution, Dropout, DummyData, ELU, Eltwise, Embed, EuclideanLoss, Exp, Filter, Flatten, HDF5Data, HDF5Output, HingeLoss, Im2col, ImageData, InfogainLoss, InnerProduct, LRN, Log, MVN, MemoryData, MultinomialLogisticLoss, PReLU, Pooling, Power, ReLU, Reduction, Reshape, SPP, Scale, Sigmoid, SigmoidCrossEntropyLoss, Silence, Slice, Softmax, SoftmaxWithLoss, Split, TanH, Threshold, Tile, VideoData, WindowData)
*** Check failure stack trace: ***
    @     0x7f7d4c835daa  (unknown)
    @     0x7f7d4c835ce4  (unknown)
    @     0x7f7d4c8356e6  (unknown)
    @     0x7f7d4c838687  (unknown)
    @     0x7f7d4cbd97cd  caffe::LayerRegistry<>::CreateLayer()
    @     0x7f7d4cd05fd5  caffe::Net<>::Init()
    @     0x7f7d4cd08145  caffe::Net<>::Net()
    @     0x7f7d4cced03a  caffe::Solver<>::InitTrainNet()
    @     0x7f7d4ccee14c  caffe::Solver<>::Init()
    @     0x7f7d4ccee47a  caffe::Solver<>::Solver()
    @     0x7f7d4cc984c3  caffe::Creator_SGDSolver<>()
    @           0x40e42e  caffe::SolverRegistry<>::CreateSolver()
    @           0x40778b  train()
    @           0x4056ec  main
    @     0x7f7d4bb43ec5  (unknown)
    @           0x405e21  (unknown)
    @              (nil)  (unknown)
Aborted (core dumped)

Any help understanding this error is appreciated!

Thanks

[chuckcho]

Sorry. NdPooling/NdConvolution layers depend on CUDA. While it's possible to implement them in CPU, I don't have resource to do that. If upstream does get that update later, I'll pull that in.

[fengwhu]

@chuckcho I use solver_mode: GPU, but I still get this error. Is there anything wrong when I build caffe?

[chuckcho]

@fengwhu I assume you have Cuda set up, but still crash with the same message? video-caffe does depend on Cuda (and optionally CuDNN).

[fengwhu]

@chuckcho Yes, I have Cuda set up, and can run the Mnist example with Cuda in caffe. The CaffeConfig.cmake file in build directory of video-caffe also shows as
# Cuda support variables
set(Caffe_CPU_ONLY OFF)
set(Caffe_HAVE_CUDA TRUE)
set(Caffe_HAVE_CUDNN FALSE)

[chuckcho]

Haven't had an issue for me (on several machines) -- so, it's hard to troubleshoot. I think it's related to the building process. Could you try cmake if you used make before? Or, vice versa.

[mzolfaghari]

I get the same error!

415 20:07:18.243010 27280 layer_factory.hpp:81] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: NdConvolution (known types: AbsVal, Accuracy, ArgMax, BNLL, BatchNorm, BatchReindex, Bias, Concat, ContrastiveLoss, Convolution, Data, Deconvolution, Dropout, DummyData, ELU, Eltwise, Embed, EuclideanLoss, Exp, Filter, Flatten, HDF5Data, HDF5Output, HingeLoss, Im2col, ImageData, InfogainLoss, InnerProduct, LRN, Log, MVN, MemoryData, MultinomialLogisticLoss, PReLU, Pooling, Power, ReLU, Reduction, Reshape, SPP, Scale, Sigmoid, SigmoidCrossEntropyLoss, Silence, Slice, Softmax, SoftmaxWithLoss, Split, TanH, Threshold, Tile, VideoData, WindowData)
*** Check failure stack trace: ***
@ 0x7fea959a8b7d google::LogMessage::Fail()
@ 0x7fea959aac7f google::LogMessage::SendToLog()
@ 0x7fea959a876c google::LogMessage::Flush()
@ 0x7fea959ab51d google::LogMessageFatal::~LogMessageFatal()
@ 0x7fea95fe70dd caffe::LayerRegistry<>::CreateLayer()
@ 0x7fea960ef435 caffe::Net<>::Init()
@ 0x7fea960f15a5 caffe::Net<>::Net()
@ 0x7fea95f9202a caffe::Solver<>::InitTrainNet()
@ 0x7fea95f9313c caffe::Solver<>::Init()
@ 0x7fea95f9346a caffe::Solver<>::Solver()
@ 0x7fea960c9f13 caffe::Creator_SGDSolver<>()
@ 0x40fdce caffe::SolverRegistry<>::CreateSolver()
@ 0x409180 train()
@ 0x406bdc main
@ 0x7fea94cbaec5 (unknown)
@ 0x407311 (unknown)
Aborted (core dumped)

[chuckcho]

@timanish @mzolfaghari @timanish I suspect this is related to CUDNN. NdConvolution/NdPooling requires CUDNN. Can you guys check?

[fengwhu]

Yes. It is exactly related to CUDNN! I install CUDNN and build video-caffe again, it works.

[mzolfaghari]

Finally, video-caffe build successfully and network starts working. First of all check if you installed CuDNN version >= 4. I tried with CuDNN version 3 but couldn't built it!

set path of cudnn and cuda library and include directories in you bashrc and Makefile.config. Also, you should add path of opencv and all required libraries.
Start with the following commands in the root folder of video-caffe:
make all ( use make all -j8 for faster run)
to test if everything works fine:
make test
make runtest

If you want to use cmake for building the video-caffe (I coudn't build successfully with cmake!), check if CuDNN configured correctly. With using cmake you should get the following configuration:
-- Found cuDNN (include: /usr/local/cuda-7.0/include, library: /usr/local/cuda-7.0/lib64/libcudnn.so)

-- NVIDIA CUDA:
-- Target GPU(s) : Auto
-- GPU arch(s) : sm_30
-- cuDNN : Yes

if there is some problem with cmake to recognize the CUDNN try running ccmake and setting CUDNN_INCLUDE, CUDNN_LIBRARY and CUDNN_ROOT.

or you use the following command with your paths :
cmake -DCUDNN_INCLUDE="/your/path/to/include" -DCUDNN_LIBRARY="/your/path/to/lib" /path/to/caffe

Best,

[chuckcho]

@mzolfaghari thanks for sharing this information, and it's good that you finally built and run it. :)