Train on KITTI dataset

[goan15910]

Hi @BichenWuUCB, I got some problem loading the pkl file of caffemodel you provide.
The version of related packages on my server are listed below:
easydict==1.6
joblib==0.10.3
tensorflow==0.10.0rc0

And Here's the error log:
Traceback (most recent call last):
File "./src/train.py", line 281, in
tf.app.run()
File "/home/extra/goan15910/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "./src/train.py", line 277, in main
train()
File "./src/train.py", line 114, in train
model = VGG16ConvDet(mc, FLAGS.gpu)
File "/tmp3/jeff/squeezeDet/src/nets/vgg16_convDet.py", line 25, in init
self._add_forward_graph()
File "/tmp3/jeff/squeezeDet/src/nets/vgg16_convDet.py", line 39, in _add_forward_graph
self.caffemodel_weight = joblib.load(mc.PRETRAINED_MODEL_PATH)
File "/home/extra/goan15910/.local/lib/python2.7/site-packages/joblib/numpy_pickle.py", line 575, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "/home/extra/goan15910/.local/lib/python2.7/site-packages/joblib/numpy_pickle.py", line 507, in _unpickle
obj = unpickler.load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatchkey
File "/home/extra/goan15910/.local/lib/python2.7/site-packages/joblib/numpy_pickle.py", line 340, in load_build
self.stack.append(array_wrapper.read(self))
File "/home/extra/goan15910/.local/lib/python2.7/site-packages/joblib/numpy_pickle.py", line 183, in read
array = self.read_array(unpickler)
File "/home/extra/goan15910/.local/lib/python2.7/site-packages/joblib/numpy_pickle.py", line 144, in read_array
array.shape = self.shape
ValueError: total size of new array must be unchanged

I have no problem running the demo example.
Is this a package version problem?

[BichenWuUCB]

@goan15910 What's your command for launching the training thread? Did you make any modification to the config file?

[goan15910]

I use the vgg16 version in the train.sh and all config are the same as original.

[skyz8421]

When I tried to train SqueezeDet on KITTI dataset, I met the problem about joblib as following:

Traceback (most recent call last):
File "./src/train.py", line 281, in
tf.app.run()
File "/home/lsk/anaconda2/envs/tensorflow/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "./src/train.py", line 277, in main
train()
File "./src/train.py", line 122, in train
model = SqueezeDet(mc, FLAGS.gpu)
File "/home/lsk/my_tensorflow/squeezeDet/src/nets/squeezeDet.py", line 24, in init
self._add_forward_graph()
File "/home/lsk/my_tensorflow/squeezeDet/src/nets/squeezeDet.py", line 38, in _add_forward_graph
self.caffemodel_weight = joblib.load(mc.PRETRAINED_MODEL_PATH)
File "/home/lsk/anaconda2/envs/tensorflow/lib/python2.7/site-packages/joblib/numpy_pickle.py", line 459, in load
obj = unpickler.load()
File "/home/lsk/anaconda2/envs/tensorflow/lib/python2.7/pickle.py", line 864, in load
dispatchkey
File "/home/lsk/anaconda2/envs/tensorflow/lib/python2.7/pickle.py", line 1096, in load_global
klass = self.find_class(module, name)
File "/home/lsk/anaconda2/envs/tensorflow/lib/python2.7/pickle.py", line 1132, in find_class
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'NumpyArrayWrapper'

My tensorflow version is 0.12, easydict is 1.6 joblib is 0.10.3.
I can also run the demo code, have you solved your problem? @goan15910

[goan15910]

Hi, @samlong-yang
I made the pretrained-weights pkl file myself and it works.
I used this repository caffe-tensorflow to convert caffe model weight into npy file and dump it into pkl file. If you are going to use this repo, make sure not to transpose the channels as the original code of src/nn_skeleton.py, since the repo already switch them from caffe order into tensorflow order.

[skyz8421]

Thanks for your reply and suggestions very much! I will try for that.

[skyz8421]

Your solution works for me @goan15910 .
I find that there is a file in the project ./src/utils/caffemodel2pkl.py, which is also helpful for converting the caffemodel into pkl weight file. It also works fine to me.

[caocuong0306]

This is probably because you used different version of joblib 0.10.3.
I just had the same problem with @samlong-yang when using joblib 0.9.4. I updated to 0.10.3 and everything is fine.

[liuwei16]

Dear @samlong-yang how do you convert the squeezenet_v1.1.caffemodel to pkl by caffemodel2pkl.py?
should the prototxt be modified in some way?
Thanks