How to convert binaryproto to npy (like ilsvrc_2012_mean.npy)? #290

Closed
ShiqiYu opened this Issue Apr 3, 2014 · 9 comments

Comments

Projects
None yet
6 participants
@ShiqiYu

ShiqiYu commented Apr 3, 2014

When I run the pretrained ImageNet model, I found the python wrapper (wrapper.py) will read this file:
python/caffe/imagenet/ilsvrc_2012_mean.npy
To my understanding, the file is the numpy version of file "imagenet_mean.binaryproto".

  1. Do anyone know how to convert "imagenet_mean.binaryproto" to "ilsvrc_2012_mean.npy"?
  2. If the mean file ("ilsvrc_2012_mean.npy") is a parameter in imagenet_deploy.prototxt, will it be better?

Thank you!

@shelhamer shelhamer added the question label Apr 3, 2014

@shelhamer shelhamer changed the title from about file ilsvrc_2012_mean.npy to How to convert binaryproto to npy (like ilsvrc_2012_mean.npy)? Apr 3, 2014

@shelhamer

This comment has been minimized.

Show comment
Hide comment
@shelhamer

shelhamer Apr 3, 2014

Member

1: For binaryproto/numpy conversion see convert.py and its utility functions. Load the binaryproto in python, yielding a blob, then call blobproto_to_array, and save the numpy array however you like (for instance as a npy).

2: The mean would ideally be embedded in the net definition, so that it doesn't need to be carried around elsewhere. There is ongoing work #148 #244 and a planned change to a channel mean that could be easily written in prototxt (that is, each channel would have a space invariant mean value, which has comparable performance to a full image mean and simplifies processing).

Member

shelhamer commented Apr 3, 2014

1: For binaryproto/numpy conversion see convert.py and its utility functions. Load the binaryproto in python, yielding a blob, then call blobproto_to_array, and save the numpy array however you like (for instance as a npy).

2: The mean would ideally be embedded in the net definition, so that it doesn't need to be carried around elsewhere. There is ongoing work #148 #244 and a planned change to a channel mean that could be easily written in prototxt (that is, each channel would have a space invariant mean value, which has comparable performance to a full image mean and simplifies processing).

@ShiqiYu

This comment has been minimized.

Show comment
Hide comment
@ShiqiYu

ShiqiYu Apr 4, 2014

Dear Evan Shelhamer,

Thank you for you help! I have read convert.py and found that the function 'blobproto_to_array'. To convert the format, I think it should be:
Step1: dat = to_load_datum('imagenet_mean.binaryproto')
Step2: arr = blobproto_to_array(dat)
Step3: ny.save('ilsvrc_2012_mean.npy', arr)
Because I'm not familiar with datum and python enough, I don't know how to load the 'imagenet_mean.binaryproto' file. Could you kindly give me some lines of code for doing that? Thank you.

ShiqiYu commented Apr 4, 2014

Dear Evan Shelhamer,

Thank you for you help! I have read convert.py and found that the function 'blobproto_to_array'. To convert the format, I think it should be:
Step1: dat = to_load_datum('imagenet_mean.binaryproto')
Step2: arr = blobproto_to_array(dat)
Step3: ny.save('ilsvrc_2012_mean.npy', arr)
Because I'm not familiar with datum and python enough, I don't know how to load the 'imagenet_mean.binaryproto' file. Could you kindly give me some lines of code for doing that? Thank you.

@shelhamer

This comment has been minimized.

Show comment
Hide comment
@shelhamer

shelhamer Apr 26, 2014

Member

The binaryproto file is a protobuf for BlobProto. Refer to the protobuf python tutorial and do from caffe.proto import caffe_pb2 to make a blob = caffe_pb2.BlobProto().

Member

shelhamer commented Apr 26, 2014

The binaryproto file is a protobuf for BlobProto. Refer to the protobuf python tutorial and do from caffe.proto import caffe_pb2 to make a blob = caffe_pb2.BlobProto().

@shelhamer shelhamer closed this Apr 26, 2014

@chrislybaer

This comment has been minimized.

Show comment
Hide comment
@chrislybaer

chrislybaer Jul 15, 2014

As a tip for people who are reading this now, the file: convert.py does not exist anymore, instead it's function blobproto_to_array moved into the caffe.io module.

As a tip for people who are reading this now, the file: convert.py does not exist anymore, instead it's function blobproto_to_array moved into the caffe.io module.

@HoldenCaulfieldRye

This comment has been minimized.

Show comment
Hide comment
@HoldenCaulfieldRye

HoldenCaulfieldRye Aug 24, 2014

I think @mezn means the caffe/python/caffe/io.py module

I think @mezn means the caffe/python/caffe/io.py module

@sg90

This comment has been minimized.

Show comment
Hide comment
@sg90

sg90 Nov 13, 2014

#808

import caffe
import numpy as np
import sys

if len(sys.argv) != 3:
print "Usage: python convert_protomean.py proto.mean out.npy"
sys.exit()

blob = caffe.proto.caffe_pb2.BlobProto()
data = open( sys.argv[1] , 'rb' ).read()
blob.ParseFromString(data)
arr = np.array( caffe.io.blobproto_to_array(blob) )
out = arr[0]
np.save( sys.argv[2] , out )

sg90 commented Nov 13, 2014

#808

import caffe
import numpy as np
import sys

if len(sys.argv) != 3:
print "Usage: python convert_protomean.py proto.mean out.npy"
sys.exit()

blob = caffe.proto.caffe_pb2.BlobProto()
data = open( sys.argv[1] , 'rb' ).read()
blob.ParseFromString(data)
arr = np.array( caffe.io.blobproto_to_array(blob) )
out = arr[0]
np.save( sys.argv[2] , out )
@RafaRuiz

This comment has been minimized.

Show comment
Hide comment
@RafaRuiz

RafaRuiz Jun 10, 2015

where are all these files? 404 or doesn't exist now

where are all these files? 404 or doesn't exist now

@chrislybaer

This comment has been minimized.

Show comment
Hide comment
@chrislybaer

chrislybaer Jun 13, 2015

The utility functions to read protofiles and to write them into other formats are in the io module. You can find the code here:
CPP:
https://github.com/BVLC/caffe/blob/master/src/caffe/util/io.cpp
Python:
https://github.com/BVLC/caffe/blob/master/python/caffe/io.py

The utility functions to read protofiles and to write them into other formats are in the io module. You can find the code here:
CPP:
https://github.com/BVLC/caffe/blob/master/src/caffe/util/io.cpp
Python:
https://github.com/BVLC/caffe/blob/master/python/caffe/io.py

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 14, 2015

Great thanks to @sg90 for sharing such a nice snippet 👍

ghost commented Sep 14, 2015

Great thanks to @sg90 for sharing such a nice snippet 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment