Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

object_detection API protoc installation step fails #4002

Closed
radzfoto opened this issue Apr 17, 2018 · 23 comments
Closed

object_detection API protoc installation step fails #4002

radzfoto opened this issue Apr 17, 2018 · 23 comments
Assignees

Comments

@radzfoto
Copy link

System information

  • What is the top-level directory of the model you are using: ~/tensorflow/models/research
  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): NO
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): v1.7.0-3-g024aecf414 1.7.0
  • Bazel version (if compiling from source): N/A
  • CUDA/cuDNN version: Cuda compilation tools, release 9.0, V9.0.176, CUDNN: 7.0.4.31-1+cuda9.0
  • GPU model and memory: nVidia Titan Xp, 12GB, driver 390.48
  • Exact command to reproduce: protoc object_detection/protos/*.proto --python_out=.

Describe the problem

I followed object_detection installation instructions step by step. When I tried protoc command, .proto files could not be compiled to .py files.

I issued this command:

protoc object_detection/protos/*.proto --python_out=.

And obtained these errors:

object_detection/protos/ssd.proto:87:3: Expected "required", "optional", or "repeated".
object_detection/protos/ssd.proto:87:12: Expected field name.
object_detection/protos/model.proto: Import "object_detection/protos/ssd.proto" was not found or had errors.
object_detection/protos/model.proto:12:5: "Ssd" is not defined.

Note that all the .proto files exist including ssd.proto.

Source code / logs

No custom source code. Just following object_detection API installation instructions

@he44
Copy link

he44 commented Apr 17, 2018

I encountered the same issue and I think it's because of the update/latest commit. I bypassed the protoc error by checking out an older version (2018-04-13).

git reset --hard 490813bdb3499290633919a9867eb0bb6d346d87 

(How to checkout previous commit: https://stackoverflow.com/questions/3555107/git-clone-particular-version-of-remote-repository/18037563 )

@denglixi
Copy link

Upgrade your protoc.

@derekjchow
Copy link
Contributor

@denglixi is correct. Update your protobuf compiler/runtime to 3.x.x

@radzfoto
Copy link
Author

@denglixi and @derekjchow, you were both correct. I removed the 2.6 protobuf-compiler and installed the 3.5.1 protobuf-compiler from github. What is frustrating is that the installation instructions in the object_detection api github still say to use the 2.6 version. I hope that this gets updated. Thank you!

@derekjchow
Copy link
Contributor

Will update the docs for next release

@lechatthecat
Copy link

related?
#4047

@lechatthecat
Copy link

I have protobuf 3.5.2 and still fail to compile. Why?

object_detection/protos/ssd.proto:87:3: Expected "required", "optional", or "repeated".
object_detection/protos/ssd.proto:87:12: Expected field name.
object_detection/protos/model.proto: Import "object_detection/protos/ssd.proto" was not found or had errors.
object_detection/protos/model.proto:12:5: "Ssd" is not defined.

@sandunadhikari
Copy link

Make sure you grab the latest version

curl -OL https://github.com/google/protobuf/releases/download/v3.2.0/protoc-3.2.0-linux-x86_64.zip

Unzip

unzip protoc-3.2.0-linux-x86_64.zip -d protoc3
change your protobuf to protobuf3

Move protoc to /usr/local/bin/

sudo mv protoc3/bin/* /usr/local/bin/

Move protoc3/include to /usr/local/include/

sudo mv protoc3/include/* /usr/local/include/

Optional: change owner

sudo chwon [user] /usr/local/bin/protoc
sudo chwon -R [user] /usr/local/include/google

@AliceDinh
Copy link

AliceDinh commented May 3, 2018

@radzfoto: please show how did you install the 3.5.1 protobuf-compiler from github please!
I grab the latest version from:
https://github.com/google/protobuf/releases/download/v3.2.0/protoc-3.2.0-linux-x86_64.zip
then I don't know what to do next

@radzfoto
Copy link
Author

radzfoto commented May 3, 2018

Hi @AliceDinh, here are some basic steps that may help to get you started:
To download version 3.5.1 you need to go here:
then download file protoc-3.5.1-osx-x86_64.zip. Unzip the file in some directory, let's say something like ~/protobuf/

If you are using Ubuntu or similar, then use the archive manager to unzip everything in the file, or install unzip. Make sure that you put the protoc zip file in your target directory, then with unzip in a terminal window

$ cd ~/protobuf
$ sudo apt-get install unzip
$ unzip protoc-3.5.1-osx-x86_64.zip

You will now see a readme.txt file and two directories, bin and include. The protobuf compiler is a file in the bin directory called protoc. It doesn't hurt to make sure that protoc is executable, so do

$ cd bin
$ chmod +x protoc

Now you could add this directory to your PATH so you can execute from anywhere, for example:

$ export PATH=$PATH:~/protobuf/bin

If you want this to be permanent, add that line to your .bashrc file

To use the protobuf compiler now, all you have to do is

$ protoc /path/to/the/protobuf/file/you/want/to/compile

I hope this helps.

@Thaiph1308
Copy link

@radzfoto
I follow you but only 3.3 version have bin and include folder, and when I using protoc --version , it still returns
libprotoc 2.6.1
so I still got the same issue like #4223.
It's mean that I failed when installing new protoc right?
Is there any way I can check I follow exactly what you say?
And how can I install protobuf 3.5.1 version (newest version)? It's doesn't contain bin and include folders.
I tried install like @sandunadhikari but it still return libprotoc 2.6.1
i'm using ubuntu on Window 10.
Thank for your help.

@radzfoto
Copy link
Author

@Thaiph1308, if you download the protoc zip file, it should have everything inside. Please make sure you are downloading the correct file and unzipping it correctly. It is very important that you then execute this new version of protoc. You can make sure by putting the directory in the PATH environment variable. And make sure that you remove any other versions of protoc.
I am very confused about your comment that you are using "ubuntu on Windows 10". What does that mean? Are you using Ubuntu Linux? Or are your using Windows 10? If you are using a virtualized version of Ubuntu running on something like VirtualBox on Windows, then you are just using Ubuntu.

@huyutong
Copy link

@radzfoto
I use "which protoc ,sudo rm /usr/local/bin/protoc" , uninstall the protoc2.6.1 , then follow the way you give as above to install protoc3.2.0. when I run " protoc object_detection/protos/*.proto --python_out=." ,it shows "sudo apt install protobuf-compiler", but even if I run ''sudo apt install protobuf-compiler'', it still does not work, it means the protoc was't installed correctly. so,do u have any idea about this ? thans very much

@radzfoto
Copy link
Author

Hi @huyutong, if you read my suggested procedure more carefully, I suggest that you do NOT use apt to install the protobuf compiler. Instead go directly to the download website here:
https://developers.google.com/protocol-buffers/docs/downloads
And then click on the "release page" to select the version for your operating system. Once you download the zip file, just unzip it to a folder, look for the "bin" subdirectory and you will find the protoc executable there. Put that in your PATH and you should be all set.

@huyutong
Copy link

@radzfoto
I have followed the way you given, but it still shows ''protoc is not installed,you can install by sudo apt install protobuf-compiler'' ,then I tried ''sudo apt install protobuf-compiler'', it didnot work

@Thaiph1308
Copy link

Thaiph1308 commented May 15, 2018

@radzfoto
Thanks for your help, I have successfully installed protoc.
But I have some concern about file protobuf-python-3.5.1.tar.gz. It's used in Window OS right? How can we know a file is used for what OS?
And I'm using Ubuntu from Microsoft Store, direct Ubuntu from Window.
@huyutong
What OS did u use? If Ubuntu, u have to install some library (make,gcc ...) to compile protobuf-compiler before use it.
More details here https://github.com/google/protobuf/blob/master/src/README.md

@edisonchensy
Copy link

I got the same problem, please give me a help. my protobuf is 3.5.2 version.

edison@amax-server:/google_TF_object_detection/models/research$ protoc object_detection/protos/*.proto --python_out=.
object_detection/protos/ssd.proto:87:3: Expected "required", "optional", or "repeated".
object_detection/protos/ssd.proto:87:12: Expected field name.
object_detection/protos/model.proto: Import "object_detection/protos/ssd.proto" was not found or had errors.
object_detection/protos/model.proto:12:5: "Ssd" is not defined.
edison@amax-server:
/google_TF_object_detection/models/research$ pip install protobuf
Requirement already satisfied: protobuf in /home/edison/anaconda3/lib/python3.6/site-packages (3.5.2.post1)
Requirement already satisfied: setuptools in /home/edison/anaconda3/lib/python3.6/site-packages (from protobuf) (39.0.1)
Requirement already satisfied: six>=1.9 in /home/edison/anaconda3/lib/python3.6/site-packages (from protobuf) (1.11.0)

@ETucks
Copy link

ETucks commented Jun 16, 2018

I had the same problem, and attempted resolve things by updating to protobuf 3.5.2, but was unsuccessful.
However, commmenting out line 87 of object_detection/protos/ssd.proto resolved the issue (As mentioned by mastash3ff in issue #4047 : )

@HyperWayne
Copy link

Following @sandunadhikari steps, I check my protoc version, it's still 2.6.1 (--version).
After

source ~/.bashrc

error message for this compilation disappear.

protoc object_detection/protos/*.proto --python_out=.

Update version of protoc to 3.2.0 works for me.

@dogod621
Copy link

try this pyscript... change the researchRoot for yourself.
It works perfectly for me ( I use protobuf 3.6 ).
you can execute the script at any folder.

import sys
import os
import subprocess

researchRoot = os.path.normpath( "D:/Program Files (x86)/Microsoft Visual Studio/Shared/Python36_64/Lib/site-packages/tensorflow/models/research/" )
protoRoots_ = os.listdir(researchRoot)
protoRoots = []

for protoRoot_ in protoRoots_:
    if ( protoRoot_ in protoRoots ):
        continue
    
    for root, dirs, files in os.walk(os.path.join(researchRoot, protoRoot_)):
        if ( protoRoot_ in protoRoots ):
            break
        
        for name in files:
            if ( os.path.splitext(name)[1] == ".proto" ):
                protoRoots.append( protoRoot_ )
                break


for protoRoot in protoRoots:
    for root, dirs, files in os.walk(os.path.join(researchRoot, protoRoot)):
        for name in files:
            if ( os.path.splitext(name)[1] == ".proto" ):
                cmdStr = \
                        "protoc " + name + \
                        " --proto_path=\"" + root + "\" " + \
                        " --proto_path=\"" + os.path.join(researchRoot, protoRoot) + "\" " + \
                        " --proto_path=\"" + researchRoot + "\" " + \
                        "--python_out=."
                    
                print(cmdStr)
                
                with subprocess.Popen(cmdStr, stdout=subprocess.PIPE, shell=True, cwd=researchRoot) as proc:
                    stdout, stderr = proc.communicate()
                    cmdExit = proc.returncode

                    if ( cmdExit != 0 ):
                        print ("cmdExit == " + str(cmdExit) )
                        print ("stdout")
                        print (stdout)
                        print ("stderr")
                        print (stderr)
                    else:  
                        print ("cmdExit == 0")

@bozcani
Copy link

bozcani commented May 1, 2020

I downloaded and installed protoc from source, and it worked for me:

# You should be in folder tensorflow/models/research/
$ wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
$ unzip protobuf.zip
$ ./bin/protoc object_detection/protos/*.proto --python_out=.

Then, test the installation:

python object_detection/builders/model_builder_test.py

@upasana-mittal
Copy link

I downloaded and installed protoc from source, and it worked for me:

# You should be in folder tensorflow/models/research/
$ wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
$ unzip protobuf.zip
$ ./bin/protoc object_detection/protos/*.proto --python_out=.

Then, test the installation:

python object_detection/builders/model_builder_test.py

It is not working for me @bozcani
I tried with the version you have given 3.0.0 also with 3.4.0 but it gave me same error

object_detection/protos/*.proto: No such file or directory

Can you please help?

@AliceDinh
Copy link

@upasana-mittal
Try to run these:
protoc object_detection/protos/anchor_generator.proto --python_out=.
protoc object_detection/protos/argmax_matcher.proto --python_out=.
protoc object_detection/protos/bipartite_matcher.proto --python_out=.
protoc object_detection/protos/box_coder.proto --python_out=.
protoc object_detection/protos/box_predictor.proto --python_out=.
protoc object_detection/protos/eval.proto --python_out=.
protoc object_detection/protos/faster_rcnn.proto --python_out=.
protoc object_detection/protos/faster_rcnn_box_coder.proto --python_out=.
protoc object_detection/protos/grid_anchor_generator.proto --python_out=.
protoc object_detection/protos/hyperparams.proto --python_out=.
protoc object_detection/protos/image_resizer.proto --python_out=.
protoc object_detection/protos/input_reader.proto --python_out=.
protoc object_detection/protos/keypoint_box_coder.proto --python_out=.
protoc object_detection/protos/losses.proto --python_out=.
protoc object_detection/protos/matcher.proto --python_out=.
protoc object_detection/protos/mean_stddev_box_coder.proto --python_out=.
protoc object_detection/protos/model.proto --python_out=.
protoc object_detection/protos/multiscale_anchor_generator.proto --python_out=.
protoc object_detection/protos/optimizer.proto --python_out=.
protoc object_detection/protos/pipeline.proto --python_out=.
protoc object_detection/protos/post_processing.proto --python_out=.
protoc object_detection/protos/preprocessor.proto --python_out=.
protoc object_detection/protos/region_similarity_calculator.proto --python_out=.
protoc object_detection/protos/square_box_coder.proto --python_out=.
protoc object_detection/protos/ssd.proto --python_out=.
protoc object_detection/protos/ssd_anchor_generator.proto --python_out=.
protoc object_detection/protos/string_int_label_map.proto --python_out=.
protoc object_detection/protos/train.proto --python_out=.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests