graph optimization fail

[chiyuzhang94]

This issue shows up:
I can successfully use this application on my MacBook. Then, I tried to apply it on a cloud server, but it cannot work and return error: FileNotFoundError: graph optimization fails and returns empty result. Could you please help me to figure out this issue? Thanks!
...

[hanxiao]

Please fill out the issue form, those version and env-info help me diagnose the issue faster.

Also please copy paste the entire server log instead of just the last line.

I'm guessing you don't have permission to write on the cloud server. That's why optimize graph failed.

[kuhung]

@chiyuzhang94
I met this problem when I try BERT-Base, Chinese and sovled it by redownload the pretrained model. Maybe your file is broken, have a MD5 check could help.

[hanxiao]

@chiyuzhang94 is this issue still there?

[chiyuzhang94]

@chiyuzhang94 is this issue still there?

Hi Han, Thanks for your attention and sorry for the inconveniences. It still doesn't work. I am using a virtual environment with python 3.6 on a cloud server and version 1.6.4 for bert-serving-server. I unzipped BERT-Large, Uncased in a fold call "uncased_L-24_H-1024_A-16", and created a shell file bert-server.sh : #!/bin/bash bert-serving-start -model_dir ~/py3.6/uncased_L-24_H-1024_A-16 -num_worker=8
Then, I used salloc to run this job: salloc --time=10:0:0 --ntasks=1 --cpus-per-task=8 --mem=16G --gres=gpu:1 --account=***** --job-name=bert-server srun $VIRTUAL_ENV/bert-server.sh

Finally, the termial return such errors:

[chiyuzhang94]

@chiyuzhang94
I met this problem when I try BERT-Base, Chinese and sovled it by redownload the pretrained model. Maybe your file is broken, have a MD5 check could help.
Thanks, I tried to use other models, it still didn't work. But any model can work on my local computer.

[saravananpsg]

@hanxiao Thank you for the repo.

Even I had the same issue however when I re-downloaded the recommended pre-trained model, I solved this error. Not sure why it occurred in the first place. Also, I have increased the number of workers to 8

bert-serving-start -model_dir /tmp/multi_cased_L-12_H-768_A-12 -num_worker=8

[hanxiao]

Could you please double check the python version on the cloud server? I found this line is particular suspicious.

invalid syntax (<string>, line 28)

which points to this place:
https://github.com/hanxiao/bert-as-service/blob/5e5bad07911dc44fd959dec75d7fb56bc7ac5f53/server/bert_serving/server/graph.py#L28-L30
This often indicates a wrong python version (2 instead of 3, or some other versions that don't appreciate @staticmethod in Enum); or some random typos in the code. We can rule out the latter if you are installing bert-serving-server via pip. But that said, if you are using py2, the server shouldn't be able to start in the first place.

[hanxiao]

@saravananpsg hmm, strange. I will add more logging there and print the full error traceback in the next version.

[chiyuzhang94]

Could you please double check the python version on the cloud server? I found this line is particular suspicious.

invalid syntax (<string>, line 28)

which points to this place:
bert-as-service/server/bert_serving/server/graph.py

Lines 28 to 30 in 5e5bad0

@staticmethod
def from_string(s):
try:

This often indicates a wrong python version (2 instead of 3, or some other versions that don't appreciate @staticmethod in Enum); or some random typos in the code. We can rule out the latter if you are installing bert-serving-server via pip. But that said, if you are using py2, the server shouldn't be able to start in the first place.

I checked my python version is 3.6.3.

[hanxiao]

thanks, I just included the traceback for the graph optimization part, could you update via

pip install -U bert-serving-server bert-serving-client

and give the new version a try? It wasn't for solving this bug, but for providing more details.

please copy paste the server log here, thanks

[chiyuzhang94]

Thanks! I updated the modules and tried again. Here is the error report.

[hanxiao]

Thanks gotcha. Now I see the invalid syntax is raised from single_image_random_dot_sterograms not from bert_as_service. After googling, I found it's a tensorflow problem, and this issue as well as this PR discussed and addressed the exact problem.

You may follow the instruction in those threads or google the solution by yourself, or post issue to the tensorflow repo.

I'm closing this issue as it's unrelated to bert-as-service.

[chiyuzhang94]

Thanks gotcha. Now I see the invalid syntax is raised from single_image_random_dot_sterograms not from bert_as_service. After googling, I found it's a tensorflow problem, and this issue as well as this PR discussed and addressed the exact problem.

You may follow the instruction in those threads or google the solution by yourself, or post issue to the tensorflow repo.

I'm closing this issue as it's unrelated to bert-as-service.

Sure. Thanks!

[PaulZhangIsing]

Thanks gotcha. Now I see the invalid syntax is raised from single_image_random_dot_sterograms not from bert_as_service. After googling, I found it's a tensorflow problem, and this issue as well as this PR discussed and addressed the exact problem.
You may follow the instruction in those threads or google the solution by yourself, or post issue to the tensorflow repo.
I'm closing this issue as it's unrelated to bert-as-service.

Sure. Thanks!

so may I know how did u manage to solve this issue??

[chenbaicheng]

@chiyuzhang94 which version of tensorflow are you using?

[chiyuzhang94]

@chiyuzhang94 which version of tensorflow are you using?

I am using Version: 1.12.0+{cloud service name}, but I think it is a special version for the cloud service.

[w6688j]

How is this problem solved?

[brandonchow1997]

How is this problem solved?

I got the same problem...Wondering how to solve it??

[hanxiao]

@w6688j @brandonchow1997 look at the error traceback, you can see that the MemoryError is thrown from tensorflow. So it's highly likely a tensorflow problem.

[chenbaicheng]

@hanxiao can give a advice ？ which tensorflow version can use for this project。。。thanks

[hanxiao]

requirements are already written in README chenbaicheng <notifications@github.com>于2019年1月24日 周四下午8:48写道：

…

@hanxiao <https://github.com/hanxiao> can give a advice ？ which tensorflow version can use for this project。。。thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#163 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB8l6i-W7a0wJ8e6r1R5S3C6h05yoTLmks5vGaubgaJpZM4Zhdys> .