AttributeError: 'SymbolDatabase' object has no attribute 'RegisterServiceDescriptor'

[n-a-s-s]

System information:

• Google cloud shell (debian linux)

• tensorflow==1.4.1

• tensorflow-tensorboard==0.4.0rc3

• tensorflow-transform==0.1.10

• protobuf==3.4.0

• Python 2.7.9

• Exact command to reproduce:
  JOB_ID="smallclicks_linear_$(date +%Y%m%d_%H%M%S)"
  gcloud ml-engine jobs submit training "$JOB_ID"
  --module-name trainer.task
  --package-path trainer
  --staging-bucket "$BUCKET"
  --region us-central1
  --config config-small.yaml
  --async
  --
  --dataset kaggle
  --model_type linear
  --l2_regularization 100
  --output_path "${GCS_PATH_SMALL}/model/${JOB_ID}"
  --raw_metadata_path "${PREPROCESS_OUTPUT_SMALL}/raw_metadata"
  --transformed_metadata_path "${PREPROCESS_OUTPUT_SMALL}/transformed_metadata"
  --transform_savedmodel "${PREPROCESS_OUTPUT_SMALL}/transform_fn"
  --eval_data_paths "${PREPROCESS_OUTPUT_SMALL}/features_eval*"
  --train_data_paths "${PREPROCESS_OUTPUT_SMALL}/features_train*"

• Tensorflow Transform environment (if applicable, see below):

Describe the problem:
The jobs fail with the:

• AttributeError: 'SymbolDatabase' object has no attribute 'RegisterServiceDescriptor'
  error, I did use the protobf version 3.4.0 which is suggeested in this "AttributeError: 'SymbolDatabase' object has no attribute 'RegisterServiceDescriptor'" #99
• I also used a virtualenv but still jobs failed

code / logs:
The replica master 0 exited with a non-zero status of 1. Termination reason: Error. Traceback (most recent call last): [...] File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 3147, in @_call_aside File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 3131, in _call_aside f(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 3175, in _initialize_master_working_set for dist in working_set File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 3175, in for dist in working_set File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 2701, in activate declare_namespace(pkg) File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 2260, in declare_namespace _handle_ns(packageName, path_item) File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 2195, in _handle_ns loader.load_module(packageName) File "/usr/lib/python2.7/pkgutil.py", line 246, in load_module mod = imp.load_module(fullname, self.file, self.filename, self.etc) File "/root/.local/lib/python2.7/site-packages/google/cloud/pubsub/init.py", line 30, in from google.cloud.pubsub.client import Client File "/root/.local/lib/python2.7/site-packages/google/cloud/pubsub/client.py", line 29, in from google.cloud.pubsub._gax import _PublisherAPI as GAXPublisherAPI File "/root/.local/lib/python2.7/site-packages/google/cloud/pubsub/_gax.py", line 19, in from google.cloud.gapic.pubsub.v1.publisher_client import PublisherClient File "/root/.local/lib/python2.7/site-packages/google/cloud/gapic/pubsub/v1/publisher_client.py", line 37, in from google.iam.v1 import iam_policy_pb2 File "/root/.local/lib/python2.7/site-packages/google/iam/v1/iam_policy_pb2.py", line 296, in _sym_db.RegisterServiceDescriptor(_IAMPOLICY) AttributeError: 'SymbolDatabase' object has no attribute 'RegisterServiceDescriptor' The replica worker 2 exited with a non-zero status of 1. Termination reason: Error. Traceback (most recent call last): [...] File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 3147, in @_call_aside File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 3131, in _call_aside f(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 3175, in _initialize_master_working_set for dist in working_set File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 3175, in for dist in working_set File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 2701, in activate declare_namespace(pkg) File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 2260, in declare_namespace _handle_ns(packageName, path_item) File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 2195, in _handle_ns loader.load_module(packageName) File "/usr/lib/python2.7/pkgutil.py", line 246, in load_module mod = imp.load_module(fullname, self.file, self.filename, self.etc) File "/root/.local/lib/python2.7/site-packages/google/cloud/pubsub/init.py", line 30, in from google.cloud.pubsub.client import Client File "/root/.local/lib/python2.7/site-packages/google/cloud/pubsub/client.py", line 29, in from google.cloud.pubsub._gax import _PublisherAPI as GAXPublisherAPI File "/root/.local/lib/python2.7/site-packages/google/cloud/pubsub/_gax.py", line 19, in from google.cloud.gapic.pubsub.v1.publisher_client import PublisherClient File "/root/.local/lib/python2.7/site-packages/google/cloud/gapic/pubsub/v1/publisher_client.py", line 37, in from google.iam.v1 import iam_policy_pb2 File "/root/.local/lib/python2.7/site-packages/google/iam/v1/iam_policy_pb2.py", line 296, in _sym_db.RegisterServiceDescriptor(_IAMPOLICY) AttributeError: 'SymbolDatabase' object has no attribute 'RegisterServiceDescriptor'

[andrewm4894]

I'm getting this as well when running the reddit_tft example.

My google-cloud-ml job id is "reddit_comments_linear_20180124_150717"

[andrewm4894]

p.s. looks like this might be related and may be some workarounds - just not sure how to apply them here.

googleapis/google-cloud-python#3967

[puneith]

@andrewm4894 That definitely looks like its related. Can you please confirm that you used the versions specified in the setup.py and requirements.txt?

[n-a-s-s]

@puneith I use exactly this and it worked

requirement.txt:
tensorflow==1.3.0
tensorflow-transform==0.3.1
protobuf==3.4.0

setup.py:
import setuptools

NAME = 'trainer'
VERSION = '1.0'
TENSORFLOW_TRANSFORM = 'tensorflow-transform==0.3.1'
PROTOBUF = 'protobuf==3.4.0'
TENSORFLOW = 'tensorflow==1.3.0'
if name == 'main':
setuptools.setup(name=NAME, version=VERSION, packages=['trainer'],
install_requires=[
TENSORFLOW_TRANSFORM,
PROTOBUF,
TENSORFLOW
])

[andrewm4894]

Hmm I'll try with that on Monday to be sure, think might have been tf 1.4

…

On Fri, 26 Jan 2018, 19:12 n-a-s-s, ***@***.***> wrote: @puneith <https://github.com/puneith> I use exactly this and it worked requirement.txt: tensorflow==1.3.0 tensorflow-transform==0.3.1 protobuf==3.4.0 setup.py: import setuptools NAME = 'trainer' VERSION = '1.0' TENSORFLOW_TRANSFORM = 'tensorflow-transform==0.3.1' PROTOBUF = 'protobuf==3.4.0' TENSORFLOW = 'tensorflow==1.3.0' if *name* == '*main*': setuptools.setup(name=NAME, version=VERSION, packages=['trainer'], install_requires=[ TENSORFLOW_TRANSFORM, PROTOBUF, TENSORFLOW ]) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#146 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACE89O_Ovv1A0ZUPhMXEuH1zcTwTgCK-ks5tOiOOgaJpZM4RZd1O> .

[puneith]

@n-a-s-s Thanks that's good to know. @andrewm4894 I will close this issue for now and if things don't work for you feel free to reopen or file another issue.

[andrewm4894]

@puneith will try now by changing this file. I'm running it on cloud-ml so i think maybe the setup.py might need to be changed in the sample to reflect what worked for @n-a-s-s

https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/reddit_tft/setup.py

Just in case is something others might run into.