New 'MessageToDict' implementation breaks with unknown entry types #2674

tseaver · 2016-11-03T17:38:36Z

>>> from google.cloud.logging import Client
>>> client = Client()
>>> def do_something_with(entry): pass
... 
>>> for entry in client.list_entries():
...     do_something_with(entry)
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/cloud/iterator.py", line 211, in _items_iter
    for item in page:
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/cloud/iterator.py", line 155, in next
    result = self._item_to_value(self._parent, item)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/cloud/logging/_gax.py", line 478, in _item_to_entry
    resource = MessageToDict(entry_pb)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 133, in MessageToDict
    return printer._MessageToJsonObject(message)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 164, in _MessageToJsonObject
    return self._RegularMessageToJsonObject(message, js)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 196, in _RegularMessageToJsonObject
    js[name] = self._FieldToJsonObject(field, value)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 230, in _FieldToJsonObject
    return self._MessageToJsonObject(value)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 162, in _MessageToJsonObject
    return methodcaller(_WKTJSONMETHODS[full_name][0], message)(self)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 266, in _AnyMessageToJsonObject
    sub_message = _CreateMessageFromTypeUrl(type_url)
  File "/home/tseaver/projects/agendaless/Google/src/google-cloud-python/.tox/docs/lib/python2.7/site-packages/google/protobuf/json_format.py", line 341, in _CreateMessageFromTypeUrl
    'Can not find message descriptor by type_url: {0}.'.format(type_url))
TypeError: Can not find message descriptor by type_url: type.googleapis.com/google.cloud.audit.AuditLog.

dhermes · 2016-11-03T17:43:28Z

@tseaver Yup, @bjwatson had reported this earlier and I confirmed with @waprin yesterday. Sorry for not filing. I'd been meaning to resolve this.

bjwatson · 2016-11-03T17:50:33Z

@tseaver @dhermes We're tracking this in https://github.com/googleapis/googleapis/issues/187.

What's the relative priority? We're focused on landing our beta surfaces this month, and the related issue #2572 took me down a long tangential path for the past week.

dhermes · 2016-11-03T19:10:31Z

@bjwatson This is something that's my fault, not your fault. There isn't anything you need to fix, we'll always encounter types we don't know about. The proper behavior is just to leave them as Any instances, and I'll make that fix in google-cloud-python

tseaver · 2016-11-03T19:14:11Z

@bjwatson I can exercise the new logging snippets with GRPC disabled for now.

bjwatson · 2016-11-03T19:26:53Z

Ok, thanks @dhermes and @tseaver.

salrashid123 · 2017-02-06T20:31:00Z

ran into this issue as well. Just to document, you can disable gRPC using

os.environ["GOOGLE_CLOUD_DISABLE_GRPC"] = "true"

dhermes · 2017-03-31T20:08:12Z

I may just end up reverting #2600.

dhermes · 2017-03-31T20:41:05Z

It's fairly relevant that the only Any in the logging protobufs is LogEntry.proto_payload (which is part of the payload oneof)

Currently digging in to a nice repro provided by @Fkawala, I made it totally distinct of this library here in a gist.

This is temporary as googleapis#2674 is still being sorted out. This way we don't block an umbrella release or the release of error reporting.

…stry. Fixes googleapis#2674.

…stry. (#3270) * Gracefully continue if LogEntry.proto_payload type URL is not in registry. Fixes #2674. * Adding unit / system tests for "graceful" continuation on LogEntry parsing. * Docs fix for PR 3270. * Adding rtype/returns to metric_update and sink_update in logging _http module. * Add trailing commas.

mikelambert · 2017-04-21T01:02:58Z

I just started looking into logging google.appengine.logging.v1.RequestLog, so am happy to see this has been fixed. What's the best way to use this?

I assume that installing google-cloud-logging 1.0.0 won't work if I want to use other google-cloud-* libraries too, due to conflicts on google.cloud.logging package? (Or am I misunderstanding how they interoperate)

So instead, I assume I'm waiting on any version of google-cloud that is post-0.24?

dhermes · 2017-04-21T02:26:46Z

Installing google-cloud-logging will work just fine with other packages. For most usage, you can just ignore the google-cloud "uber"-package and focus on the services you actually want to use.

The only place to worry about a version collision is in google-cloud-core, since each package google-cloud-FOO depends on google-cloud-core.

This is temporary as googleapis#2674 is still being sorted out. This way we don't block an umbrella release or the release of error reporting.

…stry. (googleapis#3270) * Gracefully continue if LogEntry.proto_payload type URL is not in registry. Fixes googleapis#2674. * Adding unit / system tests for "graceful" continuation on LogEntry parsing. * Docs fix for PR 3270. * Adding rtype/returns to metric_update and sink_update in logging _http module. * Add trailing commas.

eyalev · 2017-08-29T05:42:02Z

When I try to run the code here: https://gist.github.com/Fkawala/654093301c47241386b5cd4e02791e56#file-do_log-py

I get an error:

...
...

  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/logger.py", line 302, in log_proto
    client.logging_api.write_entries([entry_resource])
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/_gax.py", line 127, in write_entries
    entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/_gax.py", line 499, in _log_entry_mapping_to_pb
    ParseDict(mapping, entry_pb)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 402, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 433, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 539, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse protoPayload field: Can not find message descriptor by type_url: type.googleapis.com/google.appengine.logging.v1.RequestLog..

Does it suppose to work? Or is this feature deprecated?
I couldn't fully understand from the conversation..

yan-hic · 2018-05-09T13:10:09Z

Unclear if this should be reopened or is a new issue but I get same error when trying to parse a Protobuf payload written by GCP services: json_format.MessageToJson(payload_pb)

It seems the only way to consume GCP logging outside of the UI is to export to bigquery sink and query from there. Or is logging only meant to work with a logger ?

Btw, disabling gRPC has no effect in this case.

tseaver · 2018-05-09T20:48:29Z

@yiga2 Looks like you opened #5313 to track the parsing issue for protobuf payloads.

yan-hic · 2018-05-09T21:58:59Z

Related but #5313 is to request more documentation whereas here is to report an error with MessageToJson.

If the documentation highlights a different method to use, I agree my present comment is not applicable.

tseaver · 2018-05-10T19:33:04Z

@yiga2 json_format.MessageToJson is defined in the protobuf package: please report issues with formatting your particular message type there (I see a couple of open issues related to that function already).

snarfed · 2019-12-19T18:01:59Z

as an example, here's how i generated my own request_log_pb2.py file based on request_log.prototo parse google.appengine.logging.v1.RequestLog serialized protobufs: https://stackoverflow.com/a/59414384/186123. obviously not ideal, i'd rather use an official version maintained by Google instead, but this is an ok workaround in the meantime.

(this also sounds like @mikelambert's use case in #2674 (comment).

yeruva7 · 2021-03-03T21:52:51Z

When I try to run the code here: https://gist.github.com/Fkawala/654093301c47241386b5cd4e02791e56#file-do_log-py

I get an error:

...
...

  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/logger.py", line 302, in log_proto
    client.logging_api.write_entries([entry_resource])
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/_gax.py", line 127, in write_entries
    entry_pbs = [_log_entry_mapping_to_pb(entry) for entry in entries]
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/logging/_gax.py", line 499, in _log_entry_mapping_to_pb
    ParseDict(mapping, entry_pb)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 402, in ParseDict
    parser.ConvertMessage(js_dict, message)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 433, in ConvertMessage
    self._ConvertFieldValuePair(value, message)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 539, in _ConvertFieldValuePair
    raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
google.protobuf.json_format.ParseError: Failed to parse protoPayload field: Can not find message descriptor by type_url: type.googleapis.com/google.appengine.logging.v1.RequestLog..

Does it suppose to work? Or is this feature deprecated?
I couldn't fully understand from the conversation..

@tseaver Are you able to fix this issue?

yeruva7 · 2021-03-03T21:54:53Z

Need some help for me, I am facing same issue

tseaver added api: logging Issues related to the Cloud Logging API. grpc labels Nov 3, 2016

dhermes mentioned this issue Nov 14, 2016

Upgrading core to version to 0.21.0. #2733

Merged

danoscarmike added this to Logging in First 4 (GA) Feb 22, 2017

dhermes mentioned this issue Mar 31, 2017

Logging seems to not support type.googleapis.com/google.cloud.audit.AuditLog. #3218

Closed

dhermes added the priority: p2 Moderately-important priority. Fix may not be included in next release. label Mar 31, 2017

dhermes mentioned this issue Mar 31, 2017

Missing protobuf descriptor: google.appengine.logging.v1.RequestLog #2572

Closed

dhermes added the status: acknowledged label Mar 31, 2017

bjwatson added the type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. label Apr 1, 2017

dhermes added a commit to dhermes/google-cloud-python that referenced this issue Apr 1, 2017

Downgrading logging from 1.0.0 to 0.24.0.

68a35e1

This is temporary as googleapis#2674 is still being sorted out. This way we don't block an umbrella release or the release of error reporting.

dhermes mentioned this issue Apr 1, 2017

Downgrading logging from 1.0.0 to 0.24.0. #3261

Merged

dhermes added a commit to dhermes/google-cloud-python that referenced this issue Apr 3, 2017

Gracefully continue if LogEntry.proto_payload type URL is not in regi…

6684ddd

…stry. Fixes googleapis#2674.

dhermes mentioned this issue Apr 3, 2017

Gracefully continue if LogEntry.proto_payload type URL is not in registry. #3270

Merged

dhermes closed this as completed in #3270 Apr 6, 2017

richkadel pushed a commit to richkadel/google-cloud-python that referenced this issue May 6, 2017

Downgrading logging from 1.0.0 to 0.24.0.

7464d53

This is temporary as googleapis#2674 is still being sorted out. This way we don't block an umbrella release or the release of error reporting.

JustinBeckwith assigned dhermes Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New 'MessageToDict' implementation breaks with unknown entry types #2674

New 'MessageToDict' implementation breaks with unknown entry types #2674

tseaver commented Nov 3, 2016

dhermes commented Nov 3, 2016

bjwatson commented Nov 3, 2016

dhermes commented Nov 3, 2016

tseaver commented Nov 3, 2016

bjwatson commented Nov 3, 2016

salrashid123 commented Feb 6, 2017

dhermes commented Mar 31, 2017

dhermes commented Mar 31, 2017 •

edited

mikelambert commented Apr 21, 2017

dhermes commented Apr 21, 2017

eyalev commented Aug 29, 2017

yan-hic commented May 9, 2018 •

edited

tseaver commented May 9, 2018

yan-hic commented May 9, 2018

tseaver commented May 10, 2018

snarfed commented Dec 19, 2019 •

edited

yeruva7 commented Mar 3, 2021

yeruva7 commented Mar 3, 2021

New 'MessageToDict' implementation breaks with unknown entry types #2674

New 'MessageToDict' implementation breaks with unknown entry types #2674

Comments

tseaver commented Nov 3, 2016

dhermes commented Nov 3, 2016

bjwatson commented Nov 3, 2016

dhermes commented Nov 3, 2016

tseaver commented Nov 3, 2016

bjwatson commented Nov 3, 2016

salrashid123 commented Feb 6, 2017

dhermes commented Mar 31, 2017

dhermes commented Mar 31, 2017 • edited

mikelambert commented Apr 21, 2017

dhermes commented Apr 21, 2017

eyalev commented Aug 29, 2017

yan-hic commented May 9, 2018 • edited

tseaver commented May 9, 2018

yan-hic commented May 9, 2018

tseaver commented May 10, 2018

snarfed commented Dec 19, 2019 • edited

yeruva7 commented Mar 3, 2021

yeruva7 commented Mar 3, 2021

dhermes commented Mar 31, 2017 •

edited

yan-hic commented May 9, 2018 •

edited

snarfed commented Dec 19, 2019 •

edited