Skip to content
This repository was archived by the owner on Nov 21, 2023. It is now read-only.

Bug 1347283 - Catch submission deserialization exceptions #116

Closed
wants to merge 1 commit into from
Closed

Bug 1347283 - Catch submission deserialization exceptions #116

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion moztelemetry/heka/message_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
import gzip
import logging
import ssl
import struct

Expand All @@ -15,6 +16,8 @@
from .message import Message, Header


logger = logging.getLogger(__name__)

def parse_heka_message(message):
try:
for record, total_bytes in unpack(message):
Expand All @@ -35,7 +38,10 @@ def _parse_heka_record(record):
# because it contains null values for JSON fields that have been
# split out.
if field.name == 'submission':
payload = _parse_json(field.value_bytes[0].decode('utf-8'))
try:
payload = _parse_json(field.value_bytes[0].decode('utf-8'))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also wrap the above call where we parse message.payload?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably don't need to do this, since message.payload is only ever populated by something that is guaranteed to be (escaped) UTF8.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It appears simply calling .decode('utf-8', 'ignore') may be the better solution. The test case to reproduce in the bug works and this would map more closely to what we're doing in scala.

except UnicodeDecodeError:
logger.exception("Error parsing 'submission' field")
break

if payload is None:
Expand Down