Fix telemetry unicode errors #1937

larohra · 2020-07-07T19:28:53Z

Description

This PR mainly deals with ensuring that the telemetry events sent by the agent are encoded properly in UTF-8 for both Py2 and Py3. This is to support special characters in telemetry.

Issue #

PR information

The title of the PR is clear and informative.
There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
If applicable, the PR references the bug/issue that it fixes in the description.
New Unit tests were added for the changes made and Travis.CI is passing.

Quality of Code and Contribution Guidelines

I have read the contribution guidelines.

codecov · 2020-07-07T19:46:18Z

Codecov Report

Merging #1937 into develop will increase coverage by 0.01%.
The diff coverage is 79.26%.

@@             Coverage Diff             @@
##           develop    #1937      +/-   ##
===========================================
+ Coverage    69.65%   69.66%   +0.01%     
===========================================
  Files           85       85              
  Lines        11925    11983      +58     
  Branches      1666     1676      +10     
===========================================
+ Hits          8306     8348      +42     
- Misses        3251     3263      +12     
- Partials       368      372       +4

Impacted Files	Coverage Δ
azurelinuxagent/common/utils/restutil.py	`90.76% <ø> (ø)`
azurelinuxagent/common/utils/textutil.py	`61.53% <53.84%> (-0.49%)`	⬇️
azurelinuxagent/common/protocol/wire.py	`76.56% <80.00%> (-0.44%)`	⬇️
azurelinuxagent/ga/monitor.py	`77.96% <87.50%> (+0.64%)`	⬆️
azurelinuxagent/common/event.py	`86.10% <90.47%> (+0.17%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2416e79...4aa8880. Read the comment docs.

azurelinuxagent/common/protocol/wire.py

tests/common/test_event.py

kevinclark19a · 2020-07-07T20:51:22Z

tests/protocol/test_wire.py

@@ -362,7 +362,7 @@ def test_send_event(self, mock_http_request, *args):

        event_str = u'a test string'
        client = WireProtocol(WIRESERVER_URL).client
-        client.send_event("foo", event_str)
+        client.send_event("foo", event_str.encode('utf-8'))


Should inputs to send_event be of type str, or of type ustr? This maybe harkens back to my prior comment about keeping encode secluded within communicating with the outside world.

I'm assuming that this is, like explained in the other comment, designed to keep changes out of the send_event function.

Technically it should be ustr (which is the same as str for Py3) here and only be encoded in the restutil function which actually sends the data out. But since right now I'm only focusing the changes for the telemetry event, we're encoding the string in this function which is just 1 level up from the restutil function to limit the blast radius of this change.

Does that make sense? We can discuss more on this offline if you have additional doubts.

azurelinuxagent/common/utils/textutil.py

…to fix-telemetry-unicode-errors

azurelinuxagent/ga/monitor.py

azurelinuxagent/common/event.py

azurelinuxagent/common/protocol/wire.py

azurelinuxagent/common/utils/restutil.py

azurelinuxagent/common/utils/textutil.py

narrieta

A couple of small comments

narrieta · 2020-07-16T17:50:01Z

azurelinuxagent/common/protocol/wire.py

@@ -488,7 +488,7 @@ def put_page_blob(self, url, data):


 def event_param_to_v1(param):
-    param_format = '<Param Name="{0}" Value={1} T="{2}" />'
+    param_format = ustr('<Param Name="{0}" Value={1} T="{2}" />')


can we add utf_8 somewhere in the function name to be more explicit on what it returns ?

…to fix-telemetry-unicode-errors

This reverts commit 6d10203.

larohra added 13 commits June 25, 2020 15:10

Make params and events ustr for special chars

f5321e7

encoding the telemetry events properly

ec1f5ee

Fix failing test

e548799

Fix failing tests

a36a104

Dont encode xml if not a string (i.e. already encoded)

636eeb3

Added a test case

ee2f85c

Added encoding declaration for Py2

48056fb

Converted all event strings to unicode

492a2cb

Changing the log data back to unicode for logging if not ustr

0b4be2b

Encoding string properly when logging

6d21865

Fixed failing py2 tests

8c8082b

Remove extra comments and added a try catch

3e7c3a0

Code cleanup and improved comments

157ae5d

larohra requested review from kevinclark19a, narrieta, pgombar and ZhidongPeng as code owners July 7, 2020 19:28

Test fix and code fix

e7bb36e

larohra commented Jul 7, 2020

View reviewed changes

azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved

larohra commented Jul 7, 2020

View reviewed changes

azurelinuxagent/common/protocol/wire.py Outdated Show resolved Hide resolved

Merge branch 'develop' into fix-telemetry-unicode-errors

ffa48ea

kevinclark19a reviewed Jul 7, 2020

View reviewed changes

azurelinuxagent/common/protocol/wire.py Outdated Show resolved Hide resolved

kevinclark19a reviewed Jul 7, 2020

View reviewed changes

tests/common/test_event.py Show resolved Hide resolved

kevinclark19a reviewed Jul 7, 2020

View reviewed changes

narrieta reviewed Jul 8, 2020

View reviewed changes

azurelinuxagent/common/utils/textutil.py Show resolved Hide resolved

narrieta reviewed Jul 8, 2020

View reviewed changes

azurelinuxagent/common/utils/textutil.py Show resolved Hide resolved

larohra added 2 commits July 8, 2020 23:03

Added error handling and reporting for dropped telemetry events.

705b7d1

Merge remote-tracking branch 'origin/fix-telemetry-unicode-errors' in…

0b982a7

…to fix-telemetry-unicode-errors

larohra commented Jul 9, 2020

View reviewed changes

azurelinuxagent/ga/monitor.py Outdated Show resolved Hide resolved

Added a todo comment

8079ed0

pgombar reviewed Jul 14, 2020

View reviewed changes

larohra added 2 commits July 14, 2020 16:52

Improved debug message

e0f60ab

Merge branch 'develop' into fix-telemetry-unicode-errors

a57adae

narrieta previously approved these changes Jul 15, 2020

View reviewed changes

kevinclark19a previously approved these changes Jul 15, 2020

View reviewed changes

pgombar previously approved these changes Jul 15, 2020

View reviewed changes

narrieta reviewed Jul 16, 2020

View reviewed changes

larohra added 2 commits July 16, 2020 12:14

Made encoding a parameter and renamed functions to be more specific

62ad236

Merge remote-tracking branch 'origin/fix-telemetry-unicode-errors' in…

6913447

…to fix-telemetry-unicode-errors

larohra dismissed stale reviews from pgombar, kevinclark19a, and narrieta via 6913447 July 16, 2020 19:15

larohra added 2 commits July 16, 2020 12:20

Merge branch 'develop' into fix-telemetry-unicode-errors

e2acd76

Merge branch 'develop' into fix-telemetry-unicode-errors

4aa8880

larohra requested review from narrieta, kevinclark19a and pgombar July 17, 2020 17:50

narrieta approved these changes Jul 17, 2020

View reviewed changes

pgombar approved these changes Jul 20, 2020

View reviewed changes

larohra merged commit 6d10203 into Azure:develop Jul 20, 2020

larohra deleted the fix-telemetry-unicode-errors branch July 20, 2020 17:24

larohra mentioned this pull request Sep 14, 2020

[BUG] Exception 'latin-1' codec can't encode characters in position when with Chinese characters in RG #2003

Closed

pgombar added a commit to pgombar/WALinuxAgent that referenced this pull request Nov 5, 2020

Revert "Fix telemetry unicode errors (Azure#1937)"

5adeaeb

This reverts commit 6d10203.

This was referenced Nov 5, 2020

Revert "Fix telemetry unicode errors (#1937)" #2067

Closed

Revert unicode fix manually (#1937) #2070

Merged

pgombar added a commit that referenced this pull request Nov 5, 2020

Revert unicode fix manually (#1937) (#2070)

072933d

larohra mentioned this pull request Jan 20, 2021

[BUG] [Wireserver Exception] 'latin-1' codec can't encode character '\u2018' in position 545 #2132

Closed

larohra mentioned this pull request Jun 22, 2021

Fix telemetry unicode errors (Re-add #1937) #2278

Merged

5 tasks

larohra added a commit that referenced this pull request Jun 23, 2021

Fix telemetry unicode errors (Re-add #1937) (#2278)

aaa0b6f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix telemetry unicode errors #1937

Fix telemetry unicode errors #1937

larohra commented Jul 7, 2020

codecov bot commented Jul 7, 2020 •

edited

kevinclark19a Jul 7, 2020

kevinclark19a Jul 7, 2020

larohra Jul 7, 2020

narrieta left a comment

narrieta Jul 16, 2020

larohra Jul 16, 2020

Fix telemetry unicode errors #1937

Fix telemetry unicode errors #1937

Conversation

larohra commented Jul 7, 2020

Description

PR information

Quality of Code and Contribution Guidelines

codecov bot commented Jul 7, 2020 • edited

Codecov Report

kevinclark19a Jul 7, 2020

Choose a reason for hiding this comment

kevinclark19a Jul 7, 2020

Choose a reason for hiding this comment

larohra Jul 7, 2020

Choose a reason for hiding this comment

narrieta left a comment

Choose a reason for hiding this comment

narrieta Jul 16, 2020

Choose a reason for hiding this comment

larohra Jul 16, 2020

Choose a reason for hiding this comment

codecov bot commented Jul 7, 2020 •

edited