Truncate messages based on bytes, not unicode characters #181
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kislyuk
requested changes
Jan 28, 2023
watchtower/__init__.py
Outdated
| @@ -45,6 +45,20 @@ def _boto_filter(record): | |||
| return True | |||
|
|
|||
|
|
|||
| def _size(msg): | |||
There was a problem hiding this comment.
Can you move _size to be a method of CloudWatchLogHandler to simplify customization by subclassing?
watchtower/__init__.py
Outdated
| ) + CloudWatchLogHandler.EXTRA_MSG_PAYLOAD_SIZE | ||
|
|
||
|
|
||
| def _truncate(msg, max_size): |
There was a problem hiding this comment.
Same here - Can you move _truncate to be a method of CloudWatchLogHandler to simplify customization by subclassing?
tomkins
force-pushed
the
fix-unicode-truncation
branch
from
89fc46b
to
f8e3d4b
Compare
|
Updated, and hopefully found an okay place to put them. Kept them as |
|
Released in v3.0.1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We've been getting this message quite a bit in one of our apps:
Some of the messages we're sending contain unicode characters, and we're sending a lot of messages in the batches. However as the message size calculation is counting the length of strings, instead of the number of bytes - we're getting warnings originating from boto3.
Temporary workaround is to drop the message/batch size - however the better fix is to change the size checking and truncation to look at the bytes of the messages.