Skip to content
This repository has been archived by the owner on Oct 13, 2023. It is now read-only.

[18.09 backport] awslogs: account for UTF-8 normalization in limits #112

Merged
merged 1 commit into from
Nov 27, 2018

Conversation

thaJeztah
Copy link
Member

backport of moby#37986 for 18.09
fixes moby#37747 for 18.09

git checkout -b 18.09_backport_moby_37747 ce-engine/18.09
git cherry-pick -s -S -x 1e8ef386279e2e28aff199047e798fad660efbdd
git push -u origin

cherry-pick was clean; no conflicts

The CloudWatch Logs API defines its limits in terms of bytes, but its
inputs in terms of UTF-8 encoded strings. Byte-sequences which are not
valid UTF-8 encodings are normalized to the Unicode replacement
character U+FFFD, which is a 3-byte sequence in UTF-8. This replacement
can cause the input to grow, exceeding the API limit and causing failed
API calls.

This commit adds logic for counting the effective byte length after
normalization and splitting input without splitting valid UTF-8
byte-sequences into two invalid byte-sequences.

Fixes moby#37747

The CloudWatch Logs API defines its limits in terms of bytes, but its
inputs in terms of UTF-8 encoded strings.  Byte-sequences which are not
valid UTF-8 encodings are normalized to the Unicode replacement
character U+FFFD, which is a 3-byte sequence in UTF-8.  This replacement
can cause the input to grow, exceeding the API limit and causing failed
API calls.

This commit adds logic for counting the effective byte length after
normalization and splitting input without splitting valid UTF-8
byte-sequences into two invalid byte-sequences.

Fixes moby#37747

Signed-off-by: Samuel Karp <skarp@amazon.com>
(cherry picked from commit 1e8ef38)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah thaJeztah modified the milestones: 18.09.0, 18.09.1 Nov 8, 2018
@thaJeztah
Copy link
Member Author

Failure on Janky is unrelated; it's caused by the swagger-generation, and could be fixed by moby#36714 but that needs a carry

12:23:26 The result of hack/generate-swagger-api.sh differs
12:23:26 
12:23:26 diff --git a/api/types/container/container_wait.go b/api/types/container/container_wait.go
12:23:26 index 9e3910a6b4..06b0f02077 100644
12:23:26 --- a/api/types/container/container_wait.go
12:23:26 +++ b/api/types/container/container_wait.go
12:23:26 @@ -7,14 +7,6 @@ package container

Copy link

@andrewhsu andrewhsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andrewhsu andrewhsu merged commit 08a77f1 into docker-archive:18.09 Nov 27, 2018
@thaJeztah thaJeztah deleted the 18.09_backport_moby_37747 branch November 27, 2018 18:07
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants