Jira string truncation does not take into account UTF-8 encoding

[alexcottner]

Jira uses UTF-8 encoding for unicode characters. This means that some characters will take up multiple characters of space in the string. For example, 🀍 is considered 2 characters as it takes 4 bytes. With the current spec, it seems possible for some to take up 6 bytes which should be 3 characters? Although I couldn't find any examples of this.

We should adjust our string truncation to take this into account. I'm hoping there is some standard string encoding functiosn we can use to adjust this.

jira-bugzilla-integration/jbi/jira/utils.py

Lines 13 to 15 in ac3dec1

	if max_length > 0 and len(jira_output) > max_length:
	# Truncate on last word.
	jira_output = jira_output[:max_length].rsplit(maxsplit=1)[0]

[smarnach]

UTF-8 characters can only take up 4 bytes at most. A long time ago, the specification allowed for at most 6 bytes, but that was never used, and it's now completely ruled out that a character uses more than 4 bytes.

I don't understand how a four-byte character is considered two characters. What is counting the single character as two characters and for what purpose?

[alexcottner]

The Jira API appears to be considering all 3 and 4 byte characters as 2 characters when validating string lengths, or maybe when storing the data. You can test this yourself by trying to enter a ticket with 30k "A" characters vs 17k "🀍" characters. I don't know if it's a validation rule or if their data layer is rejecting it.

Although that is good to know.

[smarnach]

I asked ChatGPT about this, and it turns out Jira simply uses Java's string length function, and Java counts UTF-16 code units. To get the same string length in Python, we can use len(s.encode("utf-16-le")) // 2. To truncate a string to 32,767 characters, we can conceptuallyuse something like this:

max_length = 32_767
truncated_comment = comment.encode("utf-16-le")[:max_length * 2].decode("utf-16-le")

However, we also need to make sure we are not truncating in the middle of a surrogate pair, since that would make the string invalid. The simplest approach to do so I can think of is catching the UnicodeDecodeError that we get if that happens, and strip off another two bytes at the end of the string in that case.

(Ideally, we would also make sure that we are not truncating in the middle of a grapheme cluster, but that would require an additional library.)

[alexcottner]

However, we also need to make sure we are not truncating in the middle of a surrogate pair, since that would make the string invalid.

Currently we truncate to the end of the previous word, so I don't think we'll have a problem if we keep doing this.

[smarnach]

Sure, but then we'll need to figure out how to find a word boundary in UTF-16 raw bytes. We can't convert back to an actual string unless we make sure we haven't truncated in the middle of a surrogate pair, so we probably need to do both:

comment_utf16 = comment.encode("utf-16-le")[:2 * max_length]
try:
    truncated_comment = comment_utf16.decode("utf-16-le")
except UnicodeDecodeError:
    truncated_comment = comment_utf16[:-2].decode("utf-16-le")
truncated_comment = truncated_comment.rsplit(maxsplit=1)[0]