Npgsql-4.0.7 Copy Writer with emoji : Unable to translate unicode character \\uD83D at index 499 to specified code page

[leafduan]

The issue

when using copy importer from binary to witer text column with emoji

before writer, substring(0, 500) for string

Exception message: Unable to translate Unicode character \\uD83D at index 499 to specified code page.
Stack trace:
at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index) 
at System.Text.EncoderFallbackBuffer.InternalFallback(Char ch, Char*& chars) 
at System.Text.UTF8Encoding.GetByteCount(Char* chars, Int32 count, EncoderNLS baseEncoder) 
at System.Text.UTF8Encoding.GetByteCount(String chars) 
at Npgsql.TypeHandlers.TextHandler.ValidateAndGetLength(String value, NpgsqlLengthCache& lengthCache, NpgsqlParameter parameter) 
at lambda_method(Closure , NpgsqlTypeHandler , Object , NpgsqlLengthCache& , NpgsqlParameter )
at Npgsql.NpgsqlParameter.ValidateAndGetLength()
at Npgsql.NpgsqlBinaryImporter.Write[T](T value, NpgsqlParameter param) // T is string

Further technical details

Npgsql version: 4.0.7
PostgreSQL version:

PostgreSQL 10.10 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36), 64-bit
LC_CTYPE=zh_CN.UTF-8
LC_COLLATE=zh_CN.UTF-8

Operating system (App):

CentOS Linux release 7.2.1511 (Core)
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"

[roji]

What exact string are you trying to write there?

I'm not Unicode expert, but unless I'm mistaken \uD83d is a UTF-16 High Surrogate - it's not a full character in itself, but rather signals that another value is coming. If you specify a surrogate alone, that's an invalid UTF-16 string that can't be encoded into UTF-8.

For example, the following fails with your exception message:

writer.Write("Hello \uD83D");

But the following properly writes an emoji:

writer.Write("Hello \uD83D\uDE04");

Note that this has nothing to do with PostgreSQL or Npgsql, it's a general C#/UTF-16/encoding question.

[leafduan]

@roji Thanks, I understand.
The string end with emoji, eg "😊" ("\uD83D\uDE09")

var emoji = "😊";
emoji.Length == 2;  // true

var substr = emoji.Substring(0, 1);
substr.Length == 1;  // true

// and substr is high surrogate '\uD83D', it's not a valid unicode char. 
// high and low surrogates cannot be interpreted when they do not appear as part of a surrogate pair.

So when saving substr to database, an encoder fallback exception throw.

[roji]

Yes, that's correct. The fact that emoji.Length returns 2 is... questionable - it returns the number of 16-bit characters in the string, which doesn't correspond to the number of characters if any surrogates are being used.