Fix support for mysql_enable_utf8 and mysql_enable_utf8mb4

[pali]

WARNING: This pull request changes behavior of mysql_enable_utf8 and mysql_enable_utf8mb4 attributes. New behavior is documented in POD section. Changes should be properly reviewed and tested as in some situations it can be backward incompatible change...

Probably people affected by UTF-8 bugs in DBD::mysql should test and verify that it really fix reported problems. Maybe we should discuss more about this pull request.

Reported bugs:
https://rt.cpan.org/Public/Bug/Display.html?id=25590
https://rt.cpan.org/Public/Bug/Display.html?id=53130
https://rt.cpan.org/Public/Bug/Display.html?id=60987
https://rt.cpan.org/Public/Bug/Display.html?id=87428

---

Important changes:

• Fix support for prepared statement numeric storage, DBI SQL bind types
• Make consistency between DBI, mysql and perl types
• Do not UTF-8 encode input binded params which are of DBI SQL BINARY types
• Do not UTF-8 decode output fields which are not send in utf8 (resp. utf8mb4) charset from mysql server
• Use SvPVutf8() for retrieving UTF-8 encoded char * because SvPV() can return char * in Latin1
• Remove #ifdef checks for sv_utf8_decode and SvUTF8 as they are not needed anymore

---

Part of this pull request is @davel (Dave Lambley) test for UTF-8 params from davel/DBD-mysql@8695068 references by https://rt.cpan.org/Public/Bug/Display.html?id=60987. I slightly modified it, but it is still great test case for scenario with different table charset, different session charset and state of mysql_enable_utf8.

---

Prepared statements now uses only necessary long buffers for integer types
(not always 64bit anymore). DBI SQL types SQL_BOOLEAN and SQL_TINYINT now
handles 8bit integers, SQL_SMALLINT 16bit integers, SQL_INTEGER 32bit
integers and SQL_BIGINT 64bit integers. SQL_FLOAT native floats and
SQL_DOUBLE or SQL_REAL native doubles.

Now when prepared statements are enabled and explicit storage of numeric
values (different integer size and floating point size) via DBI SQL is
used, conversion to storage type is done by DBD::mysql driver and not by
mysql server. It means that any overflow/underflow is not detected by mysql
server and application needs to handle it.

SQL_BOOLEAN is treated as numeric type and SQL_DECIMAL is handed by mysql
as string type (due to decimal precision). SQL_BIT, SQL_BLOB, SQL_BINARY,
SQL_VARBINARY and SQL_LONGVARBINARY are treated as perl binary types.

And MYSQL_TYPE_BIT, MYSQL_TYPE_MEDIUM_BLOB and MYSQL_TYPE_LONG_BLOB are
treated as binary perl types. MYSQL_TYPE_LONGLONG as numeric perl type on
platforms with 64 bit perl's integers and string perl type on others.
MYSQL_TYPE_NULL is now perl's undef.

---

For each fetched field mysql server tells us also charset id. Before this
commit when mysql_enable_utf8 was enabled DBD::mysql UTF-8 decoded all
fields with charset id different of 63 (means binary).

Now DBD::mysql UTF-8 decode only those fields which have charset set to
utf8 or utf8mb4. By default mysql server sends data in encoding specified
by SET NAMES command, which is by default Latin1. So received Latin1 data
are not UTF-8 decoded anymore.

---

Before this commit perl scalars (statements or bind parameters) without
UTF8 status flag were not encoded to UTF-8 even if mysql_enable_utf8 was
enabled. It caused that perl scalars with internal Latin1 encoding were
send to mysql server as Latin1 even if mysql_enable_utf8 was enabled.

Now all statements and bind parameters which are not of DBI binary type
(SQL_BIT, SQL_BLOB, SQL_BINARY, SQL_VARBINARY and SQL_LONGVARBINARY) are
automatically encoded to UTF-8 when mysql_enable_utf8 is enabled.

If mysql_enable_utf8 is not enabled and statement or bind parameter
contains wide Unicode character then DBD::mysql shows warning. If binary
parameter contains wide Unicode character then DBD::mysql shows warning
too. Similar like function print without :utf8 perlio layer.

[Grinnz]

Thank you so much. This fix is sorely needed. I will try to find time to test but one question, should SQL_BLOB also be considered a column type to prevent encoding? It seems to be used mostly by SQLite and several database abstraction tools.

[pali]

Looks like SQL_BLOB should not be encoded/decoded too... But this is probably question for DBI (API) developers which types are considered as binary and which not. I think that every DBI driver needs have similar logic if it want to have proper support for Unicode perl scalars...

[pali]

Big update of this pull request is there! It address problem reported by schmorp in discussion https://rt.cpan.org/Public/Bug/Display.html?id=87428

[pali]

version XXX needs to be changed according to next release after merge...

Also please re-check that new rewritten documentation is OK... I'm not native English speaker and documentation is important!

[Grinnz]

Easier to repaste the whole section with grammar fixes than comment on each part. (Sorry but difficult to fix the line lengths in github's editor.) This documentation section is greatly improved from the released version btw!

=item mysql_enable_utf8

This attribute affects input data from DBI (statement and bind parameters) and
output data from the mysql server.

If used as a part of the call to C<connect()> then it issues the command
C<SET NAMES utf8>.

When set, any statement or bind parameter which is not of binary type is
automatically encoded to UTF-8 octets before being sent to the MySQL server.  Any
retrieved MySQL data with a charset of C<utf8> or C<utf8mb4> from a textual column
type (char, varchar, etc) is automatically UTF-8 decoded and returned as a perl
Unicode scalar (with SvUTF8 flag on).  That enables character semantics on those
retrieved UTF-8 strings.  The MySQL charset of a retrieved value is affected by the
last C<SET NAMES> command and also could be affected by the database, table and column
configuration.  For more information, see the I<Character Set Support> chapter in
the MySQL manual: L<http://dev.mysql.com/doc/refman/5.7/en/charset.html>

When unset and a statement or bind parameter contains a wide Unicode character then
DBD::mysql gives the warning C<Wide character in ... but mysql_enable_utf8 not set>.
The MySQL protocol does not support wide characters and so DBD::mysql does not know
how to send a statement with wide characters when C<mysql_enable_utf8> is not set.

Please note that when C<mysql_enable_utf8> is set, the input statement and bind
parameters are encoded to UTF-8 octets even if the current MySQL session charset is
not C<utf8> or C<utf8mb4>!  You are responsible for calling the C<SET NAMES utf8> or
C<SET NAMES utf8mb4> command when setting the C<mysql_enable_utf8> attribute after connecting.
The same applies to unsetting the C<mysql_enable_utf8> attribute.  You are responsible
for calling C<SET NAMES latin1> (resp. with correct charset) and then passing perl
scalars in the correct encoding.  Otherwise strings will be sent to MySQL server
incorrectly!

Input bind parameters of binary types (C<SQL_BIT>, C<SQL_BLOB>, C<SQL_BINARY>,
C<SQL_VARBINARY> and C<SQL_LONGVARBINARY>) are not touched regardless of the
C<mysql_enable_utf8> attribute state.  They are treated as a sequence of octets
and sent to the MySQL server as is.  If that bind parameter contains a wide Unicode
character then DBD::mysql gives the warning C<Wide character in binary field ...>
because binary data is a sequence of octets, not Unicode characters!

Output data fetched from the MySQL server which does not have a C<utf8> or C<utf8mb4>
charset (so also binary data) is not UTF-8 decoded regardless of the
C<mysql_enable_utf8> attribute state.  They are treated as a sequence of octets
and it is your responsibility to decode them correctly.

B<WARNING>: DBD::mysql prior to version XXX had different and buggy behaviour
when the attribute C<mysql_enable_utf8> was enabled!  Input statement and bind
parameters were never encoded to UTF-8 octets and retrieved columns were
always UTF-8 decoded regardless of the column charset (except binary charsets).

=item mysql_enable_utf8mb4

Exactly the same as the attribute C<mysql_enable_utf8>.

Additionally if used as a part of the call to C<connect()> then it issues
the command C<SET NAMES utf8mb4> instead of C<utf8>.

MySQL's C<utf8mb4> charset is capable of handling 4-byte UTF-8 characters.
MySQL's C<utf8> charset is capable of handling only up to 3-byte UTF-8
characters! See MySQL manual for more information:
L<http://dev.mysql.com/doc/refman/5.7/en/charset-unicode-utf8mb4.html>

You should use MySQL's C<utf8mb4> charset instead of C<utf8> to prevent problems
with data exchange.  When the C<utf8> charset is used then you are responsible for
3-byte UTF-8 sequence checks on input perl scalar strings.  Otherwise MySQL
server can reject or modify the input statement!

[pali]

Thank you! Text updated.

[mbeijen]

This is SUPER awesome! Thanks a lot. I'll put out a new test release on CPAN with it.

Of course testing, eyeballs and comments are welcome.

[pali]

Do not forget to set correct version instead of XXX.

[mbeijen]

Sure, thanks for the pointer. I will modify that and amend the change log and then push a new development release out tomorrow. Op za 10 dec. 2016 om 12:05 schreef pali <notifications@github.com>

…

***@***.**** commented on this pull request. ------------------------------ In lib/DBD/mysql.pm <#67 (review)> : > +correct charset) and then passing perl scalars in the correct encoding. +Otherwise strings will be sent to MySQL server incorrectly! + +Input bind parameters of binary types (C<SQL_BIT>, C<SQL_BLOB>, C<SQL_BINARY>, +C<SQL_VARBINARY> and C<SQL_LONGVARBINARY>) are not touched regardless of the +C<mysql_enable_utf8> attribute state. They are treated as a sequence of octets +and sent to the MySQL server as is. If that bind parameter contains a wide Unicode +character then DBD::mysql gives the warning C<Wide character in binary field ...> +because binary data is a sequence of octets, not Unicode characters! + +Output data fetched from the MySQL server which does not have a C<utf8> or +C<utf8mb4> charset (so also binary data) is not UTF-8 decoded regardless of the +C<mysql_enable_utf8> attribute state. They are treated as a sequence of octets +and it is your responsibility to decode them correctly. + +B<WARNING>: DBD::mysql prior to version XXX had different and buggy behaviour Do not forget to set correct version instead of XXX. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#67 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAoQMGf_kzh5nG5MyVubTxyxEMIVqtQCks5rGodlgaJpZM4Kd4ac> .