-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Selecting from JSON columns does not decode them from UTF-8 #142
Comments
use strict;
use warnings;
use feature 'say';
use DBI;
binmode *STDOUT, ':utf8';
my $dbh = DBI->connect("DBI:MariaDB:test");
$dbh->do(q(CREATE TEMPORARY TABLE unicode_json_test(my_json JSON)));
$dbh->do(q(INSERT INTO unicode_json_test SET my_json = JSON_OBJECT('drink', CONVERT(UNHEX('537AC5916CC5916CC3A9') USING utf8mb4))));
say "JSON output: " . $dbh->selectrow_array(q(SELECT my_json FROM unicode_json_test));
say "Server version: " . $dbh->selectrow_array(q(SELECT VERSION()));
say "Driver version: " . $dbh->{Driver}->{Version};
say "DBI version: $DBI::VERSION"; Output is:
Seems that DBD::MariaDB 1.21 with MariaDB 10.3.10 is working fine. Or is there any problem? |
And here is another test case with decoding RAW jsons: use strict;
use warnings;
use DBI;
use Cpanel::JSON::XS;
use feature 'say';
use utf8;
binmode *STDOUT, ':utf8';
my $cjson = Cpanel::JSON::XS->new->allow_nonref;
my $dbh = DBI->connect("DBI:MariaDB:test");
$dbh->do(q(CREATE TEMPORARY TABLE unicode_json_test(my_json JSON)));
$dbh->do(q(INSERT INTO unicode_json_test SET my_json = JSON_OBJECT('drink', 'Szőlőlé')));
say "RAW JSON output: " . $dbh->selectrow_array(q(SELECT my_json FROM unicode_json_test));
say "Decoded drink JSON output: " . $cjson->decode($dbh->selectrow_array(q(SELECT my_json FROM unicode_json_test)))->{drink};
say "RAW JSON field output: " . $dbh->selectrow_array(q(SELECT json_extract(my_json, "$.drink") FROM unicode_json_test));
say "Decoded JSON field output: " . $cjson->decode($dbh->selectrow_array(q(SELECT json_extract(my_json, "$.drink") FROM unicode_json_test)));
say "Server version: " . $dbh->selectrow_array(q(SELECT VERSION()));
say "Driver version: " . $dbh->{Driver}->{Version};
say "DBI version: $DBI::VERSION"; Output is:
So output seems to be correct too. If there is something wrong let me know. Otherwise I will close this issue as invalid. |
The test cases were done on MySQL, so perhaps it is a difference between MySQL and MariaDB. |
Ok, so it depends on a server version? Can you (or anybody else) specify the version which doesn't work, together with the output of the above test cases? Without it, it is impossible to debug this problem (if there really is any). |
@ViktorNacht specified his test cases were run in MySQL 5.7.22 and 8.0.13 |
See jhthorsen/mojo-mysql#57 (comment) and jhthorsen/mojo-mysql#57 (comment). This is bug in MySQL server. So please report it to the MySQL server bug tracker. |
I'm closing this issue as this reported problem is on MySQL server, not in DBD::MariaDB driver. |
FYI: I have opened a bug against MySQL here, but I still have no way to reproduce or test it myself. https://bugs.mysql.com/bug.php?id=95698 |
@Grinnz: Do you have any response about that reported bug in MySQL bugtracker? |
I see no sign of life on the side of MySQL since July 10. |
There's been no response other than confirming the bug so far. |
I don't have a new enough MySQL handy, but in a bug report for Mojo::mysql it was indicated with a test case that when selecting from a JSON type column, the output is not UTF-8 decoded (jhthorsen/mojo-mysql#57 (comment)). MySQL documents that when JSON is interacted with as a string it is treated as the utf8mb4 charset so it should be returned that way, the same way it is passed as a utf8mb4 string.
The issue should be reproducible by inserting into a JSON type column with a query like so:
then selecting that column. JSON and encoding is a tricky subject because it is commonly represented as UTF-8 encoded bytes rather than text, but as MySQL treats it on input and output as a character string, it should be decoded as the input string is encoded; this also is what happens in DBD::Pg.
Also opened for DBD::mysql perl5-dbi/DBD-mysql#309
The text was updated successfully, but these errors were encountered: