Failing Test for german special characters

[m0veax]

Hey,

i've created a test which is populating a MySQL Database with german special chars, dumping it into fixtures and repopulate it from the fixtures.

You have to set Environment variables, to run the test against a MySQL database:

export FIXTURETEST_DSN="DBI:mysql:database=fixtures;host=localhost"
export FIXTURETEST_USER=mysqluser
export FIXTURETEST_PASS=mysqlpassword

t/19-populate-unicode.t fails after importing the fixtures back into the MySQL DB.

We couldn't find the bug, but can help if you give us a hint.

Thanks in advance,

Patrick

[samuelckaufman]

@jjn1056 just FYI I've been taking to @jegade on #dbix-class about this issue.
From what I can tell, it is not neccesarily DBIx::Class::Fixtures, it's whatever underlying DB driver that pulls data out of the DB's encoding settings. If it does not come back as a wide string, it gets screwy.

[fheyer]

I'm experiencing the problems described above if the connection to mysql uses the option
mysql_enable_utf8=1.
With this setting enabled german umlauts loaded from fixtures are broken.

After disabling this option and loading the fixtures again with populate the umlauts are correctly stored.

Without looking into the code i assume that strings are not encoded to utf8 before saving them to the DB. However, with mysql_enable_utf8=1 the driver expects strings in UTF8.

It would be great if using fixtures would work if mysql_enable_utf8 is used symmetrically:
A fixture created with mysql_enable_utf8=0 can and should be loaded with the same setting. This should also be true for mysql_enable_utf8=1.

[samuelckaufman]

@fheyer I have to dig up my irc logs tomorrow to refresh my memory on what the underlying issue was, I don't have access to my logs here.

[samuelckaufman]

@fheyer https://gist.github.com/skaufman-socialflow/67a6e797a29aa37f221fb142f1c402f2 FYI

[samuelckaufman]

--- Mon Jun 08 2015
17:48 < skaufman> jegade: if you are poking around ::Fixtures, try debugging near 
17:48 < skaufman> https://metacpan.org/source/SKAUFMAN/DBIx-Class-Fixtures-1.001032/lib/DBIx/Class/Fixtures.pm#L851
17:48 < skaufman> $ds->title should be a wide string
17:49 < skaufman> if is UTf-8 encoded there, then it's wrong
17:49 < skaufman> i just broke into a repl if $title matched 'Chars'
17:52 < jegade> skaufman: ok, then i should remove mysql_enable_utf8? 
17:53 < jegade> this is setting the strings to utf8 encoding
17:54 < skaufman> jegade: nah i think you should leave that in,
17:54 < skaufman> but from the docs it looks like DBD::mysql does some other things to check if it should be UTF-8 decoded on the way out
17:54 < skaufman> based on the db's encoding settings
17:54 < skaufman> so first of all make sure you're on the latest DBD::mysql
17:55 < skaufman> then probably just try to see why stuff is not coming back UTF-8 decoded
17:55 < skaufman> When set, a data retrieved from a textual column type (char, varchar, etc) will have the UTF-8 flag turned on if necessary.
17:55 < skaufman> the 'if neccesary' is a bit fishy
18:00 < skaufman> also might want to dig around in this test
18:00 < skaufman> https://metacpan.org/source/CAPTTOFU/DBD-mysql-4.032_01/t/55utf8.t
18:50 < jegade> skaufman: is the problem at dumping or population the fixtures? just to be sure i understand it right
18:51 < skaufman> jegade: at dumping
18:51 < skaufman> er sorry
18:51 < skaufman> jegade: it populates in that lib
18:51 < skaufman> then it dumps 
18:51 < skaufman> then it installs the fixtures
18:51 < skaufman> so the populate seems fine,
18:52 < skaufman> then it pulls the data out,
18:52 < skaufman> at that point it is not a wide string
18:52 < skaufman> which it should be
18:52 < skaufman> then it dumps it to file UTF-8 encoded,
18:52 < skaufman> reads it back UTF-8 encoded
18:52 < skaufman> and double encodes on the way in
18:52 < skaufman> at least thats what happens with sqlite````