Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Honor =encoding directive when decoding raw resonses
  • Loading branch information
rwstauner committed Mar 29, 2013
1 parent 055bf9c commit 502a86f
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 2 deletions.
15 changes: 13 additions & 2 deletions lib/MetaCPAN/Web/Model/API.pm
Expand Up @@ -100,8 +100,19 @@ sub raw_api_response {
# we have to assume an encoding; doing nothing is like assuming latin1
# we'll probably have the least number of issues if we assume utf8
try {
# decode so the template doesn't double-encode and return mojibake
$data &&= $encoding->decode( $data, $encode_check );
if( $data ){
my $enc;
# honor encoding if specified (as something other than utf-8)
if( ($enc) = ($data =~ /^=encoding\s+(\S+)\s*$/m) and $enc !~ /^utf-?8$/i ){
$data = Encode::decode( $enc, $data, $encode_check );
}
# theoretically we could check for a BOM here
# else try UTF-8
else {
# decode so the template doesn't double-encode and return mojibake
$data = $encoding->decode( $data, $encode_check );
}
}
}
catch {
warn $_[0];
Expand Down
16 changes: 16 additions & 0 deletions t/encoding.t
Expand Up @@ -90,6 +90,7 @@ foreach my $ctype ( 'text/plain', 'application/json' ){
# test that a raw, non-utf8 response is unchanged
foreach my $bad (
[ encode(utf8 => "foo\x{FFFF_FFFF}bar"), 'encoded lax perl utf8 chars' ],
[ encode(utf8 => "=encoding utf8\n\nfoo\x{FFFF_FFFF}bar"), '=encoding utf8 with bad chars' ],
[ "\225 cp1252 bullet", 'invalid utf-8 bytes' ],
){
test_raw_response($bad->[0], $bad->[0], $bad->[1] . " come back as is",
Expand All @@ -113,6 +114,21 @@ foreach my $ctype ( 'text/plain', 'application/json' ){
'utf-8 bytes decode to perl string'
);

# EURO SIGN
test_raw_response(
"=pod\n\n=encoding latin9\n\nsome pod \xa4\n\n=cut\n",
"=pod\n\n=encoding latin9\n\nsome pod \x{20ac}\n\n=cut\n",
'pod in other encoding converted to utf-8',
);

# bad encoding
test_raw_response(
("=pod\n\n=encoding foo-bar-baz\n\nsome char \xfe") x 2,
'pod with unknown encoding left as is',
warnings => [qr/Unknown encoding/, 'unknown encoding'],
not_utf8 => 1,
);

# not sure if we'll ever actually get undef
is get_raw(undef), '', 'undef becomes blank';
ok !@warnings, 'no warnings for undef' or diag shift @warnings;
Expand Down

2 comments on commit 502a86f

@monken
Copy link
Contributor

@monken monken commented on 502a86f Mar 29, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See metacpan/metacpan-api#258 (comment)
I don't think we should apply that level of magic.

@rwstauner
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I admit I wasn't entirely sure about this either, but the way the page got screwed up without it didn't seem right either.

We're telling the browser that we're providing utf-8, and then instead we're serving up mixed encodings.

Well actually i guess that's not true. Without the detection, we attempt to decode those bytes as utf-8 but that will fail,
so then the Catalyst plugin will treat the euc-jp encoded bytes as latin1 chars and turns it into weird symbols and replacement characters.

I am willing to revert this since it's neither 100% satisfactory nor 100% correct.

Besides, everyone should be using UTF-8, right? ;-)

Please sign in to comment.