Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Honor =encoding directive when decoding raw resonses

  • Loading branch information...
commit 502a86f8b14a3d686ebdcda41ebbbb3699da0304 1 parent 055bf9c
@rwstauner rwstauner authored
Showing with 29 additions and 2 deletions.
  1. +13 −2 lib/MetaCPAN/Web/Model/API.pm
  2. +16 −0 t/encoding.t
View
15 lib/MetaCPAN/Web/Model/API.pm
@@ -100,8 +100,19 @@ sub raw_api_response {
# we have to assume an encoding; doing nothing is like assuming latin1
# we'll probably have the least number of issues if we assume utf8
try {
- # decode so the template doesn't double-encode and return mojibake
- $data &&= $encoding->decode( $data, $encode_check );
+ if( $data ){
+ my $enc;
+ # honor encoding if specified (as something other than utf-8)
+ if( ($enc) = ($data =~ /^=encoding\s+(\S+)\s*$/m) and $enc !~ /^utf-?8$/i ){
+ $data = Encode::decode( $enc, $data, $encode_check );
+ }
+ # theoretically we could check for a BOM here
+ # else try UTF-8
+ else {
+ # decode so the template doesn't double-encode and return mojibake
+ $data = $encoding->decode( $data, $encode_check );
+ }
+ }
}
catch {
warn $_[0];
View
16 t/encoding.t
@@ -90,6 +90,7 @@ foreach my $ctype ( 'text/plain', 'application/json' ){
# test that a raw, non-utf8 response is unchanged
foreach my $bad (
[ encode(utf8 => "foo\x{FFFF_FFFF}bar"), 'encoded lax perl utf8 chars' ],
+ [ encode(utf8 => "=encoding utf8\n\nfoo\x{FFFF_FFFF}bar"), '=encoding utf8 with bad chars' ],
[ "\225 cp1252 bullet", 'invalid utf-8 bytes' ],
){
test_raw_response($bad->[0], $bad->[0], $bad->[1] . " come back as is",
@@ -113,6 +114,21 @@ foreach my $ctype ( 'text/plain', 'application/json' ){
'utf-8 bytes decode to perl string'
);
+ # EURO SIGN
+ test_raw_response(
+ "=pod\n\n=encoding latin9\n\nsome pod \xa4\n\n=cut\n",
+ "=pod\n\n=encoding latin9\n\nsome pod \x{20ac}\n\n=cut\n",
+ 'pod in other encoding converted to utf-8',
+ );
+
+ # bad encoding
+ test_raw_response(
+ ("=pod\n\n=encoding foo-bar-baz\n\nsome char \xfe") x 2,
+ 'pod with unknown encoding left as is',
+ warnings => [qr/Unknown encoding/, 'unknown encoding'],
+ not_utf8 => 1,
+ );
+
# not sure if we'll ever actually get undef
is get_raw(undef), '', 'undef becomes blank';
ok !@warnings, 'no warnings for undef' or diag shift @warnings;

2 comments on commit 502a86f

@monken
Owner

See CPAN-API/cpan-api#258 (comment)
I don't think we should apply that level of magic.

@rwstauner
Owner

I admit I wasn't entirely sure about this either, but the way the page got screwed up without it didn't seem right either.

We're telling the browser that we're providing utf-8, and then instead we're serving up mixed encodings.

Well actually i guess that's not true. Without the detection, we attempt to decode those bytes as utf-8 but that will fail,
so then the Catalyst plugin will treat the euc-jp encoded bytes as latin1 chars and turns it into weird symbols and replacement characters.

I am willing to revert this since it's neither 100% satisfactory nor 100% correct.

Besides, everyone should be using UTF-8, right? ;-)

Please sign in to comment.
Something went wrong with that request. Please try again.