How to handle "could not map to Unicode" warnings?

[dagolden]

Options:

• fatalize them
• leave them alone
• catch them in a __WARN__ handler and ignore them

If we're trying to read UTF-8 and the file isn't UTF-8, the "correct" thing to do might be to throw an error instead of allowing it through. Both :utf8 and :encoding(UTF-8) can have the "utf8" category fatalized, but that might be too restrictive if the warning occurs in common cases when the substitution character would be put in.

The other extreme is to catch warnings and ignore, since users are getting exactly what they asked for and it's not our fault if that's not right. We can turn off utf8 warnings for :utf8, but it's not respected by encoding(UTF-8) so we would need a __WARN__ handler.

Since I don't know the right answer, I'll do nothing for now but leave this as a placeholder.

[kentfredric]

Couldn't tell if this is a new bug, or the same bug:

Can't decode ill-formed UTF-8 octet sequence <E9> in position 5623 at /home/kent/perl5/perlbrew/perls/perl-5.19.5/lib/site_perl/5.19.5/Path/Tiny.pm line 679.
Can't decode ill-formed UTF-8 octet sequence <E9> in position 10096 at /home/kent/perl5/perlbrew/perls/perl-5.19.5/lib/site_perl/5.19.5/Path/Tiny.pm line 679.

This annoying, because its cropping up in the midst of processing several hundred files, and I have no idea which file the warning is pertaining to.

R-ing the source says its not any underlying mechanisms job to report this, because the underlying mechanisms aren't dealing with files at all, they only deal with the bytes.

So it makes sense its Path::Tiny's job to add filename context to this warning, ... though how to do that is anyones guess.

[dagolden]

Could you please tell me more about what you're processing: i.e. how and why? I'd like to better understand some use cases before deciding on what Path::Tiny should do.

[kentfredric]

I'm just interating the contents of Dist/Zilla/Plugin/* , oddly enough, seems the problem is Test::Compile, specificially, Test::Compile has a =pod section in ISO-8859-1

https://metacpan.org/source/ETHER/Dist-Zilla-Plugin-Test-Compile-2.037/lib/Dist/Zilla/Plugin/Test/Compile.pm#L198

[chansen]

Unicode::UTF8 supports fallbacks for encode_utf8() and decode_utf8() where you can report any warnings or throw exceptions.

Example:

diff --git a/lib/Path/Tiny.pm b/lib/Path/Tiny.pm
index c914332..32a343a 100644
--- a/lib/Path/Tiny.pm
+++ b/lib/Path/Tiny.pm
@@ -1137,7 +1137,24 @@ sub slurp_raw { $_[1] = { binmode => ":unix" }; goto &slurp }

 sub slurp_utf8 {
     if ( defined($HAS_UU) ? $HAS_UU : $HAS_UU = _check_UU() ) {
-        return Unicode::UTF8::decode_utf8( slurp( $_[0], { binmode => ":unix" } ) );
+        my $path = $_[0]->[PATH];
+        my $fallback = sub {
+            my ($octets, $usv, $position) = @_;
+
+            my $msg;
+            if ($usv) {
+                $msg = sprintf "Can't interchange noncharacter code point U+%X in file '%s' at position %d",
+                    $usv, $path, $position;
+            }
+            else {
+                $msg = sprintf "Can't decode ill-formed UTF-8 octet sequence <%s> in file '%s' at position %d",
+                    join(' ', map { sprintf '%.2X', ord } split //, $octets), $path, $position;
+            }
+            Carp::carp($msg);
+            return "\x{FFFD}";
+        };
+        no warnings 'utf8';
+        return Unicode::UTF8::decode_utf8( slurp( $_[0], { binmode => ":unix" } ), $fallback);
     }
     else {
         $_[1] = { binmode => ":raw:encoding(UTF-8)" };

Would output:

 $ perl -Mlib=lib -MPath::Tiny -wle 'path("~/dev/bad")->slurp_utf8;'
Can't decode ill-formed UTF-8 octet sequence <EF BF> in file '/Users/chansen/dev/bad' at position 4 at -e line 1

[ilmari]

PerlIO::encoding takes its fallback behaviour from the value of $PerlIO::encoding::fallback when the layer is applied, so you can set that to one of the Encoding::FB_* values.

[xdg]

On reflection, I'm going to close this "won't fix". Users can disable warnings in various ways if as a Tiny module, I don't think it's the right move to add callback overhead handling malformed characters.