Skip to content

Commit

Permalink
Fix already decoded BOM in headers
Browse files Browse the repository at this point in the history
  • Loading branch information
H.Merijn Brand - Tux committed Jun 9, 2017
1 parent 44b29b3 commit ed26ab6
Show file tree
Hide file tree
Showing 4 changed files with 33 additions and 1 deletion.
3 changes: 2 additions & 1 deletion CSV_XS.pm
Expand Up @@ -26,7 +26,7 @@ use DynaLoader ();
use Carp;

use vars qw( $VERSION @ISA @EXPORT_OK );
$VERSION = "1.30";
$VERSION = "1.31";
@ISA = qw( DynaLoader Exporter );
@EXPORT_OK = qw( csv );
bootstrap Text::CSV_XS $VERSION;
Expand Down Expand Up @@ -812,6 +812,7 @@ sub header {
elsif ($hdr =~ s/^\x0e\xfe\xff//) { $enc = "scsu" }
elsif ($hdr =~ s/^\xfb\xee\x28//) { $enc = "bocu-1" }
elsif ($hdr =~ s/^\x84\x31\x95\x33//) { $enc = "gb-18030" }
elsif ($hdr =~ s/^\x{feff}//) { $enc = "" }

if ($enc) {
if ($enc =~ m/([13]).le$/) {
Expand Down
3 changes: 3 additions & 0 deletions ChangeLog
@@ -1,3 +1,6 @@
1.31 - 2017-06-09, H.Merijn Brand
* Fix already decoded BOM in headers

1.30 - 2017-06-08, H.Merijn Brand
* Add csv (..., out => ...) syntax examples (issue 7)
* Disable escape_null for undefined escape_char
Expand Down
2 changes: 2 additions & 0 deletions sandbox/pm-1192242.csv
@@ -0,0 +1,2 @@
"Volume.label","Volume.serialno","Volume.vtype","Volume.netname","Volume.filesystem","Path.name",,"Thumbnail.checksum","Thumbnail.width","Thumbnail.height","Thumbnail.horiz_res","Thumbnail.vert_res","Thumbnail.colortype","Thumbnail.colordepth","Thumbnail.gamma","Thumbnail.thumbnail_width","Thumbnail.thumbnail_height","Thumbnail.thumbnail_type","Thumbnail.thumbnail_size","Thumbnail.name","Thumbnail.metric1","Keywords.pkeywords",
PCD0138,,4037894171,5,\\ddb\r$,CDFS,PHOTO_CD\IMAGES,1,0,0,"1996-09-30T21:38:57","2002-10-12T00:29:25",3368960,2147483648,512,768,0,0,0,24,0,68,100,518,336,IMG0002.PCD,m000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,b00000000000000000000000000000000,,0,";",lose
26 changes: 26 additions & 0 deletions sandbox/pm-1192242.pl
@@ -0,0 +1,26 @@
#!/pro/bin/perl

# Read the export from Thumbs Plus including keywords from filename given.

use 5.18.2; # 5.12 or later to get "unicode_strings" feature
use utf8; # so literals and identifiers can be in UTF-8
use warnings qw(FATAL utf8); # fatalize encoding glitches
use Data::Peek; # debug
use Text::CSV;

my $csv = Text::CSV->new ({ binary => 1, auto_diag => 1 });

my $fn = "pm-1192242.csv";
-d "sandbox" and substr $fn, 0, 0, "sandbox/";
say $fn;
open my $fh, "<", $fn or die "$fn: $!\n";
#open my $fh, "<:encoding(utf-8)", $fn or die "$fn: $!\n";
say "Point a";

# Returns "the instance" -- of what? Do I care?
my @hdr = $csv->header ($fh, {
munge_column_names => sub { state $i; $_ || "nothing.".$i++ }});
say "Point b";

DDumper { csv => $csv, hdr => \@hdr };
say for @hdr;

4 comments on commit ed26ab6

@wilx
Copy link

@wilx wilx commented on ed26ab6 Jun 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have hit the issue that this change set fixes. It would be really nice to have a new CPAN release for the module.

@Tux
Copy link
Owner

@Tux Tux commented on ed26ab6 Jun 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am in preparation phase. A real release for Text::CSV_XS takes a full working day: it shall be tested on ALL 180 versions of perl it supports, on all - to me - available architectures (x64, i365, PA-RISC, Itanium, powerpc, ...). It has to be run against the last 15 builds to secure I did not loose speed, and it has to be tested against all downriver modules that depend on it.
Expect 1.31 soonish, but for sure not before this weekend.

@Tux
Copy link
Owner

@Tux Tux commented on ed26ab6 Jun 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wilx released. Get it now that it is hot.

@wilx
Copy link

@wilx wilx commented on ed26ab6 Jun 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wilx released. Get it now that it is hot.

Thank you. The release fixes the issue for me.

Please sign in to comment.