Tied filehandles not working with XS-based I/O

[p5pRT]

Migrated from rt.perl.org#125981 (status was 'resolved')

Searchable as RT125981$

[p5pRT]

From jdv@base2bio.com

Created by jdv@base2bio.com

Tied filehandles don't appear to be working as expected in combination
with modules using XS-based I/O (e.g. PerlIO_read). For example,
passing a tied filehandle to the core Digest​::MD5​::addfile() croaks
with 'No filehandle passed'. If the filehandle is first opened on one
file before being tied and opened on another file, the XS-based class
will read from the first file opened and not pass read() calls through
to the tied class' READ method. Plugging in a pure-Perl module instead
works as expected.

Discussed at https://www.perlmonks.org/?node_id=1140411

A MWE example follows. The same behavior is observed using other
Tie​::Handle​:: classes from CPAN and other XS-based modules which take
filehandles as input.

## BEGIN CODE

#!/usr/bin/perl

package TieTest;
use Symbol;

sub TIEHANDLE { return bless gensym(), shift; }
sub OPEN { open $_[0], $_[1], $_[2]; }
sub READ { warn "...tied read\n"; read $_[0], $_[1], $_[2],
$_[3]; } sub SEEK { seek $_[0], $_[1], $_[2]; }

package main;

use strict;
use warnings;
use Digest​::MD5;
use Digest​::Perl​::MD5;

my ($testfile_0, $testfile_1) = @​ARGV;

open FH_PLAIN, '<', $testfile_0;

#open FH_TIED, '<', $testfile_1; # try uncommenting this
tie(*FH_TIED, 'TieTest');
open FH_TIED, '<', $testfile_0;

my %handles = (PLAIN => \*FH_PLAIN, TIED => \*FH_TIED);
for my $title (qw/PLAIN TIED/) {

  my $fh = $handles{$title};
  for my $class (qw/Digest​::Perl​::MD5 Digest​::MD5/) {

  my $sum = eval {$class-&gt;new-&gt;addfile($fh)-&gt;hexdigest} || $@​;
  printf "%-7s%-19s%-32s\n", $title, $class, $sum;
  seek $fh, 0, 0;
 
  }

}

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.20.2:

Configured by Debian Project at Sun May  3 16:16:25 UTC 2015.

Summary of my perl5 (revision 5 version 20 subversion 2) configuration:
   
  Platform:
    osname=linux, osvers=3.2.0-4-amd64,
archname=x86_64-linux-gnu-thread-multi uname='linux x86-csail-01
3.2.0-4-amd64 #1 smp debian 3.2.68-1+deb7u1 x86_64 gnulinux '
config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN
-D_FORTIFY_SOURCE=2 -g -O2 -fstack-protector-strong -Wformat
-Werror=format-security -Dldflags= -Wl,-z,relro -Dlddlflags=-shared
-Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu
-Dprefix=/usr -Dprivlib=/usr/share/perl/5.20
-Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.20 -Dvendorprefix=/usr
-Dvendorlib=/usr/share/perl5
-Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.20
-Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.20.2
-Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.20.2
-Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3
-Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3
-Duse64bitint -Dman1ext=1 -Dman3ext=3perl
-Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio
-Uusenm -Ui_libutil -Uversiononly -DDEBUGGING=-g -Doptimize=-O2
-Duseshrplib -Dlibperl=libperl.so.5.20.2 -des' hint=recommended,
useposix=true, d_sigaction=define useithreads=define,
usemultiplicity=define use64bitint=define, use64bitall=define,
uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv
-fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64', optimize='-O2 -g', cppflags='-D_REENTRANT
-D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe
-I/usr/local/include' ccversion='', gccversion='4.9.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8 alignbytes=8, prototype=define Linker and Libraries:
ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=-ldl -lm -lpthread -lc -lcrypt libc=libc-2.19.so, so=so,
useshrplib=true, libperl=libperl.so.5.20 gnulibc_version='2.19' Dynamic
Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef,
ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared
-L/usr/local/lib -fstack-protector'

Locally applied patches:
    DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS
default for modules installed from CPAN. DEBPKG:debian/db_file_ver -
http://bugs.debian.org/340047 Remove overly restrictive DB_File version
check. DEBPKG:debian/doc_info - Replace generic man(1) instructions
with Debian-specific information. DEBPKG:debian/enc2xs_inc -
http://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and
ignore missing @INC directories. DEBPKG:debian/errno_ver -
http://bugs.debian.org/343351 Remove Errno version check due to upgrade
problems with long-running processes. DEBPKG:debian/libperl_embed_doc -
http://bugs.debian.org/186778 Note that libperl-dev package is required
for embedded linking DEBPKG:fixes/respect_umask - Respect umask during
installation DEBPKG:debian/writable_site_dirs - Set umask approproately
for site install directories DEBPKG:debian/extutils_set_libperl_path -
EU:MM: set location of libperl.a under /usr/lib
DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or
perllocal.pod for perl or vendor DEBPKG:debian/prefix_changes - Fiddle
with *PREFIX and variables written to the makefile
DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the
binary targets. DEBPKG:debian/instmodsh_doc - Debian policy doesn't
install .packlist files for core or vendor. DEBPKG:debian/ld_run_path -
Remove standard libs from LD_RUN_PATH as per Debian policy.
DEBPKG:debian/libnet_config_path - Set location of libnet.cfg
to /etc/perl/Net as /usr may not be writable. DEBPKG:debian/mod_paths -
Tweak @INC ordering for Debian
DEBPKG:debian/module_build_man_extensions -
http://bugs.debian.org/479460 Adjust Module::Build manual page
extensions for the Debian Perl policy DEBPKG:debian/prune_libs -
http://bugs.debian.org/128355 Prune the list of libraries wanted to
what we actually need. DEBPKG:fixes/net_smtp_docs - [rt.cpan.org
#36038] http://bugs.debian.org/100195 Document the Net::SMTP 'Port'
option DEBPKG:debian/perlivp - http://bugs.debian.org/510895 Make
perlivp skip include directories in /usr/local
DEBPKG:debian/deprecate-with-apt - http://bugs.debian.org/747628 Point
users to Debian packages of deprecated core modules
DEBPKG:debian/squelch-locale-warnings - http://bugs.debian.org/508764
Squelch locale warnings in Debian package maintainer scripts
DEBPKG:debian/skip-upstream-git-tests - Skip tests specific to the
upstream Git repository DEBPKG:debian/patchlevel -
http://bugs.debian.org/567489 List packaged patches for 5.20.2-3+deb8u1
in patchlevel.h DEBPKG:debian/skip-kfreebsd-crash -
http://bugs.debian.org/628493 [perl #96272] Skip a crashing test case
in t/op/threads.t on GNU/kFreeBSD
DEBPKG:fixes/document_makemaker_ccflags - http://bugs.debian.org/628522
[rt.cpan.org #68613] Document that CCFLAGS should include
$Config{ccflags} DEBPKG:debian/find_html2text -
http://bugs.debian.org/640479 Configure CPAN::Distribution with correct
name of html2text DEBPKG:debian/perl5db-x-terminal-emulator.patch -
http://bugs.debian.org/668490 Invoke x-terminal-emulator rather than
xterm in perl5db.pl DEBPKG:debian/cpan-missing-site-dirs -
http://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with
nonexisting site dirs if a parent is writable
DEBPKG:fixes/memoize_storable_nstore - [rt.cpan.org #77790]
http://bugs.debian.org/587650 Memoize::Storable: respect 'nstore'
option not respected DEBPKG:debian/regen-skip - Skip a regeneration
check in unrelated git repositories DEBPKG:fixes/regcomp-mips-optim -
[perl #122817] http://bugs.debian.org/754054 Downgrade the optimization
of regcomp.c on mips and mipsel due to a gcc-4.9 bug
DEBPKG:debian/makemaker-pasthru - http://bugs.debian.org/758471 Pass LD
settings through to subdirectories DEBPKG:fixes/perldoc-less-R -
[rt.cpan.org #98636] http://bugs.debian.org/758689 Tell the 'less'
pager to allow terminal escape sequences
DEBPKG:fixes/pod_man_reproducible_date - http://bugs.debian.org/759405
Support POD_MAN_DATE in Pod::Man for the left-hand footer
DEBPKG:fixes/io_uncompress_gunzip_inmemory -
http://bugs.debian.org/747363 [rt.cpan.org #95494] Fix gunzip to
in-memory file handle DEBPKG:fixes/socket_test_recv_fix -
http://bugs.debian.org/758718 [perl #122657] Compare recv return value
to peername in socket test DEBPKG:fixes/hurd_socket_recv_todo -
http://bugs.debian.org/758718 [perl #122657] TODO checking the result
of recv() on hurd DEBPKG:fixes/regexp-performance - [0fa70a0]
http://bugs.debian.org/777556 [perl #123743] simpify and speed
up /.*.../ handling DEBPKG:fixes/failed_require_diagnostics -
http://bugs.debian.org/781120 [perl #123270] Report inaccesible file on
failed require DEBPKG:fixes/array-cloning -
http://bugs.debian.org/779357 [perl #124127] [902d169] fix cloning
arrays with unused elements DEBPKG:fixes/perldb-threads -
http://bugs.debian.org/779357 [perl #124127] [41ef2c6] lib/perl5db.pl:
Restore noop lock prototype


@INC for perl 5.20.2:
    /home/jeremy/perl5/lib/perl5/x86_64-linux-gnu-thread-multi
    /home/jeremy/perl5/lib/perl5
    /etc/perl
    /usr/local/lib/x86_64-linux-gnu/perl/5.20.2
    /usr/local/share/perl/5.20.2
    /usr/lib/x86_64-linux-gnu/perl5/5.20
    /usr/share/perl5
    /usr/lib/x86_64-linux-gnu/perl/5.20
    /usr/share/perl/5.20
    /usr/local/lib/site_perl
    .


Environment for perl 5.20.2:
    HOME=/home/jeremy
    LANG=en_US.utf8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/jeremy/.local/bin:/home/jeremy/.gem/ruby/1.9.1/bin:/home/jeremy/perl5/bin:/opt/bin
    PERL5LIB=/home/jeremy/perl5/lib/perl5
    PERL_BADLANG (unset)
    PERL_MB_OPT=--install_base "/home/jeremy/perl5"
    PERL_MM_OPT=INSTALL_BASE=/home/jeremy/perl5
    SHELL=/bin/bash

[p5pRT]

From @tonycoz

On Thu Sep 03 15​:50​:47 2015, jdv@​base2bio.com wrote​:

Tied filehandles don't appear to be working as expected in combination
with modules using XS-based I/O (e.g. PerlIO_read). For example,
passing a tied filehandle to the core Digest​::MD5​::addfile() croaks
with 'No filehandle passed'. If the filehandle is first opened on one
file before being tied and opened on another file, the XS-based class
will read from the first file opened and not pass read() calls through
to the tied class' READ method. Plugging in a pure-Perl module instead
works as expected.

Discussed at https://www.perlmonks.org/?node_id=1140411

A MWE example follows. The same behavior is observed using other
Tie​::Handle​:: classes from CPAN and other XS-based modules which take
filehandles as input.

This isn't a bug.

PerlIO is the layer below the I/O you see at the perl level, by the time you have a PerlIO * value, the tied information (stored in the GV) isn't available.

If you do want your interface visible at the PerlIO level you'll need to implement it as a PerlIO layer, possibly using PerlIO​::via.

Unfortunately there's no simple XS interface for performing I/O on a GV that honors tie() - I've done it by switching between a callback that uses PerlIO vs a callback that uses call_sv() to call back to perl subs that print/read/seek as needed. Which won't help you for Digest​::MD5.

Tony

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @tonycoz

On Sun Sep 13 18​:50​:34 2015, tonyc wrote​:

This isn't a bug.

Closing.

Tony

[p5pRT]

@tonycoz - Status changed from 'open' to 'resolved'