Can't get Unicode to work with backticks

[stoecker]

Passing commandline arguments in UTF-8 via backticks seems not to work at all.

The command
perl -e 'use utf8; print qx/host stöcker.eu/;'
always returns "Host stöcker.eu not found: 3(NXDOMAIN)" instead of the correct answer (which is interesting, as the output is readable, so the encoding somehow seems to match, but not fully).

The similar command
perl -e 'use utf8; system("host stöcker.eu");'
works as expected.

I tried many different variants with -C argument on commandline or use open ":std", ":encoding(UTF-8)", using encode("utf-8", ...) and so on. Nothing really changes the behaviour.

Following line works, but that's no option as this is only a reduced example of a longer script which needs proper utf-8.
perl -e 'print qx/host stöcker.eu/;'

I searched many websites and documentation, but found no switch or setting to change it, so I think it's a bug.

Perl configuration

I'm using Perl 5.38 on openSUSE Tumbleweed (also tried perl 5.34 on Ubuntu):

Summary of my perl5 (revision 5 version 38 subversion 2) configuration:
   
  Platform:
    osname=linux
    osvers=6.7.5-1-default
    archname=x86_64-linux-thread-multi
    uname='reproducible'
    config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Dd_dbm_open -Duseshrplib=true -Doptimize=-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -g -Wall -pipe -Duse64bitint -Dotherlibdirs=/usr/lib/perl5/site_perl -Dinc_version_list=5.38.0/x86_64-linux-thread-multi 5.38.0'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='cc'
    ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='-O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -g -Wall -pipe'
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong'
    ccversion=''
    gccversion='13.2.1 20240206 [revision 67ac78caf31f7cb3202177e6428a46d829b70f23]'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -L/usr/local/lib64 -fstack-protector-strong'
    libpth=/usr/local/lib /usr/x86_64-suse-linux/lib /usr/lib /usr/lib64 /usr/local/lib64
    libs=-lm -ldl -lcrypt -lpthread
    perllibs=-lm -ldl -lcrypt -lpthread
    libc=/lib64//lib64/libc.so.6
    so=so
    useshrplib=true
    libperl=libperl.so
    gnulibc_version='2.39'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.38.2/x86_64-linux-thread-multi/CORE'
    cccdlflags='-fPIC'
    lddlflags='-shared -L/usr/local/lib64 -fstack-protector-strong'


Characteristics of this binary (from libperl): 
  Compile-time options:
    HAS_LONG_DOUBLE
    HAS_STRTOLD
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_HASH_FUNC_SIPHASH13
    PERL_HASH_USE_SBOX32
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_USE_SAFE_PUTENV
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
    USE_REENTRANT_API
    USE_THREAD_SAFE_LOCALE
  Built under linux
  Compiled at Nov 27 2023 00:00:00
  %ENV:
    PERL5LIB="/home/stoecker/sources/multinav/linux/modules/perl"
  @INC:
    /home/stoecker/sources/multinav/linux/modules/perl
    /usr/lib/perl5/site_perl/5.38.2/x86_64-linux-thread-multi
    /usr/lib/perl5/site_perl/5.38.2
    /usr/lib/perl5/vendor_perl/5.38.2/x86_64-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.38.2
    /usr/lib/perl5/5.38.2/x86_64-linux-thread-multi
    /usr/lib/perl5/5.38.2
    /usr/lib/perl5/vendor_perl/5.38.0/x86_64-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.38.0
    /usr/lib/perl5/site_perl

[xenu]

This isn't Perl's fault.

host is weird and enables IDN support only if the stdout is a TTY.

This can be reproduced in a shell:

> host stöcker.eu
xn--stcker-xxa.eu has address 78.46.162.30
xn--stcker-xxa.eu has IPv6 address 2a01:4f8:c17:15d5:0:1:1:99
xn--stcker-xxa.eu mail is handled by 10 mail.stoecker.eu.
> host stöcker.eu | cat
Host stöcker.eu not found: 3(NXDOMAIN)

Consider using something like Net::DNS instead.

PS. If you really have to do this, there are a few ways to simulate a TTY.

[stoecker]

Ah. Given the description I agree it's not a perl problem.

Though it is not only host, but also dig and nslookup (well, all from same vendor :-). Didn't find any resolver tool which works. Arrgh I had some tty/non-tty issues in the past, but that's really an insane non-obvious behavior.

Created a new bug: https://gitlab.isc.org/isc-projects/bind9/-/issues/4637

[stoecker]

P.S. Thanks for the fast and helpful response. That's really seldom.

[stoecker]

P.P.S. The bind ticket shows what's the normal reaction I receive for tickets: That's not a bug! That's intended! Use it this (also non working) way! Was so refreshing to get a proper reaction once even if it only pointed to the real problem.

I do so many upstream bugreports for openSUSE packaging and it's tiring. It's no wonder many packagers don't report fixes upstream at all...