[meta] Win32 Unicode support

[p5pRT]

Migrated from rt.perl.org#134286 (status was 'open')

Searchable as RT134286$

[p5pRT]

From @tonycoz

Created by @tonycoz

Perl typically uses so called "ANSI" APIs, due to compatibility
with other platforms, eg. calling unlink() is cross-platform, and
to a certain extent works in Unicode locales (using the encoded
bytes rather than characters), but the equivalents on Win32 don't
work.

chcp 65001 isn't a solution, since pretty much everything but
console output will still be in the system code page.

So this is a meta ticket covering tickets improving Perl's Win32
Unicode support.

Any changes need to be switchable (so a user with old ANSI
encoded filenames for example don't lose a bunch of work), and
need to not break other platforms.

I can see at least the following issues​:

- command line arguments

- filenames across many different operators

- process creation (system, exec, readpipe/qx(), pipe open())

- environment variables

- console (maybe)

- many similar changes in bundled modules

Discussion is welcome, patches that satsify the above
requirements more so.

Perl Info

Flags:
    category=core
    severity=low

Site configuration information for perl 5.28.1:

Configured by strawberry-perl at Sun Dec  2 14:25:09 2018.

Summary of my perl5 (revision 5 version 28 subversion 1) configuration:
   
  Platform:
    osname=MSWin32
    osvers=10.0.17134.407
    archname=MSWin32-x64-multi-thread
    uname='Win32 strawberry-perl 5.28.1.1 #1 Sun Dec  2 14:24:00 2018 x64'
    config_args='undef'
    hint=recommended
    useposix=true
    d_sigaction=undef
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=undef
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='gcc'
    ccflags =' -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -D__USE_MINGW_ANSI_STDIO -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fwrapv -fno-strict-aliasing -mms-bitfields'
    optimize='-s -O2'
    cppflags='-DWIN32'
    ccversion=''
    gccversion='7.1.0'
    gccosandvers=''
    intsize=4
    longsize=4
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='long long'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='g++.exe'
    ldflags ='-s -L"C:\sperl-5.28.1.1-64bit-portable\perl\lib\CORE" -L"C:\sperl-5.28.1.1-64bit-portable\c\lib"'
    libpth=C:\sperl-5.28.1.1-64bit-portable\c\lib C:\sperl-5.28.1.1-64bit-portable\c\x86_64-w64-mingw32\lib C:\sperl-5.28.1.1-64bit-portable\c\lib\gcc\x86_64-w64-mingw32\7.1.0
    libs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    perllibs= -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
    libc=
    so=dll
    useshrplib=true
    libperl=libperl528.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs
    dlext=xs.dll
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags='-mdll -s -L"C:\sperl-5.28.1.1-64bit-portable\perl\lib\CORE" -L"C:\sperl-5.28.1.1-64bit-portable\c\lib"'



@INC for perl 5.28.1:
    c:/sperl-5.28.1.1-64bit-portable/perl/site/lib
    c:/sperl-5.28.1.1-64bit-portable/perl/vendor/lib
    c:/sperl-5.28.1.1-64bit-portable/perl/lib


Environment for perl 5.28.1:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=c:\sperl-5.28.1.1-64bit-portable\perl\site\bin;c:\sperl-5.28.1.1-64bit-portable\perl\bin;c:\sperl-5.28.1.1-64bit-portable\c\bin;C:\Program Files\Microsoft MPI\Bin\;C:\Program Files (x86)\Common Files\Intel\Shared Files\cpp\bin\Intel64;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files\Git\cmd;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Users\Tony\AppData\Local\Microsoft\WindowsApps;C:\Users\Tony\AppData\Local\GitHubDesktop\bin
    PERL_BADLANG (unset)
    SHELL (unset)

[p5pRT]

From bokutin@bokut.in

I've been waiting.

Python and Ruby are already possible.

I think it will not be helpful, I have built in the past by myself.
https://github.com/bokutin/strawberry-perl-USING_WIDE-revival

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @pali

I can see at least the following issues​:
...
- filenames across many different operators
- process creation (system, exec, readpipe/qx(), pipe open())
...
Discussion is welcome, patches that satsify the above
requirements more so.

Same problem is also on Linux systems. I already created ticket
https://rt.perl.org/Public/Bug/Display.html?id=130831
where is discussion how to handle. I proposed some solution (which could
work for both Windows and Linux systems) but seems other people do not
like it.

(PS​: Please CC me for future discussion as I do not know how can I add
myself to CC list on RT)