Newline handling is broken on Windows

[p6rt]

Migrated from rt.perl.org#126881 (status was 'resolved')

Searchable as RT126881$

[p6rt]

From @nanis

64-bit Windows 10.0.10586. 64-bit perl6 on MoarVM built using MS
Visual Studio 2013 tools. See below for more details.

I noticed[1] that several tests fail if the test script has Unix line
endings instead of Windows. These tests succeed if the test script is
converted using unix2dos.

This is problematic behavior. Tests should pass or fail independently
of the EOL convention used in the script file.

E.g. see test 20 in t\spec\S02-literals\heredocs.t​:

  my $e = q​:to<END>;

END
  is no-r($e), "\n", 'Heredoc one empty line';

This fails with​:

# Failed test 'Heredoc one empty line'
# at t\spec\S02-literals\heredocs.t line 171
# expected​: "\r\n"
# got​: "\n"

That is, when the script source uses Unix line endings, $e contains
"\n", but, because the test is being run on Windows, the comparison
string, "\n" becomes "\r\n". Converting the file with unix2dos fixes
the failures, but that is not a solution. "\n" should not be converted
to "\r\n" behind the scenes in a program. EOL conversion should only
happen at input-output boundaries.

What happens in this and other tests that fail depending on whether
the test script is saved using Unix or Windows line endings is
contrary to decades of convention of doing EOL translation only at
input-output boundaries.

For more exploration, see

[1]​: https://www.nu42.com/2015/12/perl6-newline-translation-botched.html

moar​::syslibs[6]=iphlpapi
moar​::dcbrule=@​​:
moar​::ar=lib
moar​::mastdir=C​:\opt\perl6\share\nqp\lib\MAST
moar​::tomrule=$(AR) $(ARFLAGS) /out​:$@​ 3rdparty\libtommath\*.obj
moar​::syslibs[2]=mswsock
moar​::shaobjects=3rdparty\sha1\sha1.obj
moar​::ccshared=
moar​::be=0
moar​::dynasmlua=.\3rdparty\dynasm\dynasm.lua
moar​::ld=link
moar​::mtlib=3rdparty\tinymt\tinymt.lib
moar​::mtobjects=3rdparty\tinymt\tinymt64.obj
moar​::noreturnspecifier=__declspec(noreturn)
moar​::perl=c​:\opt\perl\5.23.5\bin\perl.exe
moar​::shaclean=$(RM) 3rdparty\sha1\sha1.lib 3rdparty\sha1\*.obj
moar​::static_inline=static __inline
moar​::ccdebugflags=/Zi
moar​::auxclean=$(RM) moar.ilk moar.pdb moar.dll.lib moar.dll.exp vc100.pdb
moar​::thirdpartylibs=3rdparty\dyncall\dyncall\libdyncall_s.lib
3rdparty\dyncall\dyncallback\libdyncallback_s.lib
3rdparty\dyncall\dynload\libdynload_s.lib 3rdparty\tinymt\tinymt.lib
3rdparty\sha1\sha1.lib 3rdparty\libtommath\tommath.lib
3rdparty\libuv\uv.lib
moar​::config=--optimize --prefix=C​:\opt\perl6 --make-install
moar​::ccinc=/I
moar​::ldout=/out​:
moar​::moarlib=moar.lib
moar​::mtrule=$(AR) $(ARFLAGS) /out​:$@​ 3rdparty\tinymt\*.obj
moar​::syslibs[0]=shell32
moar​::dcrule=cd 3rdparty\dyncall && configure.bat /target-x64 &&
$(MAKE) -f Nmakefile
moar​::dcbobjects=
moar​::cflags=/nologo /MT /Ox /GL /DNDEBUG /DWIN32 /DAO_ASSUME_WINDOWS98
moar​::laolib=__lao__
moar​::libdir=C​:\opt\perl6\bin
moar​::nul=NUL
moar​::osvers=10.0
moar​::dlrule=@​​:
moar​::cat=type
moar​::can_unaligned_int32=1
moar​::ldimp=%s.dll.lib
moar​::win32_compiler_toolchain=win32
moar​::ccdef=/D
moar​::lib=%s.lib
moar​::tomclean=$(RM) 3rdparty\libtommath\tommath.lib 3rdparty\libtommath\*.obj
moar​::dlllocal=
moar​::dcbclean=$(RM) 3rdparty\dyncall\dyncallback\libdyncallback_s.lib
moar​::noreturnattribute=
moar​::ptr_size=8
moar​::syslibs[5]=psapi
moar​::dllimport=__declspec(dllimport)
moar​::ccout=/Fo
moar​::can_unaligned_num64=1
moar​::has_pthread_yield=0
moar​::ldrpath=
moar​::ldusr=%s.lib
moar​::mainflags=/DMVM_SHARED
moar​::name=moar
moar​::dll=%s.dll
moar​::cincludes= /I3rdparty\libuv\include /I3rdparty\libuv\src
/I3rdparty\libatomic_ops\src /I3rdparty\libtommath /I3rdparty\dynasm
/I3rdparty\dyncall\dynload /I3rdparty\dyncall\dyncall
/I3rdparty\dyncall\dyncallback
moar​::uvrule=$(AR) $(ARFLAGS) /out​:$@​ 3rdparty\libuv\src\*.obj
3rdparty\libuv\src\win\*.obj
moar​::tomlib=3rdparty\libtommath\tommath.lib
moar​::syslibs[3]=rpcrt4
moar​::staticlib=
moar​::dlclean=$(RM) 3rdparty\dyncall\dynload\libdynload_s.lib
moar​::make=nmake
moar​::ccinstflags=
moar​::ldflags=/nologo /LTCG
moar​::crossconf=
moar​::obj=.obj
moar​::dlobjects=
moar​::defs[0]=WIN32
moar​::cppout=/Fi
moar​::ccoptiflags=/Ox /GL /DNDEBUG
moar​::asmswitch=/c /FAs
moar​::ldoptiflags=/LTCG
moar​::mkflags=/nologo
moar​::moar=moar.dll
moar​::version=2015.11
moar​::dcclean=$(RM) 3rdparty\dyncall\ConfigVars
3rdparty\dyncall\dyncall\libdyncall_s.lib
3rdparty\dyncall\dyncallback\libdyncallback_s.lib
3rdparty\dyncall\dynload\libdynload_s.lib
3rdparty\dyncall\dyncall\*.obj 3rdparty\dyncall\dyncallback\*.obj
3rdparty\dyncall\dynload\*.obj
moar​::moarshared=/implib​:moar.dll.lib
moar​::sharule=$(AR) $(ARFLAGS) /out​:$@​ 3rdparty\sha1\*.obj
moar​::ccwarnflags=
moar​::jit=$(JIT_WIN32_X64)
moar​::moardll=moar.dll
moar​::asm=.asm
moar​::mknoisy=!IF $(NOISY) != 1MSG = @​echoCMD = @​NOOUT = > NULNOERR =
2> NUL!ENDIF
moar​::dclib=3rdparty\dyncall\dyncall\libdyncall_s.lib
moar​::ccdefflags=/DWIN32 /DAO_ASSUME_WINDOWS98
moar​::cc=cl
moar​::exe=.exe
moar​::lddir=/libpath​:
moar​::uvlib=3rdparty\libuv\uv.lib
moar​::dllib=3rdparty\dyncall\dynload\libdynload_s.lib
moar​::shaincludedir=3rdparty\sha1
moar​::versionmajor=2015
moar​::cppswitch=/P
moar​::can_unaligned_int64=1
moar​::lua=.\3rdparty\dynasm\minilua.exe
moar​::dcobjects=
moar​::cancgoto=0
moar​::ldlibs=shell32.lib ws2_32.lib mswsock.lib rpcrt4.lib
advapi32.lib psapi.lib iphlpapi.lib
moar​::mainlibs=/libpath​:. moar.dll.lib
moar​::nativecall_backend=dyncall
moar​::objflags=/DMVM_BUILD_SHARED
moar​::versionpatch=0
moar​::laoclean=@​​:
moar​::versionminor=11
moar​::ccmiscflags=/nologo /MT
moar​::install= $(MKPATH) $(DESTDIR)$(PREFIX)\include\libuv $(CP)
3rdparty\libuv\include\*.h $(DESTDIR)$(PREFIX)\include\libuv
$(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\armcc
  $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\gcc
  $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\hpc
  $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\ibmc
  $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\icc
  $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\loadstore
  $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\msftc
  $(MKPATH) $(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\sunc
  $(CP) 3rdparty\libatomic_ops\src\*.h
$(DESTDIR)$(PREFIX)\include\libatomic_ops $(CP)
3rdparty\libatomic_ops\src\atomic_ops\*.h
$(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops $(CP)
3rdparty\libatomic_ops\src\atomic_ops\sysdeps\*.h
$(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps $(CP)
3rdparty\libatomic_ops\src\atomic_ops\sysdeps\armcc\*.h
$(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\armcc
$(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\gcc\*.h
$(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\gcc
$(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\hpc\*.h
$(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\hpc
$(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\ibmc\*.h
$(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\ibmc
$(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\icc\*.h
$(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\icc
$(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\loadstore\*.h
$(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\loadstore
  $(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\msftc\*.h
$(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\msftc
$(CP) 3rdparty\libatomic_ops\src\atomic_ops\sysdeps\sunc\*.h
$(DESTDIR)$(PREFIX)\include\libatomic_ops\atomic_ops\sysdeps\sunc
$(CP) 3rdparty\libtommath\*.h $(DESTDIR)$(PREFIX)\include\libtommath
$(MKPATH) $(DESTDIR)$(PREFIX)\include\dyncall $(CP)
3rdparty\dyncall\dynload\*.h $(DESTDIR)$(PREFIX)\include\dyncall
$(CP) 3rdparty\dyncall\dyncall\*.h $(DESTDIR)$(PREFIX)\include\dyncall
  $(CP) 3rdparty\dyncall\dyncallback\*.h
$(DESTDIR)$(PREFIX)\include\dyncall
moar​::laoobjects=
moar​::mtclean=$(RM) 3rdparty\tinymt\tinymt.lib 3rdparty\tinymt\*.obj
moar​::rm=del
moar​::uvclean=$(RM) 3rdparty\libuv\uv.lib 3rdparty\libuv\src\*.obj
3rdparty\libuv\src\win\*.obj
moar​::sharedlib=moar.dll.lib
moar​::dllexport=__declspec(dllexport)
moar​::shalib=3rdparty\sha1\sha1.lib
moar​::arout=/out​:
moar​::lddebugflags=/debug /pdb​:$@​.pdb
moar​::osname=MSWin32
moar​::prefix=C​:\opt\perl6
moar​::uvobjects=3rdparty\libuv\src\fs-poll.obj
3rdparty\libuv\src\inet.obj 3rdparty\libuv\src\threadpool.obj
3rdparty\libuv\src\uv-common.obj 3rdparty\libuv\src\version.obj
3rdparty\libuv\src\win\async.obj 3rdparty\libuv\src\win\core.obj
3rdparty\libuv\src\win\dl.obj 3rdparty\libuv\src\win\error.obj
3rdparty\libuv\src\win\fs-event.obj 3rdparty\libuv\src\win\fs.obj
3rdparty\libuv\src\win\getaddrinfo.obj
3rdparty\libuv\src\win\getnameinfo.obj
3rdparty\libuv\src\win\handle.obj
3rdparty\libuv\src\win\loop-watcher.obj
3rdparty\libuv\src\win\pipe.obj 3rdparty\libuv\src\win\poll.obj
3rdparty\libuv\src\win\process-stdio.obj
3rdparty\libuv\src\win\process.obj 3rdparty\libuv\src\win\req.obj
3rdparty\libuv\src\win\signal.obj 3rdparty\libuv\src\win\stream.obj
3rdparty\libuv\src\win\tcp.obj 3rdparty\libuv\src\win\thread.obj
3rdparty\libuv\src\win\timer.obj 3rdparty\libuv\src\win\tty.obj
3rdparty\libuv\src\win\udp.obj 3rdparty\libuv\src\win\util.obj
3rdparty\libuv\src\win\winapi.obj 3rdparty\libuv\src\win\winsock.obj
moar​::dcblib=3rdparty\dyncall\dyncallback\libdyncallback_s.lib
moar​::bindir=C​:\opt\perl6\bin
moar​::asmout=/Fa
moar​::laorule=@​​:
moar​::os=win32
moar​::sh=cmd
moar​::arflags=/nologo
moar​::ldshared=/dll
moar​::defs[1]=AO_ASSUME_WINDOWS98
moar​::impinst=moar.dll.lib
moar​::ldinstflags=/Profile
moar​::ldmiscflags=/nologo
moar​::ldsys=%s.lib
moar​::platform=$(PLATFORM_WIN32)
moar​::ccswitch=/c
moar​::formatattribute=
moar​::tomobjects=3rdparty\libtommath\bn_error.obj
3rdparty\libtommath\bn_fast_mp_invmod.obj
3rdparty\libtommath\bn_fast_mp_montgomery_reduce.obj
3rdparty\libtommath\bn_fast_s_mp_mul_digs.obj
3rdparty\libtommath\bn_fast_s_mp_mul_high_digs.obj
3rdparty\libtommath\bn_fast_s_mp_sqr.obj
3rdparty\libtommath\bn_mp_2expt.obj 3rdparty\libtommath\bn_mp_abs.obj
3rdparty\libtommath\bn_mp_add.obj 3rdparty\libtommath\bn_mp_add_d.obj
3rdparty\libtommath\bn_mp_addmod.obj 3rdparty\libtommath\bn_mp_and.obj
3rdparty\libtommath\bn_mp_clamp.obj
3rdparty\libtommath\bn_mp_clear.obj
3rdparty\libtommath\bn_mp_clear_multi.obj
3rdparty\libtommath\bn_mp_cmp.obj 3rdparty\libtommath\bn_mp_cmp_d.obj
3rdparty\libtommath\bn_mp_cmp_mag.obj
3rdparty\libtommath\bn_mp_cnt_lsb.obj
3rdparty\libtommath\bn_mp_copy.obj
3rdparty\libtommath\bn_mp_count_bits.obj
3rdparty\libtommath\bn_mp_div.obj 3rdparty\libtommath\bn_mp_div_2.obj
3rdparty\libtommath\bn_mp_div_2d.obj
3rdparty\libtommath\bn_mp_div_3.obj
3rdparty\libtommath\bn_mp_div_d.obj
3rdparty\libtommath\bn_mp_dr_is_modulus.obj
3rdparty\libtommath\bn_mp_dr_reduce.obj
3rdparty\libtommath\bn_mp_dr_setup.obj
3rdparty\libtommath\bn_mp_exch.obj
3rdparty\libtommath\bn_mp_expt_d.obj
3rdparty\libtommath\bn_mp_exptmod.obj
3rdparty\libtommath\bn_mp_exptmod_fast.obj
3rdparty\libtommath\bn_mp_exteuclid.obj
3rdparty\libtommath\bn_mp_fread.obj
3rdparty\libtommath\bn_mp_fwrite.obj 3rdparty\libtommath\bn_mp_gcd.obj
3rdparty\libtommath\bn_mp_get_int.obj
3rdparty\libtommath\bn_mp_get_long.obj
3rdparty\libtommath\bn_mp_grow.obj 3rdparty\libtommath\bn_mp_init.obj
3rdparty\libtommath\bn_mp_init_copy.obj
3rdparty\libtommath\bn_mp_init_multi.obj
3rdparty\libtommath\bn_mp_init_set.obj
3rdparty\libtommath\bn_mp_init_set_int.obj
3rdparty\libtommath\bn_mp_init_size.obj
3rdparty\libtommath\bn_mp_invmod.obj
3rdparty\libtommath\bn_mp_invmod_slow.obj
3rdparty\libtommath\bn_mp_is_square.obj
3rdparty\libtommath\bn_mp_jacobi.obj
3rdparty\libtommath\bn_mp_karatsuba_mul.obj
3rdparty\libtommath\bn_mp_karatsuba_sqr.obj
3rdparty\libtommath\bn_mp_lcm.obj 3rdparty\libtommath\bn_mp_lshd.obj
3rdparty\libtommath\bn_mp_mod.obj 3rdparty\libtommath\bn_mp_mod_2d.obj
3rdparty\libtommath\bn_mp_mod_d.obj
3rdparty\libtommath\bn_mp_montgomery_calc_normalization.obj
3rdparty\libtommath\bn_mp_montgomery_reduce.obj
3rdparty\libtommath\bn_mp_montgomery_setup.obj
3rdparty\libtommath\bn_mp_mul.obj 3rdparty\libtommath\bn_mp_mul_2.obj
3rdparty\libtommath\bn_mp_mul_2d.obj
3rdparty\libtommath\bn_mp_mul_d.obj
3rdparty\libtommath\bn_mp_mulmod.obj
3rdparty\libtommath\bn_mp_n_root.obj 3rdparty\libtommath\bn_mp_neg.obj
3rdparty\libtommath\bn_mp_or.obj
3rdparty\libtommath\bn_mp_prime_fermat.obj
3rdparty\libtommath\bn_mp_prime_is_divisible.obj
3rdparty\libtommath\bn_mp_prime_is_prime.obj
3rdparty\libtommath\bn_mp_prime_miller_rabin.obj
3rdparty\libtommath\bn_mp_prime_next_prime.obj
3rdparty\libtommath\bn_mp_prime_rabin_miller_trials.obj
3rdparty\libtommath\bn_mp_prime_random_ex.obj
3rdparty\libtommath\bn_mp_radix_size.obj
3rdparty\libtommath\bn_mp_radix_smap.obj
3rdparty\libtommath\bn_mp_rand.obj
3rdparty\libtommath\bn_mp_read_radix.obj
3rdparty\libtommath\bn_mp_read_signed_bin.obj
3rdparty\libtommath\bn_mp_read_unsigned_bin.obj
3rdparty\libtommath\bn_mp_reduce.obj
3rdparty\libtommath\bn_mp_reduce_2k.obj
3rdparty\libtommath\bn_mp_reduce_2k_l.obj
3rdparty\libtommath\bn_mp_reduce_2k_setup.obj
3rdparty\libtommath\bn_mp_reduce_2k_setup_l.obj
3rdparty\libtommath\bn_mp_reduce_is_2k.obj
3rdparty\libtommath\bn_mp_reduce_is_2k_l.obj
3rdparty\libtommath\bn_mp_reduce_setup.obj
3rdparty\libtommath\bn_mp_rshd.obj 3rdparty\libtommath\bn_mp_set.obj
3rdparty\libtommath\bn_mp_set_int.obj
3rdparty\libtommath\bn_mp_set_long.obj
3rdparty\libtommath\bn_mp_shrink.obj
3rdparty\libtommath\bn_mp_signed_bin_size.obj
3rdparty\libtommath\bn_mp_sqr.obj 3rdparty\libtommath\bn_mp_sqrmod.obj
3rdparty\libtommath\bn_mp_sqrt.obj 3rdparty\libtommath\bn_mp_sub.obj
3rdparty\libtommath\bn_mp_sub_d.obj
3rdparty\libtommath\bn_mp_submod.obj
3rdparty\libtommath\bn_mp_to_signed_bin.obj
3rdparty\libtommath\bn_mp_to_signed_bin_n.obj
3rdparty\libtommath\bn_mp_to_unsigned_bin.obj
3rdparty\libtommath\bn_mp_to_unsigned_bin_n.obj
3rdparty\libtommath\bn_mp_toom_mul.obj
3rdparty\libtommath\bn_mp_toom_sqr.obj
3rdparty\libtommath\bn_mp_toradix.obj
3rdparty\libtommath\bn_mp_toradix_n.obj
3rdparty\libtommath\bn_mp_unsigned_bin_size.obj
3rdparty\libtommath\bn_mp_xor.obj 3rdparty\libtommath\bn_mp_zero.obj
3rdparty\libtommath\bn_prime_tab.obj
3rdparty\libtommath\bn_reverse.obj 3rdparty\libtommath\bn_s_mp_add.obj
3rdparty\libtommath\bn_s_mp_exptmod.obj
3rdparty\libtommath\bn_s_mp_mul_digs.obj
3rdparty\libtommath\bn_s_mp_mul_high_digs.obj
3rdparty\libtommath\bn_s_mp_sqr.obj
3rdparty\libtommath\bn_s_mp_sub.obj 3rdparty\libtommath\bncore.obj
moar​::syslibs[4]=advapi32
moar​::syslibs[1]=ws2_32
perl6​::name=rakudo
perl6​::codename=
perl6​::release-number=
perl6​::build-date=2015-12-10T20​:31​:31Z
perl6​::version=2015.11

[p6rt]

From @jnthn

On Fri Dec 11 09​:31​:44 2015, nanis@​cpan.org wrote​:

64-bit Windows 10.0.10586. 64-bit perl6 on MoarVM built using MS
Visual Studio 2013 tools. See below for more details.

I noticed[1] that several tests fail if the test script has Unix line
endings instead of Windows. These tests succeed if the test script is
converted using unix2dos.

Also was fine if you have autoclrf set in Git, fwiw.

This is problematic behavior. Tests should pass or fail independently
of the EOL convention used in the script file.

E.g. see test 20 in t\spec\S02-literals\heredocs.t​:

my $e = q​:to<END>;

END
is no-r($e), "\n", 'Heredoc one empty line';

This fails with​:

# Failed test 'Heredoc one empty line'
# at t\spec\S02-literals\heredocs.t line 171
# expected​: "\r\n"
# got​: "\n"

That is, when the script source uses Unix line endings, $e contains
"\n", but, because the test is being run on Windows, the comparison
string, "\n" becomes "\r\n". Converting the file with unix2dos fixes
the failures, but that is not a solution. "\n" should not be converted
to "\r\n" behind the scenes in a program. EOL conversion should only
happen at input-output boundaries.

What happens in this and other tests that fail depending on whether
the test script is saved using Unix or Windows line endings is
contrary to decades of convention of doing EOL translation only at
input-output boundaries.

For more exploration, see

[1]​: https://www.nu42.com/2015/12/perl6-newline-translation-
botched.html

The logical \n in strings (mirroring logical \n in regexes) didn't work out so well in reality as it did in theory. Now​:

* We do translation of \r\n graphemes to \n on all input read as text except sockets, independent of platform
* We do translation of all \n graphemes to \r\n on text output to handles except sockets, on Windows only
* \n is now, unless `use newline` is in force, always \x0A
* We don't do any such translation when using .encode/.decode, and of course when reading/writing Bufs to files, providing an escape hatch from translation if needed

This hopefully addresses the various issues. Various tests have been updated accordingly.

[p6rt]

The RT System itself - Status changed from 'new' to 'open'

[p6rt]

@jnthn - Status changed from 'open' to 'resolved'