locale.t on Win32 failing "Verify that strings with embedded NUL collate" and "Verify that strings with embedded NUL and extra trailing NUL collate"

[p5pRT]

Migrated from rt.perl.org#128629 (status was 'resolved')

Searchable as RT128629$

[p5pRT]

From @bulk88

Created by @bulk88

------------------------------------------------------
ok 410 Verify that isn't both [​:cntrl​:] and [​:print​:]
ok 411 Verify that isn't both [​:alpha​:] and [​:digit​:]
ok 412 Verify that isn't both [​:alnum​:] and [​:punct​:]
ok 413 Verify that isn't both [​:xdigit​:] and [​:punct​:]
ok 414 Verify that isn't both [​:graph​:] and [​:space​:]
ok 415 Verify that cmp works with a read-only scalar; no- vs locale
ok 416 Verify that "le", "ne", etc work
ok 417 Skip in locales where \001 has primary sorting weight; otherwise
verify that \0 doesn't have primary sorting weight
#
# For more details, rerun, with environment variable PERL_DEBUG_FULL_TEST=1.
# Then look at that output for lines that match 'failed 418'
not ok 418 Verify that strings with embedded NUL collate
#
# For more details, rerun, with environment variable PERL_DEBUG_FULL_TEST=1.
# Then look at that output for lines that match 'failed 419'
not ok 419 Verify that strings with embedded NUL and extra trailing NUL
collate
ok 420 Verify that empty strings collate
ok 421 Skip in non-UTF-8 locales; otherwise verify that UTF8ness doesn't
matter with collation
ok 422 Skip in UTF-8 locales; otherwise verify that single byte collates
before 0x100 and above
ok 423 Skip in UTF-8 locales; otherwise verify that 0x100 and above
collate in code point order
ok 424 Verify that an intervening printf doesn't change assignment results
ok 425 Verify that an intervening sprintf doesn't change assignment results
ok 426 Verify that a different locale radix works when doing "==" with a
constant
ok 427 Verify that a different locale radix works whe
---------------------------------------------------------------
ok 659 verify that is tainted
ok 660 fc("Ā") in C locale (use locale; encoded in utf8) should be "ā",
got "ā"
ok 661 verify that is tainted
ok 662 fc("") in C locale (use locale '​:not_characters'; not encoded in
utf8) should be "", got ""
ok 663 verify that isn't tainted
ok 664 fc("") in C locale (use locale '​:not_characters'; encoded in
utf8) should be "", got ""
ok 665 verify that isn't tainted
ok 666 fc("A") in C locale (use locale '​:not_characters'; not encoded in
utf8) should be "a", got "a"
ok 667 verify that isn't tainted
ok 668 fc("A") in C locale (use locale '​:not_characters'; encoded in
utf8) should be "a", got "a"
ok 669 verify that isn't tainted
ok 670 fc("À") in C locale (use locale '​:not_characters'; not encoded in
utf8) should be "à", got "à"
ok 671 verify that isn't tainted
ok 672 fc("À") in C locale (use locale '​:not_characters'; encoded in
utf8) should be "à", got "à"
ok 673 verify that isn't tainted
ok 674 fc("ſ") in C locale (use locale '​:not_characters'; not encoded in
utf8) should be "s", got "s"
ok 675 verify that isn't tainted
ok 676 fc("ſ") in C locale (use locale '​:not_characters'; encoded in
utf8) should be "s", got "s"
ok 677 verify that isn't tainted
ok 678 fc("Ā") in C locale (use locale '​:not_characters'; not encoded in
utf8) should be "ā", got "ā"
ok 679 verify that isn't tainted
ok 680 fc("Ā") in C locale (use locale '​:not_characters'; encoded in
utf8) should be "ā", got "ā"
ok 681 verify that isn't tainted
#
# The locale definitions
#
# Danish Norwegian da no no\@​nynorsk
#
# on your system may have errors because the locale test 418
# "Verify that strings with embedded NUL collate"
# failed in those locales.
#
# If your users are not using these locales you are safe for the moment,
# but please report this failure first to perlbug@​perl.com using the
# perlbug script (as described in the INSTALL file) so that the exact
# details of the failures can be sorted out first and then your operating
# system supplier can be alerted about these anomalies.
#
#
# The locale definitions
#
# Danish Norwegian da no no\@​nynorsk
#
# on your system may have errors because the locale test 419
# "Verify that strings with embedded NUL and extra trailing NUL collate"
# failed in those locales.
#
# If your users are not using these locales you are safe for the moment,
# but please report this failure first to perlbug@​perl.com using the
# perlbug script (as described in the INSTALL file) so that the exact
# details of the failures can be sorted out first and then your operating
# system supplier can be alerted about these anomalies.
#
# The following locales
#
# Afrikaans Albanian American Basque Bulgarian C Catalan
# Català Croatian Czech Dutch English Estonian Faroese Finnish
# French Føroyskt Galician German Greek Gáidhlig Hungarian
# Icelandic Indonesian Italian Latvian Lithuanian Polish
# Portuguese Português Russian Serbian Slovak Slovenian
# Spanish Swedish Sámi Turkish Welsh af ca en es fi fr fy ga
# hu it la mo po ro ru sl
#
# tested okay.
#
# The following locales
#
# Danish Norwegian da no no\@​nynorsk
#
# had problems.
#
# For more details, rerun, with environment variable PERL_DEBUG_FULL_TEST=1.
1..681
Failed 2/681 subtests

Test Summary Report
-------------------
../lib/locale.t (Wstat​: 0 Tests​: 681 Failed​: 2)
  Failed tests​: 418-419
Files=1, Tests=681, 9 wallclock secs ( 0.13 usr + 0.00 sys = 0.13 CPU)
Result​: FAIL
--------------------------------------------------------

PERL_DEBUG_FULL_TEST run attached.

Perl Info

Flags:
     category=library
     severity=medium
     module=locale

Site configuration information for perl 5.25.3:

Configured by Owner at Wed Jul 13 21:11:05 2016.

Summary of my perl5 (revision 5 version 25 subversion 3) configuration:

   Platform:
     osname=MSWin32
     osvers=5.1
     archname=MSWin32-x86-multi-thread
     uname=''
     config_args='undef'
     hint=recommended
     useposix=true
     d_sigaction=undef
     useithreads=define
     usemultiplicity=define
     use64bitint=undef
     use64bitall=undef
     uselongdouble=undef
     usemymalloc=n
     bincompat5005=undef
   Compiler:
     cc='cl'
     ccflags ='-nologo -GF -W3 -Od -MD -Zi -DDEBUGGING -DWIN32 -D_CONSOLE
-DNO_STRICT  -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT
-DPERL_IMPLICIT_SYS -DWIN32_NO_REGISTRY -DUSE_PERLIO'
     optimize='-Od -MD -Zi -DDEBUGGING'
     cppflags='-DWIN32'
     ccversion='13.10.6030'
     gccversion=''
     gccosandvers=''
     intsize=4
     longsize=4
     ptrsize=4
     doublesize=8
     byteorder=1234
     doublekind=3
     d_longlong=undef
     longlongsize=8
     d_longdbl=define
     longdblsize=8
     longdblkind=0
     ivtype='long'
     ivsize=4
     nvtype='double'
     nvsize=8
     Off_t='__int64'
     lseeksize=8
     alignbytes=8
     prototype=define
   Linker and Libraries:
     ld='link'
     ldflags ='-nologo -nodefaultlib -debug -libpath:"c:\perl\lib\CORE"
-machine:x86'
     libpth=\lib
     libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
     perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib
netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib
odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib
     libc=msvcrt.lib
     so=dll
     useshrplib=true
     libperl=perl525.lib
     gnulibc_version=''
   Dynamic Linking:
     dlsrc=dl_win32.xs
     dlext=dll
     d_dlsymun=undef
     ccdlflags=' '
     cccdlflags=' '
     lddlflags='-dll -nologo -nodefaultlib -debug
-libpath:"c:\perl\lib\CORE" -machine:x86'



@INC for perl 5.25.3:
     lib
     C:/perl521/src/lib
     .


Environment for perl 5.25.3:
     HOME (unset)
     LANG (unset)
     LANGUAGE (unset)
     LD_LIBRARY_PATH (unset)
     LOGDIR (unset)
     PATH=C:\sperl\c\bin;C:\WINDOWS\system32;C:\Program Files\Microsoft
Visual Studio .NET 2003\Vc7\bin;C:\Program Files\Microsoft Visual Studio
.NET 2003\Common7\IDE;C:\WINDOWS;C:\Program Files\Git\cmd;C:\Program
Files\Microsoft Visual Studio .NET 2003\Common7\Tools\bin;C:\perl\bin
     PERL_BADLANG (unset)
     PERL_JSON_BACKEND=Cpanel::JSON::XS
     PERL_YAML_BACKEND=YAML
     SHELL (unset)

[p5pRT]

From @bulk88

locale_t_full.txt

[p5pRT]

From @khwilliamson

On 07/15/2016 04​:18 PM, bulk88 (via RT) wrote​:

# New Ticket Created by bulk88
# Please include the string​: [perl #128629]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128629 >

This is a bug report for perl from bulk88@​hotmail.com,
generated with the help of perlbug 1.40 running under perl 5.25.3.

-----------------------------------------------------------------
[Please describe your issue here]

------------------------------------------------------
ok 410 Verify that isn't both [​:cntrl​:] and [​:print​:]
ok 411 Verify that isn't both [​:alpha​:] and [​:digit​:]
ok 412 Verify that isn't both [​:alnum​:] and [​:punct​:]
ok 413 Verify that isn't both [​:xdigit​:] and [​:punct​:]
ok 414 Verify that isn't both [​:graph​:] and [​:space​:]
ok 415 Verify that cmp works with a read-only scalar; no- vs locale
ok 416 Verify that "le", "ne", etc work
ok 417 Skip in locales where \001 has primary sorting weight; otherwise
verify that \0 doesn't have primary sorting weight
#
# For more details, rerun, with environment variable PERL_DEBUG_FULL_TEST=1.
# Then look at that output for lines that match 'failed 418'
not ok 418 Verify that strings with embedded NUL collate
#
# For more details, rerun, with environment variable PERL_DEBUG_FULL_TEST=1.
# Then look at that output for lines that match 'failed 419'
not ok 419 Verify that strings with embedded NUL and extra trailing NUL
collate
ok 420 Verify that empty strings collate

620f73f may have fixed this. Please
let me know.

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @bulk88

On Mon Jul 18 17​:56​:50 2016, public@​khwilliamson.com wrote​:

620f73f may have fixed this. Please
let me know.

It didn't. See attached.

--
bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From @bulk88

[log 9648eab Add epigraphs for 5.22.3-RC2 and 5.24.1-RC2.txt](https://rt-archive.perl.org/perl5/Ticket/Attachment/1412499/763073/log 9648eab Add epigraphs for 5.22.3-RC2 and 5.24.1-RC2.txt)

[p5pRT]

From @khwilliamson

On 07/25/2016 05​:57 PM, bulk88 via RT wrote​:

On Mon Jul 18 17​:56​:50 2016, public@​khwilliamson.com wrote​:

620f73f may have fixed this. Please
let me know.

It didn't. See attached.

Thanks for this. Please rerun it, adding -DLv

[p5pRT]

From @bulk88

On Mon Jul 25 17​:06​:42 2016, public@​khwilliamson.com wrote​:

On 07/25/2016 05​:57 PM, bulk88 via RT wrote​:

On Mon Jul 18 17​:56​:50 2016, public@​khwilliamson.com wrote​:

620f73f may have fixed this. Please
let me know.

It didn't. See attached.

Thanks for this. Please rerun it, adding -DLv

Recompiled perl with debugging and did -DLv see attached.

--
bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From @bulk88

[log 9648eab Add epigraphs for 5.22.3-RC2 and 5.24.1-RC2.txt](https://rt-archive.perl.org/perl5/Ticket/Attachment/1412529/763098/log 9648eab Add epigraphs for 5.22.3-RC2 and 5.24.1-RC2.txt)

[p5pRT]

From @bulk88

On Mon Jul 25 18​:01​:39 2016, bulk88 wrote​:

On Mon Jul 25 17​:06​:42 2016, public@​khwilliamson.com wrote​:

On 07/25/2016 05​:57 PM, bulk88 via RT wrote​:

On Mon Jul 18 17​:56​:50 2016, public@​khwilliamson.com wrote​:

620f73f may have fixed this. Please
let me know.

It didn't. See attached.

Thanks for this. Please rerun it, adding -DLv

Recompiled perl with debugging and did -DLv see attached.

7.8 MB of text in my attachment, yeah, that post was rejected from the ML...

--
bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From @khwilliamson

On 07/25/2016 07​:22 PM, bulk88 via RT wrote​:

On Mon Jul 25 18​:01​:39 2016, bulk88 wrote​:

On Mon Jul 25 17​:06​:42 2016, public@​khwilliamson.com wrote​:

On 07/25/2016 05​:57 PM, bulk88 via RT wrote​:

On Mon Jul 18 17​:56​:50 2016, public@​khwilliamson.com wrote​:

620f73f may have fixed this. Please
let me know.

It didn't. See attached.

Thanks for this. Please rerun it, adding -DLv

Recompiled perl with debugging and did -DLv see attached.

7.8 MB of text in my attachment, yeah, that post was rejected from the ML...

Thanks. This showed up two problems. One is that there was unintended
recalculations going on if something failed. That just slows things
down and isn't relevant to the bug at hand.

The other is that your platform's strxfrm() is returning a length that
includes a trailing NUL, and the code is not expecting that. The win32
VM on dromedary doesn't do that. So I've added code to check for and
compensate for platforms like yours. This is smoking at
http​://perl5.git.perl.org/perl.git/shortlog/refs/heads/smoke-me/khw-locale

I'm not sure if this will fix the problem. It would be helpful if you
could rerun the test on that branch. I've attached a modified version
of the test file which only runs a small subset of the tests so that the
output will be much smaller.

[p5pRT]

From @khwilliamson

locale.t

[p5pRT]

From @bulk88

On Tue Jul 26 15​:32​:16 2016, public@​khwilliamson.com wrote​:

The other is that your platform's strxfrm() is returning a length that
includes a trailing NUL, and the code is not expecting that. The
win32
VM on dromedary doesn't do that. So I've added code to check for and
compensate for platforms like yours. This is smoking at
http​://perl5.git.perl.org/perl.git/shortlog/refs/heads/smoke-me/khw-
locale

I'm not sure if this will fix the problem. It would be helpful if you
could rerun the test on that branch. I've attached a modified version
of the test file which only runs a small subset of the tests so that
the
output will be much smaller.

I'll build it later, for right now, i googled up this https://github.com/apache/stdcxx/blob/4.1.3/src/collate.cpp#L477 but I can't find the ticket associated with "29935" for that project. MSVC 2003 calls itself 1310, so based on "#if defined _MSC_VER && _MSC_VER < 1400" the bug was fixed in VC 2005 and newer.

--
bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From @bulk88

On Tue Jul 26 15​:32​:16 2016, public@​khwilliamson.com wrote​:

I'm not sure if this will fix the problem. It would be helpful if you
could rerun the test on that branch. I've attached a modified version
of the test file which only runs a small subset of the tests so that
the
output will be much smaller.

Your khw-locale branch fixes the test fail.

--
bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From @khwilliamson

On Tue Jul 26 19​:41​:54 2016, bulk88 wrote​:

On Tue Jul 26 15​:32​:16 2016, public@​khwilliamson.com wrote​:

I'm not sure if this will fix the problem. It would be helpful if you
could rerun the test on that branch. I've attached a modified version
of the test file which only runs a small subset of the tests so that
the
output will be much smaller.

Your khw-locale branch fixes the test fail.

Ok, but I'll need to look at the log, now that the other issues (should) have been cleaned up
--
Karl Williamson