cpan/Test-Simple/t/Legacy/More.t: frequent failures in non-threaded builds on FreeBSD-13 #16907
Comments
From @jkeenanWe are experiencing massive test failures in our smoke-testing of the cpan/Test-Simple/t/Legacy/More.t is repeatedly failing during http://perl5.test-smoke.org/report/82470 Test failures: We only have one regular smoke-testing rig set up for FreeBSD-13, that i. Do not seem to occur on threaded builds. ii. Seem to occur on most, but not all, non-threaded builds. iii. Appear to have first reported during a smoke-test in iv. Both Carlos and I have attempted to reproduce this failure manually v. Here's the end of the file which is failing: ##### # rt.cpan.org 53469 is_deeply with regexes # These two tests must remain at the end. Thank you very much. perl perl perl |
From @khwilliamsonOn 3/25/19 6:45 PM, James E Keenan (via RT) wrote:
Have you tried manually running this with address sanitizer? |
|
The RT System itself - Status changed from 'new' to 'open' |
From @jkeenanOn 3/27/19 11:15 PM, karl williamson via RT wrote:
1. I myself have never run address sanitizer. How would I do that? 2. Even if I did that, how would that address the problem that we've Thank you very much. |
From @khwilliamsonOn 3/28/19 6:21 AM, James E Keenan wrote:
Instructions are in perlhacktips. It's just some Configure options to
There are two likely causes for something showing up only in smoke
|
From @jkeenanOn 3/28/19 10:16 AM, Karl Williamson wrote:
I should note that in this environment $TEST_JOBS is set to 2. So $> cd t; ./perl harness ../ext/XS-APItest/t/handy00.t I could not reproduce the problem. But it seems to occur regularly
I will try to test with valgrind today or tomorrow.
|
From @jkeenanOn Thu, 28 Mar 2019 15:15:50 GMT, jkeenan@pobox.com wrote:
I mistyped. I meant to say that I would try to build a perl with AddressSanitizer. See below.
My efforts to build and test a perl at blead with AddressSanitizer in this FreeBSD-13 were largely unsuccessful. After struggling with the syntax of the ./Configure invocation, I was eventually able to get 'make' to complete successfully. See attachment with name starting with 'perl_V'. However, I could not run 'make test_harness' successfully. Normally, in these VMs I have $TEST_JOBS set to 2. However, 'make test_harness' was 'Killed' in the t/re/*.t tests. See attachment starting with 'test_harness'. I then set TEST_JOBS=1 and ran 'make test' -- so no tests were running in parallel. I got only a little farther on. See attachment starting with 'make_test'. Now, I concede: (a) I've never previously attempted to build with AddressSanitizer; (b) This VM may be too small to run a perl built with AddressSanitizer. But Carlos's FreeBSD-13 smoker, while not large, is larger than mine -- and we're still getting the failures on non-threaded builds in t/Legacy/More.t which are the subject of this RT. See, e.g., http://perl5.test-smoke.org/report/83974. Thank you very much. -- |
From @jkeenant/re/no_utf8_pm ................................................ ok Stop. |
From @jkeenanSummary of my perl5 (revision 5 version 29 subversion 10) configuration: Characteristics of this binary (from libperl): |
From @jkeenan./miniperl -Ilib make_ext.pl cpan/Archive-Tar/pm_to_blib MAKE="make" LIBPERL_A=libperl.a Making utilities Stop. |
From @iabynOn Tue, Apr 02, 2019 at 02:31:16PM -0700, James E Keenan via RT wrote:
Address Sanitizer needs lots of memory. My 16Gb system struggles to Just t/re/pat_psycho.t consumes about 1.7Gb on a threaded debugging build. -- |
From @jkeenanOn Tue, 02 Apr 2019 21:31:16 GMT, jkeenan wrote:
I have since built perl5 blead on FreeBSD-13 (i) unthreaded; (ii) without AddressSanitizer; (iii) $TEST_JOBS=1. In this manner: 1. In 'make test_harness' the last test in ../cpan/Test-Simple/t/Legacy/More.t FAILs. ##### 2. However, when, in the same build, I run that file manually, that file PASSes. ##### The above was built/tested at: ##### But when I build *threaded*, the test passes during 'make test_harness'. -- |
From @khwilliamsonOn 4/11/19 7:57 PM, James E Keenan via RT wrote:
One possibility is to bisect this. You don't have that big a range of |
From @jkeenanOn Fri, 12 Apr 2019 15:52:53 GMT, public@khwilliamson.com wrote:
We don't have a value for the '--target' switch to I did, however, try to bisect manually -- but before I get * A new test failure in a very old, cpan-upstream, test. In addition, it appears that when the error does appear, it ##### Since the error is intermittent, it's quite likely that a So, over the past few days, I have built and run the test ##### regcomp.c: Use mnemonic for flag parameter ### PATCH: [perl #131551] Too deep regex compilation recursion This patch, started by Yves Orton, and refined in The patch adds a variable that can be set to increase or Why cannot I not say which of these was the first bad I don't understand the internals that were modified in those The problem persists. See, e.g., Thank you very much. -- |
From @khwilliamsonOn 4/14/19 1:59 PM, James E Keenan via RT wrote:
We could make a shell script that called make test and then returned
It's much more likely to be 6ef7fe5.
|
From @jkeenanOn Sun, 14 Apr 2019 20:22:59 GMT, public@khwilliamson.com wrote:
The branch idea sounds good, as Carlos's smokers test branches. But, as you anticipated, a simple 'git revert' yields conflicts. See http://paste.scsys.co.uk/583957. If you let me know how the conflicts should be resolved, I'll push the branch. Thank you very much. -- |
From @jkeenanOn Sun, 14 Apr 2019 20:22:59 GMT, public@khwilliamson.com wrote:
Unfortunately, reverting the second commit did not clear up the FAIL: ##### I will now try to revert only the first of the two commits in question. Thank you very much. |
From @iabynOn Sun, Apr 14, 2019 at 03:49:22PM -0700, James E Keenan via RT wrote:
The test is failing because the process's errno ($!) has not being #include <stdio.h> int main(int argc, char**argv) errno = 12; Failing that, you should be able to find error code 12 in the file #define ENOMEM 12 /* Out of memory */ It's possible that linux and BSD share the same basic error code numbers, -- |
From @jkeenanOn Wed, 24 Apr 2019 14:23:06 GMT, davem wrote:
On FreeBSD-13: ##### [c] $ cat 133958-errno.c int main(int argc, char**argv) errno = 12; [c] $ cc --version [c] $ cc 133958-errno.c -o 133958-errno [c] $ ./133958-errno [c] $ cd /usr/local/include [include] $ ack '\t(12|42)\t' /usr/include/sys/errno.h On Linux: ##### So there is no error '42' on Linux -- or at least not in /usr/include/asm-generic/errno-base.h. I wonder if, when cpan/Test-Simple/t/Legacy/More.t, was added back in 2002, '42' was considered to be a sufficiently "high" error number as to be suitable for use in testing as an impossible-to-achieve value. Note also that error number 42 is not a new one for FreeBSD-13. On FreeBSD-11.2: ##### |
From @iabynOn Wed, Apr 24, 2019 at 08:10:19AM -0700, James E Keenan via RT wrote:
The error number 42 is irrelevant. It's just an arbitrary but humorous The fact that its getting changed to ENOMEM is more interesting. This However, experimenting with 'ulimit -d', as I decrease the available I'm only seeing this test script fail on one smoker, Test-Simple was upgraded in v5.29.7-102-ga6afdf72cd |
From @jkeenanOn Thu, 25 Apr 2019 08:13:03 GMT, davem wrote:
As an experiment, I created a branch in which I reverted that upgrade to Test-Simple. I didn't really expect it to clear up the anomalous test failures on non-threaded builds on FreeBSD-13, and it did not. See http://perl.develop-help.com/?b=smoke-me%2Fjkeenan%2Frt133958-revert-a6afdf72cd. So we can rule out Test-Simple as the cause of the problem. Thank you very much. -- |
From @iabynOn Sat, Apr 27, 2019 at 06:02:20AM -0700, James E Keenan via RT wrote:
Are you able to increase the amount of virtual memory available on the -- |
From @jkeenanOn Mon, 29 Apr 2019 08:46:09 GMT, davem wrote:
There is a complex procedure for increasing the size of a VM -- complex enough that I think it can only be done when setting up a VM in the first place (and complex enough that I have somebody else do it). I doubt there's a procedure for increasing the amount of memory available for just one test. And, in any event, this is a problem that is intermittent (but frequent) and appears when tests are being run in parallel.
On my VM: ##### IIRC on the FreeBSD-13 VM Carlos uses for smoke-testing, it's 2048MB. -- |
From @khwilliamsonOn 4/29/19 5:57 AM, James E Keenan via RT wrote:
You'd think 2G would be enough. Would using $^M help in debugging this situation at all? |
From @khwilliamsonOn 4/29/19 5:57 AM, James E Keenan via RT wrote:
Another option to rule out that its locale handling, is to add The thread safe locale handling is usually only selected on threaded |
From @khwilliamsonOn 4/29/19 5:57 AM, James E Keenan via RT wrote:
Tony Cook set me up a VM of FreeBSD 13 with 4096MB. I ran the test io/through.t ....................................................... ok plus several more like that. So something is getting errno 12, and printing out the associated That makes it seem likely that its the smaller memory size available in But the question becomes, why is this happening now, and not earlier, I ran through the test suite with clang asan enabled, and got no errors, I tried to do leak detection, but it says: ==64995==AddressSanitizer: detect_leaks is not supported on this platform |
From @tonycozOn Mon, 25 Mar 2019 17:45:17 -0700, jkeenan@pobox.com wrote:
Does the attached fix this for you? I was only able to reproduce the messages from re/fold_grind_a.t, but from the mechanism of the failure (unexpected changes to errno) I'm pretty sure this will solve it for More.t too. Tony |
From @tonycoz0001-perl-133958-preserve-errno-on-successful-malloc-real.patchFrom 30dd168e6e502186d2de002acaf77323162c97ff Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Fri, 3 May 2019 14:49:50 +1000
Subject: (perl #133958) preserve errno on successful malloc/realloc
In general perl doesn't try to preserve errno (aka $!) since we're
aiming at the same behaviour as for C code - errno is only meaningful
if a function returned an error.
The exception to that is when perl is working without an explicit
request from the perl programmer.
When code is performing assignments, concatenating strings, pushing on
arrays etc, perl is exercising the memory allocation machinery,
calling malloc() and realloc().
It turns out that at least on one platform, realloc() can modify errno
on success.
It appears to be happening when jemalloc (the malloc() implementation
used on FreeBSD) tries to extend a memory arena and fails, leaving the
error number from that failure in errno, from truss:
mmap(0x80142f000,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON|MAP_EXCL,-1,0x0) ERR#12 'Cannot allocate memory'
This magic call appears to be a FreeBSD specific mechanism to resize
the anonymous mapping. On Linux the equivalent seems to be calling
mremap().
In each case for the test code mmap() is successfully called
immediately afterwards:
mmap(0x0,69632,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34390323200 (0x801d2b000)
and realloc() succeeds.
glibc() realloc seems to be simpler, AFAICT from reading the code it
only uses mremap() when the memory block is the entire mapping,
ie. for large blocks rather than for memory arenas, and it doesn't
request the same address, so it doesn't fail.
For blocks that are part of arenas, glibc tries to expand in-place
within the current arena (with no extending the arena itself) or falls
back to malloc, so there's no chance for errno to be changed on a
successful realloc().
---
util.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/util.c b/util.c
index 7276fd901d..27b221837c 100644
--- a/util.c
+++ b/util.c
@@ -132,6 +132,7 @@ Perl_safesysmalloc(MEM_SIZE size)
dTHX;
#endif
Malloc_t ptr;
+ dSAVEDERRNO;
#ifdef USE_MDH
if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size)
@@ -143,6 +144,7 @@ Perl_safesysmalloc(MEM_SIZE size)
Perl_croak_nocontext("panic: malloc, size=%" UVuf, (UV) size);
#endif
if (!size) size = 1; /* malloc(0) is NASTY on our system */
+ SAVE_ERRNO;
#ifdef PERL_DEBUG_READONLY_COW
if ((ptr = mmap(0, size, PROT_READ|PROT_WRITE,
MAP_ANON|MAP_PRIVATE, -1, 0)) == MAP_FAILED) {
@@ -154,6 +156,11 @@ Perl_safesysmalloc(MEM_SIZE size)
#endif
PERL_ALLOC_CHECK(ptr);
if (ptr != NULL) {
+ /* malloc() can modify errno() even on success, but since someone
+ writing perl code doesn't have any control over when perl calls
+ malloc() we need to hide that.
+ */
+ RESTORE_ERRNO;
#ifdef USE_MDH
struct perl_memory_debug_header *const header
= (struct perl_memory_debug_header *)ptr;
@@ -223,6 +230,7 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size)
ptr = safesysmalloc(size);
}
else {
+ dSAVE_ERRNO;
#ifdef USE_MDH
where = (Malloc_t)((char*)where-PERL_MEMORY_DEBUG_HEADER_SIZE);
if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size)
@@ -276,6 +284,11 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size)
might allocate memory/free/move memory, and until we do the fixup, it
may well be chasing (and writing to) free memory. */
if (ptr != NULL) {
+ /* realloc() can modify errno() even on success, but since someone
+ writing perl code doesn't have any control over when perl calls
+ realloc() we need to hide that.
+ */
+ RESTORE_ERRNO;
#ifdef PERL_TRACK_MEMPOOL
struct perl_memory_debug_header *const header
= (struct perl_memory_debug_header *)ptr;
--
2.21.0
|
From @tonycozOn Thu, 02 May 2019 22:11:19 -0700, tonyc wrote:
Karl did most of the work in tracking this down. Tony |
From @hvdsOn Thu, 02 May 2019 22:11:19 -0700, tonyc wrote:
I think you'll need to protect the 'header' declaration from the non-declaration above it, and similarly in the second RESTORE case. Hugo |
From @tonycozOn Fri, May 03, 2019 at 02:57:20AM -0700, Hugo van der Sanden via RT wrote:
Oops, yep. Tony |
From @jkeenanOn Fri, 03 May 2019 10:02:35 GMT, tonyc wrote:
Tony, when you revise your patch to reflect Hugo's comments above, can you push it to a smoke-me branch? That will be easier for me to test in my FreeBSD-13 VM. Thank you very much. -- |
From @khwilliamsonOn 5/3/19 5:40 AM, James E Keenan via RT wrote:
I made the changes suggested by Hugo, and pushed the result to I can confirm that this getst rid of the spurious messages in #134076 |
From @dur-randirOn Thu, 02 May 2019 22:11:19 -0700, tonyc wrote:
Shouldn't there be a new configure probe for a broken malloc, so systems without such behaviour do not suffer performance penalty? |
From @jkeenanOn Fri, 03 May 2019 15:08:59 GMT, public@khwilliamson.com wrote:
Now I'm confused. I've already started smoking the smoke-me/tonyc/133958-realloc-errno-success branch. How does that differ from the smoke-me/khw-freebsd branch? Thank you very much. -- |
From @khwilliamsonOn 5/3/19 9:24 AM, James E Keenan via RT wrote:
I didn't know Tony had pushed a new branch. There wasn't a reply on the |
From @tonycozOn Fri, 03 May 2019 04:40:11 -0700, jkeenan wrote:
As you saw, I pushed it last night and this morning here's the patch for the record. Tony |
From @tonycoz0001-perl-133958-preserve-errno-on-successful-malloc-real.patchFrom 373d744e311ee3721ab7a4ee2bd8a33b88a7f387 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Fri, 3 May 2019 14:49:50 +1000
Subject: (perl #133958) preserve errno on successful malloc/realloc
In general perl doesn't try to preserve errno (aka $!) since we're
aiming at the same behaviour as for C code - errno is only meaningful
if a function returned an error.
The exception to that is when perl is working without an explicit
request from the perl programmer.
When code is performing assignments, concatenating strings, pushing on
arrays etc, perl is exercising the memory allocation machinery,
calling malloc() and realloc().
It turns out that at least on one platform, realloc() can modify errno
on success.
It appears to be happening when jemalloc (the malloc() implementation
used on FreeBSD) tries to extend a memory arena and fails, leaving the
error number from that failure in errno, from truss:
mmap(0x80142f000,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON|MAP_EXCL,-1,0x0) ERR#12 'Cannot allocate memory'
This magic call appears to be a FreeBSD specific mechanism to resize
the anonymous mapping. On Linux the equivalent seems to be calling
mremap().
In each case for the test code mmap() is successfully called
immediately afterwards:
mmap(0x0,69632,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34390323200 (0x801d2b000)
and realloc() succeeds.
glibc() realloc seems to be simpler, AFAICT from reading the code it
only uses mremap() when the memory block is the entire mapping,
ie. for large blocks rather than for memory arenas, and it doesn't
request the same address, so it doesn't fail.
For blocks that are part of arenas, glibc tries to expand in-place
within the current arena (with no extending the arena itself) or falls
back to malloc, so there's no chance for errno to be changed on a
successful realloc().
---
util.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/util.c b/util.c
index 7276fd901d..e5320c8b8c 100644
--- a/util.c
+++ b/util.c
@@ -132,6 +132,7 @@ Perl_safesysmalloc(MEM_SIZE size)
dTHX;
#endif
Malloc_t ptr;
+ dSAVEDERRNO;
#ifdef USE_MDH
if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size)
@@ -143,6 +144,7 @@ Perl_safesysmalloc(MEM_SIZE size)
Perl_croak_nocontext("panic: malloc, size=%" UVuf, (UV) size);
#endif
if (!size) size = 1; /* malloc(0) is NASTY on our system */
+ SAVE_ERRNO;
#ifdef PERL_DEBUG_READONLY_COW
if ((ptr = mmap(0, size, PROT_READ|PROT_WRITE,
MAP_ANON|MAP_PRIVATE, -1, 0)) == MAP_FAILED) {
@@ -182,6 +184,11 @@ Perl_safesysmalloc(MEM_SIZE size)
ptr = (Malloc_t)((char*)ptr+PERL_MEMORY_DEBUG_HEADER_SIZE);
DEBUG_m(PerlIO_printf(Perl_debug_log, "0x%" UVxf ": (%05ld) malloc %ld bytes\n",PTR2UV(ptr),(long)PL_an++,(long)size));
+ /* malloc() can modify errno() even on success, but since someone
+ writing perl code doesn't have any control over when perl calls
+ malloc() we need to hide that.
+ */
+ RESTORE_ERRNO;
}
else {
#ifdef USE_MDH
@@ -223,6 +230,7 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size)
ptr = safesysmalloc(size);
}
else {
+ dSAVE_ERRNO;
#ifdef USE_MDH
where = (Malloc_t)((char*)where-PERL_MEMORY_DEBUG_HEADER_SIZE);
if (size + PERL_MEMORY_DEBUG_HEADER_SIZE < size)
@@ -296,6 +304,12 @@ Perl_safesysrealloc(Malloc_t where,MEM_SIZE size)
maybe_protect_ro(header->prev);
#endif
ptr = (Malloc_t)((char*)ptr+PERL_MEMORY_DEBUG_HEADER_SIZE);
+
+ /* realloc() can modify errno() even on success, but since someone
+ writing perl code doesn't have any control over when perl calls
+ realloc() we need to hide that.
+ */
+ RESTORE_ERRNO;
}
/* In particular, must do that fixup above before logging anything via
--
2.21.0
|
From @tonycozOn Fri, 03 May 2019 08:20:06 -0700, randir wrote:
Such behaviour is permitted by the C standard. From the specification for errno (C11): The value of errno may be set to nonzero by a library function call and the standard doesn't document any behaviour for errno for realloc(), malloc(), so the behaviour is permitted. Similarly POSIX says for errno: The setting of errno after a successful call to a function is unspecified unless the description of that function specifies that errno shall not be modified. and there's nothing in the documentation of realloc() guaranteeing errno is unchanged on success. Tony |
|
@tonycoz - Status changed from 'open' to 'pending release' |
From @khwilliamsonThank you for filing this report. You have helped make Perl better. With the release today of Perl 5.30.0, this and 160 other issues have been Perl 5.30.0 may be downloaded via: If you find that the problem persists, feel free to reopen this ticket. |
|
@khwilliamson - Status changed from 'pending release' to 'resolved' |
Migrated from rt.perl.org#133958 (status was 'resolved')
Searchable as RT133958$
The text was updated successfully, but these errors were encountered: