Update Valgrind spkg to version 3.8.1

[jpflori]

Valgrind 3.7.0 configure script complains about glibc version 2.15 (which is shipped e.g. with Ubuntu 12.04).

The usual solution is to treat this version as the previous one:
see commit r12323 in the valgrind svn, or tickets on different distribution bug tracker.

Furthermore, the autogen.sh scripts fails with recent automake version.
This is commit r12396.

All these fixes are in Valgrind 3.8.1.

Try spkg at
http://boxen.math.washington.edu/home/jpflori/spkg/valgrind-3.8.1.p0.spkg

Support is announced upstream for OS X from 10.6.X onward, but building Valgrind might fail if you're using a non Apple gcc (such as the FSF one in Sage's spkg).

Upstream: Fixed upstream, but not in a stable release.

CC: @gvol @jpflori @haraldschilly

Component: packages: optional

Keywords: valgrind spkg

Author: Jean-Pierre Flori

Reviewer: Volker Braun

Issue created by migration from https://trac.sagemath.org/ticket/13060

[jpflori]

comment:1

After applying the diff of the above commit, one has to rebuild the configure scripts with autogen.sh (which is available in the svn, not in the usual tarballs).
This basically calls auto* stuff, but the current input *.am files are incompatible with the current automake tool...
See r12396 in valgrind svn.

[jpflori]

comment:2

Here comes a slightly updated spkg:
http://perso.telecom-paristech.fr/~flori/sage/valgrind-3.7.0.p0.spkg

I've applied the two mentioned patches and rerun autogen.sh to regenerate the build system files.
The package installs and seems functional with my Sage installation.

Another solution whould be to properly package a recent valgrind svn version, although I'd prefer to use this light solution and wait for a proper stable release to make a real update to valgrind.
Feel free to disagree and rant here.

[jpflori]

Attachment: trac_13060-spkg.diff.gz

Spkg diff, for review only.

[jpflori]

comment:3

There might be something wrong with valgrind testsuite.
I'm looking into this right now.

[jpflori]

comment:4

In fact, the current spkg is not functional at all.

It builds correctly, but fails to pass its test suite and Sage does not launch with the -valgrind flag.

[jpflori]

comment:5

The startup problem seem to be caused by strlen being inlined and its symbol not found by valgrind which fails to start.

[jpflori]

Work Issues: testsuite fails

[jpflori]

Author: Jean-Pierre Flori

[jpflori]

comment:6

Installing the debug version of libc6 (libc6-dbg) on my Ubuntu installation solves both the startup problem.
Nonetheless, it might be useful to actually print something when valgrind fails for this reason.
Currently running "sage -valgrind" exits without complaining although tunning directly valgrind from the local/bin folder provides some hints.

[jpflori]

comment:7

On a debian installation on top of recent hardware (corei7-avx) I'm able to install the old spkg (debian currently ships eglibc 2.13), but get a SIGILL when launching sage, which comes as no surprise:
https://bugs.kde.org/show_bug.cgi?id=273475

The latest svn version is enough to let sage starts.

On the same install, the test suite with the old spkg yields

== 599 tests, 77 stderr failures, 56 stdout failures, 0 stderrB failures, 1 stdoutB failure, 3 post failures ==

the one proposed here yields

== 599 tests, 77 stderr failures, 56 stdout failures, 0 stderrB failures, 1 stdo
utB failure, 3 post failures ==

which was expected.
With the latest svn version (you can test it by replace p0 by jp in the above url):

== 545 tests, 4 stderr failures, 0 stdout failures, 0 stderrB failures, 1 stdo
utB failure, 0 post failures ==

On the ubuntu install mentioned above, with the

== 591 tests, 79 stderr failures, 57 stdout failures, 0 stderrB failures, 0 stdo
utB failure, 3 post failures ==

and with the latest svn version

== 534 tests, 4 stderr failures, 1 stdout failures, 0 stderrB failures, 1 stdo
utB failure, 0 post failures ==

Whatsoever, I'm not sure that the problem of dealing with avx instructions should be dealt with here.
We surely should wait for a proper release by the valgrind team, the tested development version is completely arbitrary.

About the test suite failure, I'm not convinced it should prevent including the updated p0 spkg proposed here, if they were present before.
I must admit I do not remember seeing errors when dealing with #7766, but we potentially forgot to run the testsuite all along.

If we expect these failures to be fixed, I can definitely not take care of this with my little knowledge of valgrind.

[gvol]

comment:8

Replying to @jpflori:

About the test suite failure, I'm not convinced it should prevent including the updated p0 spkg proposed here, if they were present before.
I must admit I do not remember seeing errors when dealing with #7766, but we potentially forgot to run the testsuite all along.

I know I ran the test suite last time at least once, but I think I saw a few failures on my Mac. Now trying to install it on Sage 5.0 neither will even build. I should have thought to build it when I was testing the betas. The old spkg gives

...
make  all-recursive
Making all in include
make[2]: Nothing to be done for `all'.
Making all in VEX
make  all-am
gcc -DHAVE_CONFIG_H -I. -I..  -I.. -I../include -I../VEX/pub -DVGA_amd64=1 -DVGO_darwin=1 -DVGP_amd64_darwin=1 -DVGPV_amd64_darwin_vanilla=1 -Ipriv   -arch x86_64 -O2 -g -Wall -Wmissing-prototypes -Wshadow -Wpointer-arith -Wstrict-prototypes -Wmissing-declarations -Wno-format-zero-length -fno-strict-aliasing -fno-builtin -mmacosx-version-min=10.5 -fno-stack-protector -Wbad-function-cast -Wcast-qual -Wcast-align -fstrict-aliasing -Wno-long-long  -Wno-pointer-sign -fno-stack-protector -MT libvex_amd64_darwin_a-main_globals.o -MD -MP -MF .deps/libvex_amd64_darwin_a-main_globals.Tpo -c -o libvex_amd64_darwin_a-main_globals.o `test -f 'priv/main_globals.c' || echo './'`priv/main_globals.c
gcc: error: x86_64: No such file or directory
gcc: error: unrecognized option ‘-arch’
make[3]: *** [libvex_amd64_darwin_a-main_globals.o] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
Error building Valgrind 3.7.0
...

So it looks like the former is a problem with the new gcc package:

sage -sh -c 'which gcc'
/Users/gvol/SageStuff/sage-5.0.rc0/local/bin/gcc

And the spkg for review here gives.

...
checking for a sed that does not truncate output... /usr/bin/sed
checking for perl... /usr/bin/perl
checking for gdb... /usr/bin/gdb
checking dependency style of gcc... gcc3
checking for diff -u... yes
checking for a supported version of gcc... ok (4.6.3)
configure: error: cannot run /bin/sh ./config.sub
Error configuring Valgrind 3.7.0
...

Here config.sub is a symlink to /usr/share/automake-1.11/config.sub.
This probably has something to do with you running ./autogen.sh, but I'm not an expert in how to fix that. If I run ./autgen.sh myself then it configures fine but runs into the -arch problem as before.

[jpflori]

comment:9

Replying to @gvol:

Here config.sub is a symlink to /usr/share/automake-1.11/config.sub.
This probably has something to do with you running ./autogen.sh, but I'm not an expert in how to fix that. If I run ./autgen.sh myself then it configures fine but runs into the -arch problem as before.

Oh, I didn't realize that...
I've uploaded a new spkg obtained with autoreconf -fi rather than autogen.sh which has no symlinks.

I'll look into the arch stuff but have no apple stuff to test it.

[jpflori]

comment:10

Just a reminder for myself: indeed the apple gcc supports -arch, but the fsf one does not.

[gvol]

comment:11

Replying to @jpflori:

Replying to @gvol:

Here config.sub is a symlink to /usr/share/automake-1.11/config.sub.
This probably has something to do with you running ./autogen.sh, but I'm not an expert in how to fix that. If I run ./autgen.sh myself then it configures fine but runs into the -arch problem as before.

Oh, I didn't realize that...
I've uploaded a new spkg obtained with autoreconf -fi rather than autogen.sh which has no symlinks.

Cool, thanks. I just tried it (in the same place) and got the same error though. Perhaps, it's in a different place?

I'll look into the arch stuff but have no apple stuff to test it.

I think if you can force sage to use the gcc spkg it would exhibit the same problem, unless you are already using it. What does sage -sh -c 'which gcc' return?

It might be worth asking Jeroen (or on sage-devel) since he knows it a lot better.

[jpflori]

comment:12

The spkg is up now, I was a little slow to upload it.

About the gcc -arch problem, I think the problem is with Valgrind configure script which adds this flag as soon as it detects that its run on OSX because it assumes that someone on OSX will be using an Apple gcc.
That's the reason why I get no problem on Linux with the FSF gcc, the Ubuntu one or the Sage one.

[jpflori]

comment:13

Oh and valgrind.org seems down, so I don't have access to their svn anymore to see if anythig was commited about this gcc thingy lately.

We should maybe look into what Fink does:
http://osdir.com/ml/fink-users-osx-unix/2011-11/msg00168.html
although once more I'm no Apple user so don't have access or am used to such things.

[gvol]

comment:14

Replying to @jpflori:

The spkg is up now, I was a little slow to upload it.

No problem. It works now—except for actually building. :-)

About the gcc -arch problem, I think the problem is with Valgrind configure script which adds this flag as soon as it detects that its run on OSX because it assumes that someone on OSX will be using an Apple gcc.
That's the reason why I get no problem on Linux with the FSF gcc, the Ubuntu one or the Sage one.

Ah, now I see it. I guess we'll have to patch that, or ask them to do it upstream.

Fink simply disabled mpicc http://fink.cvs.sourceforge.net/fink/dists/10.7/stable/main/finkinfo/devel/valgrind.info?view=markup which was the FSF compiler rather than gcc.

[jpflori]

comment:15

Valgrind website is back up.
After a quick search trhough the svn log and the bug tracker, I found nothing related to the -arch flag.
I guess we have to report upstream and/or provide a patch to the configure/build system.

[jpflori]

comment:16

I've included some autotools magic to test if the -arch i386/x86_64 flag is supported and fall back to the -m32/64.
Could you test the updated spkg at the same address as usual (at least the spkg still build on my linux)?

I've put the diffs into the patches directory (although they are already applied because we have to run autoreconf) and will upload them here for review purpose.

[jpflori]

Diff for upstream svn commit r12323; accept glibc 2.15.

[jpflori]

Attachment: valgrind-3.8.1.diff.gz

spkg diff, for review only

[gvol]

comment:22

It looks like the diffs checking for -arch support are not included in the new spkg. I'm running OS X 10.8.2 and it doesn't build without them. If I apply those changes manually, then it chokes on __private_extern__. Changing those to extern, it builds. However, the "impossible" happens and it doesn't work:

==415== Memcheck, a memory error detector
==415== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==415== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==415== Command: python -i
==415== Parent PID: 414
==415== 
--415-- /Users/gvol/SageStuff/sage-5.4.rc2/local/bin/python:
--415-- dSYM directory is missing; consider using --dsymutil=yes
eq_SyscallStatus:
  {78 0 43}
  {78 0 40}

valgrind: m_syswrap/syswrap-main.c:380 (eq_SyscallStatus): the 'impossible' happened.
==415==    at 0x138032720: ???
==415==    by 0x138032A03: ???
==415==    by 0x13809A395: ???
==415==    by 0x13809B5FF: ???
==415==    by 0x138098D8A: ???
==415==    by 0x1380B9F21: ???

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable
==415==    at 0x7FFF5FC242DA: _kernelrpc_mach_vm_allocate_trap (in /usr/lib/dyld)
==415==    by 0x7FFF5FC2497C: vm_allocate (in /usr/lib/dyld)
==415==    by 0x7FFF5FC13BF3: ImageLoaderMachO::reserveAnAddressRange(unsigned long, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC13A54: ImageLoaderMachO::assignSegmentAddresses(ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC13C58: ImageLoaderMachO::mapSegments(int, unsigned long long, unsigned long long, unsigned long long, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC17104: ImageLoaderMachOCompressed::instantiateFromFile(char const*, int, unsigned char const*, unsigned long long, unsigned long long, stat const&, unsigned int, unsigned int, linkedit_data_command const*, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC10F5A: ImageLoaderMachO::instantiateFromFile(char const*, int, unsigned char const*, unsigned long long, unsigned long long, stat const&, ImageLoader::LinkContext const&) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC03557: dyld::loadPhase6(int, stat const&, char const*, dyld::LoadContext const&) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC08BAE: dyld::loadPhase5(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC0860D: dyld::loadPhase4(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC084F9: dyld::loadPhase3(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC07C30: dyld::loadPhase1(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC0319A: dyld::loadPhase0(char const*, char const*, dyld::LoadContext const&, std::__1::vector<char const*, std::__1::allocator<char const*> >*) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC02F07: dyld::load(char const*, dyld::LoadContext const&) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC0598E: dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC01396: dyldbootstrap::start(macho_header const*, int, char const**, long, macho_header const*, unsigned long*) (in /usr/lib/dyld)
==415==    by 0x7FFF5FC0105D: _dyld_start (in /usr/lib/dyld)
==415==    by 0x1: ???
==415==    by 0x7FFF5FBFEBBA: ???
==415==    by 0x7FFF5FBFEBC1: ???


Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.

If that doesn't help, please report this bug to: www.valgrind.org

In the bug report, send all the above text, the valgrind
version, and what OS and version you are using.  Thanks.

For the record, the exact way that I made the "fixes" was to add

perl -pi.bak -e 's/[-]arch x86_64/\@FLAG_M64@/;' -e 's/[-]arch i386/\@FLAG_M32@/' configure.in Makefile.all.am
./autogen.sh
export CFLAGS="$CFLAGS -D__private_extern__=extern"

to spkg-install just before calling configure. I don't suggest this method of "fixing" it.

Perhaps we should ask the maintainers of valgrind the best way to proceed.

[jpflori]

comment:23

Replying to @gvol:

It looks like the diffs checking for -arch support are not included in the new spkg. I'm running OS X 10.8.2 and it doesn't build without them. If I apply those changes manually, then it chokes on __private_extern__. Changing those to extern, it builds. However, the "impossible" happens and it doesn't work:

Yup I did not include the patch we wrote before because it seemed too much of a hassle and did not lead to a successful build of Valgrind.
Sorry about that but I did not have much time to spend on this one.

For the record, the exact way that I made the "fixes" was to add

perl -pi.bak -e 's/[-]arch x86_64/\@FLAG_M64@/;' -e 's/[-]arch i386/\@FLAG_M32@/' configure.in Makefile.all.am
./autogen.sh
export CFLAGS="$CFLAGS -D__private_extern__=extern"

to spkg-install just before calling configure. I don't suggest this method of "fixing" it.

Perhaps we should ask the maintainers of valgrind the best way to proceed.

I think the best way to proceed is unfortunately to use Apple's GCC.
Valgrind build system seems completely thought for it...
That would mean an additional check in spkg-install and would be much easier than fixing upstream build system to work with an FSF GCC.

[jpflori]

Changed work issues from Test on OSX to Use Apple compiler on OS X.

[vbraun]

comment:27

On Fedora 19, I get:

checking the GLIBC_VERSION version... unsupported version 2.17
configure: error: Valgrind requires glibc version 2.2 - 2.16

[jpflori]

comment:28

I'd say extend the treatment of GLIBC 2.15 in attachment: glibc-2.15.diff (i.e. treat it as 2.14) to GLIBC 2.16 and 2.17.
A quick google search seems to show it is enough and has been adopted by various distribs.
But We should surely have a look at valgrind svn first to see if GLIBC > 2.15 needs more specific treatment...

[jpflori]

comment:29

There is even 2.18 now.

[vbraun]

comment:30

Do you want to update it? I would be fine with the current spkg, too. Its better than what we have and users of cutting-edge distros probably don't need help to install the distro valgrind.

[jpflori]

comment:31

Sure, I'll try to produce the trivial patch tonight.

Note we should surely take care of #14097 as well if wa want this spkg to be useful.

[jpflori]

comment:32

I just checked Valgrind svn and the fix from attachment: glibc-2.15.diff is also used for 2.16-18 so providing a fixed spkg will be easy.
In fact, it seems glibc 2.>=12 get the exact same treatment, and the only difference with glibc 2.>=3/<=11 lies in coregrind/m_redir.c (and is indeed needed).

[jpflori]

comment:33

References of the valgrind commits: r12743, r13228, r13504.

[jpflori]

Spkg diff, for review only.

[jpflori]

comment:34

Attachment: valgrind-3.8.1.p0.diff.gz

If we really want to be able to use Valgrind on OS X when Sage built its own GCC, let's do that on another ticket.

[jpflori]

Changed work issues from Use Apple compiler on OS X. to none

[vbraun]

comment:35

Looks good to me, thanks!

Harald: please update on the mirrors

[vbraun]

Reviewer: Volker Braun

[haraldschilly]

comment:37

done, file is on the server