Singular omalloc requires 8-byte alignment on SPARC #14429

jdemeyer · 2013-04-09T09:43:46Z

On Solaris SPARC:

buildbot@mark:~/build/sage/mark-1/mark_full/build/sage$ ./sage -t --gdb --long devel/sage/sage/crypto/mq/sr.py
exec gdb -x "$SAGE_LOCAL/bin/sage-gdb-commands" --args python "$SAGE_LOCAL/bin/sage-runtests" --serial --long --timeout=0 --stats_path=/home/buildbot/.sage/timings2.jso
n devel/sage/sage/crypto/mq/sr.py
GNU gdb (GDB) 7.5.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.10".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/buildbot/build/sage/mark-1/mark_full/build/sage-5.9.beta4/local/bin/python...done.
[Thread debugging using libthread_db enabled]
[New Thread 1 (LWP 1)]
Running doctests with ID 2013-04-09-04-41-09-5eaf23f2.
Doctesting 1 file.
sage -t --long devel/sage/sage/crypto/mq/sr.py

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1 (LWP 1)]
0xf9f743ac in add_to_basis_ideal_quotient (h=0x4f170dc, c=0x4f16b38, ip=0x0) at tgb.cc:1518
1518      c->weighted_lengths[i] = pQuality (h, c, c->lengths[i]);
(gdb) list
1513      }
1514      else
1515        pNorm (h);
1516      pNormalize (h);
1517
1518      c->weighted_lengths[i] = pQuality (h, c, c->lengths[i]);
1519      c->gcd_of_terms[i] = got;
1520    #ifdef HAVE_BOOST
1521      c->states.push_back (dynamic_bitset <> (i));
1522
(gdb) print c->weighted_lengths
$5 = (wlen_type *) 0x4f4659c

The problem is that wlen_type is a 64-bit integer which is not correctly aligned to 8 bytes.

spkg: http://boxen.math.washington.edu/home/jdemeyer/spkg/singular-3-1-5.p7.spkg (diff)

upstream: http://www.singular.uni-kl.de:8002/trac/ticket/483

Upstream: Reported upstream. No feedback yet.

Component: porting: Solaris

Keywords: SPARC alignment SIGBUS omalloc

Author: Jeroen Demeyer

Reviewer: Volker Braun

Merged: sage-5.9.beta5

Issue created by migration from https://trac.sagemath.org/ticket/14429

The text was updated successfully, but these errors were encountered:

jdemeyer · 2013-04-09T10:00:53Z

Author: Jeroen Demeyer

jdemeyer · 2013-04-09T10:00:53Z

Upstream: Reported upstream. No feedback yet.

jdemeyer · 2013-04-09T10:00:53Z

Changed keywords from SPARC alignment SIGBUS to SPARC alignment SIGBUS omalloc

jdemeyer · 2013-04-09T10:16:44Z

comment:3

Attachment: singular-3-1-5.p7.diff.gz

vbraun · 2013-04-09T13:06:21Z

comment:5

Sounds good to me.

vbraun · 2013-04-09T13:06:21Z

Reviewer: Volker Braun

jpflori · 2013-04-10T09:12:45Z

comment:7

FYI, from the errors I had reported at:

https://groups.google.com/d/msg/sage-devel/DhJcPuFIlPo/ToQuuJzu_BMJ
except for the ones fixed by fix pickling of Matrix_modn_dense_double on SPARC #13151, this fixes the error in
sage/crypto/mq/sr.py
although I still get an error in
devel/sage/sage/interfaces/expect.py

sage -t devel/sage/sage/interfaces/expect.py
**********************************************************************
File "devel/sage/sage/interfaces/expect.py", line 1089, in sage.interfaces.expec
t.Expect._crash_msg
Failed example:
    singular('2+3')
Exception raised:
    Traceback (most recent call last):
      File "/infres/post/flori/sage-5.9.beta3-infres2/local/lib/python2.7/site-p
ackages/sage/doctest/forker.py", line 460, in _run
        self.execute(example, compiled, test.globs)
      File "/infres/post/flori/sage-5.9.beta3-infres2/local/lib/python2.7/site-p
ackages/sage/doctest/forker.py", line 819, in execute
        exec compiled in globs
      File "<doctest sage.interfaces.expect.Expect._crash_msg[4]>", line 1, in <
module>
        singular('2+3')
      File "/infres/post/flori/sage-5.9.beta3-infres2/local/lib/python2.7/site-p
ackages/sage/interfaces/singular.py", line 724, in __call__
        return SingularElement(self, type, x, False)
      File "/infres/post/flori/sage-5.9.beta3-infres2/local/lib/python2.7/site-p
ackages/sage/interfaces/singular.py", line 1184, in __init__
        self._name = parent._create( value, type)
      File "/infres/post/flori/sage-5.9.beta3-infres2/local/lib/python2.7/site-p
ackages/sage/interfaces/singular.py", line 685, in _create
        self.set(type, name, value)
      File "/infres/post/flori/sage-5.9.beta3-infres2/local/lib/python2.7/site-p
ackages/sage/interfaces/singular.py", line 628, in set
        raise TypeError, msg
    TypeError: [Errno 22] Invalid argument
    Error evaluating def sage51=2+3; in Singular
**********************************************************************
File "devel/sage/sage/interfaces/expect.py", line 1114, in sage.interfaces.expec
t.Expect._synchronize
Failed example:
    R.<x> = QQ[]; f = x^3 + x + 1;  g = x^3 - x - 1; r = f.resultant(g); gap(ZZ)
; singular(R)
Expected:
    Integers
    //   characteristic : 0
    //   number of vars : 1
    //        block   1 : ordering lp
    //                  : names    x
    //        block   2 : ordering C
Got:
    Integers
    Singular crashed -- automatically restarting.
    //   characteristic : 0
    //   number of vars : 1
    //        block   1 : ordering lp
    //                  : names    x
    //        block   2 : ordering C
**********************************************************************
2 items had failures:
   1 of   6 in sage.interfaces.expect.Expect._crash_msg
   1 of   3 in sage.interfaces.expect.Expect._synchronize
    [81 tests, 2 failures, 26.3 s]
----------------------------------------------------------------------
sage -t devel/sage/sage/interfaces/expect.py  # 2 doctests failed
----------------------------------------------------------------------
Total time for all tests: 27.1 seconds
    cpu time: 3.7 seconds
    cumulative wall time: 26.3 seconds

Note that running singular('2+3') in sage works fine and so does the second problematic doctest.
I still get the numerical noise problem from

sage/rings/polynomial/polynomial_element.pyx
which is undoubtly unrelated,
and bunches of timeouts, but now I even get those on quite powerfull and recent computers...

Oh and I still had troubles building singular because of C++ headers the system has in /usr/local/include which seem incompatible with the ones Sage's GCC ships, and this path is added by Singular in src/Singular/Makefile[.in] to CPPFLAGS if gcc version is greater than 4.
You could argue that it is the computer install itself which is ill-configured, but when removing the -I/usr/local/include from Singular/Makefile[.in], everything builds fine.
I've opened a ticket upstream but thanks to Singular serving their http server on a unusual port, i cannot access it at the moment.

jdemeyer · 2013-04-10T09:25:44Z

comment:8

The devel/sage/sage/interfaces/expect.py error is #14371.

jdemeyer · 2013-04-10T09:31:03Z

comment:9

Replying to @jpflori:

I still get the numerical noise problem from

sage/rings/polynomial/polynomial_element.pyx

Did you ever report this?

jpflori · 2013-04-10T09:33:43Z

comment:10

Nope, just on sage-devel, I'll open a Trac ticket.

jdemeyer · 2013-04-10T09:39:50Z

comment:11

Replying to @jpflori:

Oh and I still had troubles building singular because of C++ headers the system has in /usr/local/include

I guess the upstream ticket is http://www.singular.uni-kl.de:8002/trac/ticket/480, but that doesn't have enough information to debug the issue. A complete log file of the failed Singular build would be a good start.

jpflori · 2013-04-10T10:06:09Z

comment:12

Here you go:

http://boxen.math.washington.edu/home/jpflori/singular-3-1-5.p7_solaris10_sparc.log
Part of it was posted here:
https://groups.google.com/d/msg/sage-release/1felF4mvFMM/L-lSYD2u2I8J

But is there really any good reason to include /usr/local/include automagically? Is that standard practice?

jdemeyer · 2013-04-10T10:11:56Z

comment:13

Please do

g++ -save-temps -O2 -g  -fPIC   -fno-implicit-templates -I. -I.. -I/infres/post/flori/sage-5.9.beta3-infres2/local  -I/usr/xpg4/include -I/infres/post/flori/sage-5.9.beta3-infres2/local/include -I/infres/post/flori/sage-5.9.beta3-infres2/local/include  -I/usr/local/include  -DNDEBUG -DOM_NDEBUG -DSunOS_5 -DHAVE_CONFIG_H -c bigintmat.cc

from the directory containing bigintmat.cc (within a Sage shell)
and send the file bigintmat.ii

jdemeyer · 2013-04-10T10:15:56Z

comment:14

Replying to @jpflori:

Is that standard practice?

I think it is. Why put stuff in /usr/local/include if you don't want it included?

jdemeyer · 2013-04-10T10:17:20Z

comment:15

Replying to @jpflori:

Part of it was posted here:

https://groups.google.com/d/msg/sage-release/1felF4mvFMM/L-lSYD2u2I8J

Yes, but that omitted the crucial information of which command caused those errors.

jpflori · 2013-04-10T11:08:10Z

comment:16

Replying to @jdemeyer:

Replying to @jpflori:

Is that standard practice?

I think it is. Why put stuff in /usr/local/include if you don't want it included?

I'd say you put stuff there intentionally (rather than just building a package and letting it install its headers by default into /usr/include) to get it included when you explicitly add "-I/usr/local/include".
I don't know if any usual software install there by default (it seems this inclusion was done to please gcc > 4, but do gcc > 4 really install stuff there by default? maybe that was the install of the Singular dev which was put there...).

And it seems the problematic software on my system is Sun's gcc 3.4.3.
I don't think that's Sun default install, but it rather looks like a hack from the sysadmin.

jpflori · 2013-04-10T11:11:25Z

comment:17

Replying to @jdemeyer:

Please do

g++ -save-temps -O2 -g  -fPIC   -fno-implicit-templates -I. -I.. -I/infres/post/flori/sage-5.9.beta3-infres2/local  -I/usr/xpg4/include -I/infres/post/flori/sage-5.9.beta3-infres2/local/include -I/infres/post/flori/sage-5.9.beta3-infres2/local/include  -I/usr/local/include  -DNDEBUG -DOM_NDEBUG -DSunOS_5 -DHAVE_CONFIG_H -c bigintmat.cc

from the directory containing bigintmat.cc (within a Sage shell)
and send the file bigintmat.ii

Attached to this ticket.

jpflori · 2013-04-10T11:11:41Z

Attachment: bigintmat.ii.gz

jdemeyer · 2013-04-10T11:13:54Z

comment:18

Replying to @jpflori:

I don't know if any usual software install there by default

Almost all software installs stuff in /usr/local/include by default.

jpflori · 2013-04-10T11:16:43Z

comment:19

True enough for user installed things, but it seems distribution packages do rather install in /usr directly.

jdemeyer · 2013-04-10T11:21:48Z

comment:20

Replying to @jpflori:

True enough for user installed things, but it seems distribution packages do rather install in /usr directly.

True again.

jdemeyer · 2013-04-10T11:24:03Z

comment:21

After looking at attachment: bigintmat.ii, I would say the problem is your system configuration, not a Sage or Singular bug.

jpflori · 2013-04-10T12:28:28Z

comment:22

On Unix at least, gcc will look in /usr/local/include anyway: http://gcc.gnu.org/onlinedocs/cpp/Search-Path.html or http://gcc.gnu.org/onlinedocs/gcc-4.6.3/cpp/Search-Path.html#Search-Path; so that's another point for not changing anything in Singular.

Note that without the -I/usr/local/include, then the headers which get picked are in

/infres/post/flori/sage-5.9.beta3-infres2/local/lib/gcc/sparc-sun-solaris2.10/4.6.3/include-fixed/sys/feature_tests.h

This looks like a very nice directory to get headers from for the Sage's gcc...
According to the above doc, adding an extra -I/usr/local/include should not change the search path, but it apparently does.
And it seems that by default, the GCC Sage built does not look there (I ran "gcc -E -v -" and it does not show /usr/local/include, only /usr/include after three directorues under $SAGE_LOCAL).

with the system-wide gcc:

gcc -E -v -
Reading specs from /usr/local/packages/gcc3/bin/../lib/gcc/sparc-sun-solaris2.10/3.4.3/specs
Configured with: /sfw10/builds/build/sfw10-patch/usr/src/cmd/gcc/gcc-3.4.3/configure --prefix=/usr/sfw --with-as=/usr/ccs/bin/as --without-gnu-as --with-ld=/usr/ccs/bin/ld --without-gnu-ld --enable-languages=c,c++ --enable-shared
Thread model: posix
gcc version 3.4.3 (csl-sol210-3_4-branch+sol_rpath)
 /usr/local/packages/gcc3/bin/../libexec/gcc/sparc-sun-solaris2.10/3.4.3/cc1 -E -quiet -v -iprefix /usr/local/packages/gcc3/bin/../lib/gcc/sparc-sun-solaris2.10/3.4.3/ - -mcpu=v7
ignoring nonexistent directory "/usr/local/packages/gcc3/bin/../lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../../sparc-sun-solaris2.10/include"
ignoring nonexistent directory "/usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/../../../../sparc-sun-solaris2.10/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/packages/gcc3/bin/../lib/gcc/sparc-sun-solaris2.10/3.4.3/include
 /usr/local/include
 /usr/sfw/include
 /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/include
 /usr/include
End of search list.

with Sage's one:

gcc -E -v -
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/infres/post/flori/sage-5.9.beta3-infres2/local/libexec/gcc/sparc-sun-solaris2.10/4.6.3/lto-wrapper
Target: sparc-sun-solaris2.10
Configured with: ../src/configure --prefix=/infres/post/flori/sage-5.9.beta3-infres2/local --with-local-prefix=/infres/post/flori/sage-5.9.beta3-infres2/local --with-gmp=/infres/post/flori/sage-5.9.beta3-infres2/local --with-mpfr=/infres/post/flori/sage-5.9.beta3-infres2/local --with-mpc=/infres/post/flori/sage-5.9.beta3-infres2/local --with-system-zlib --disable-multilib --with-as=/usr/ccs/bin/as --with-ld=/usr/ccs/bin/ld
Thread model: posix
gcc version 4.6.3 (GCC) 
COLLECT_GCC_OPTIONS='-E' '-v' '-mcpu=v9'
 /infres/post/flori/sage-5.9.beta3-infres2/local/libexec/gcc/sparc-sun-solaris2.10/4.6.3/cc1 -E -quiet -v -D__sparcv8 - -mcpu=v9
ignoring nonexistent directory "/infres/post/flori/sage-5.9.beta3-infres2/local/lib/gcc/sparc-sun-solaris2.10/4.6.3/../../../../sparc-sun-solaris2.10/include"
ignoring duplicate directory "/infres/post/flori/sage-5.9.beta3-infres2/local/include"
  as it is a non-system directory that duplicates a system directory
#include "..." search starts here:
#include <...> search starts here:
 /infres/post/flori/sage-5.9.beta3-infres2/local/lib/gcc/sparc-sun-solaris2.10/4.6.3/include
 /infres/post/flori/sage-5.9.beta3-infres2/local/include
 /infres/post/flori/sage-5.9.beta3-infres2/local/lib/gcc/sparc-sun-solaris2.10/4.6.3/include-fixed
 /usr/include
End of search list.

On a Ubuntu install, with the system-wide gcc, it indeed looks in /usr/local/include before the corresponding /usr/lib/gcc/x86_64-linux-gnu/4.6/include-fixed (but after /usr/lib/gcc/x86_64-linux-gnu/4.6/include).
In fact on the Solaris install, I guess the line correponding to /usr/local/include is $SAGE_LOCAL/include when I run "gcc -E -v -" which was defined so when Sage built GCC.
And in particular, none of these behaviors agree with GCC online doc, or rather the /usr/local should be replaced by the prefix used when GCC was built.
Indeed I have by default;

"libdir"/gcc/"target"/"version"/include
"prefix"/include
"libdir"/gcc/"target"/"version"/include-fixed
/usr/include/"target"
/usr/include
On another ia64 system, the behavior is similar:
with Sage's GCC

gcc -E -v -
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/home/jpflori/sage-5.7/local/libexec/gcc/ia64-unknown-linux-gnu/4.6.3/lto-wrapper
Target: ia64-unknown-linux-gnu
Configured with: ../src/configure --prefix=/home/jpflori/sage-5.7/local --with-local-prefix=/home/jpflori/sage-5.7/local --with-gmp=/home/jpflori/sage-5.7/local --with-mpfr=/home/jpflori/sage-5.7/local --with-mpc=/home/jpflori/sage-5.7/local --with-system-zlib --disable-multilib  
Thread model: posix
gcc version 4.6.3 (GCC) 
COLLECT_GCC_OPTIONS='-E' '-v'
 /home/jpflori/sage-5.7/local/libexec/gcc/ia64-unknown-linux-gnu/4.6.3/cc1 -E -quiet -v -
ignoring nonexistent directory "/home/jpflori/sage-5.7/local/lib/gcc/ia64-unknown-linux-gnu/4.6.3/../../../../ia64-unknown-linux-gnu/include"
ignoring duplicate directory "/home/jpflori/sage-5.7/local/include"
  as it is a non-system directory that duplicates a system directory
#include "..." search starts here:
#include <...> search starts here:
 /home/jpflori/sage-5.7/local/lib/gcc/ia64-unknown-linux-gnu/4.6.3/include
 /home/jpflori/sage-5.7/local/include
 /home/jpflori/sage-5.7/local/lib/gcc/ia64-unknown-linux-gnu/4.6.3/include-fixed
 /usr/include
End of search list.

with system-wide:

gcc -E -v -
Using built-in specs.
Target: ia64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.4.5-8' --with-bugurl=file:///usr/share/doc/gcc-4.4/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.4 --enable-shared --enable-multiarch --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.4 --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --disable-libssp --enable-objc-gc --with-system-libunwind --enable-checking=release --build=ia64-linux-gnu --host=ia64-linux-gnu --target=ia64-linux-gnu
Thread model: posix
gcc version 4.4.5 (Debian 4.4.5-8) 
COLLECT_GCC_OPTIONS='-E' '-v'
 /usr/lib/gcc/ia64-linux-gnu/4.4.5/cc1 -E -quiet -v -
ignoring nonexistent directory "/usr/local/include/ia64-linux-gnu"
ignoring nonexistent directory "/usr/local/include"
ignoring nonexistent directory "/usr/lib/gcc/ia64-linux-gnu/4.4.5/../../../../ia64-linux-gnu/include"
ignoring nonexistent directory "/usr/include/ia64-linux-gnu"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/ia64-linux-gnu/4.4.5/include
 /usr/lib/gcc/ia64-linux-gnu/4.4.5/include-fixed
 /usr/include
End of search list.

So the real question is, is adding this -I/usr/local/include really useful?
Shouldn't it rather be a user decision to add it if needed?
It seems most GCC are smart enough to correctly set their default include path anyway, not sure what the comment in Singular sources in Singular/Makefile.in is related to...
I'd like to say that if it does nothing on most systems, and just break my system (although I agree its config is not perfect, but I cannot do anything about that), then it's useless, but that would narrow minded.

jpflori · 2013-04-10T13:03:50Z

comment:23

In fact the addition of -I and -L was meant for versions of gcc (or other compilers?) before gcc 3.
It was then removed for gcc3 at Singular/Singular@833e11f#Singular/configure.in

jpflori · 2013-04-10T13:04:25Z

comment:24

So I guess it makes sense to omit it for gcc > 3 as well :)

jpflori · 2013-04-10T13:33:59Z

comment:25

It was seemingly added for GCC 2.95 at: Singular/Singular@a70441f

jpflori · 2013-04-10T13:52:25Z

comment:26

The reason for this was seemingly:

http://gcc.gnu.org/ml/gcc/1999-07n/msg00958.html
http://gcc.gnu.org/ml/gcc-patches/1999-07n/msg00489.html
http://gcc.gnu.org/ml/gcc-patches/1999-05n/msg00477.html
so the fix should really be for gcc 2.95 only...

jdemeyer · 2013-04-11T07:35:19Z

Merged: sage-5.9.beta5

jdemeyer · 2013-04-11T08:08:20Z

comment:28

Jean-Pierre: you should probably mention this upstream: http://www.singular.uni-kl.de:8002/trac/ticket/480

jpflori · 2013-04-11T08:40:41Z

comment:29

It's planned, but I cannot do it during the day (without appropiate ssh tunnels and http proxies) and have a tendency to postpone it in the evening.

jdemeyer added this to the sage-5.10 milestone Apr 9, 2013

jdemeyer added c: porting: solaris labels Apr 9, 2013

jdemeyer assigned sagetrac-drkirkby Apr 9, 2013

jdemeyer added the t: bug label Apr 9, 2013

This comment has been minimized.

Sign in to view

jdemeyer added the s: needs review label Apr 9, 2013

jdemeyer changed the title ~~Singular omalloc requires 8- byte alignment on SPARC~~ Singular omalloc requires 8-byte alignment on SPARC Apr 9, 2013

vbraun added s: positive review and removed s: needs review labels Apr 9, 2013

jdemeyer modified the milestones: sage-5.10, sage-5.9 Apr 9, 2013

jdemeyer removed the s: positive review label Apr 11, 2013

jdemeyer closed this as completed Apr 11, 2013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Singular omalloc requires 8-byte alignment on SPARC #14429

Singular omalloc requires 8-byte alignment on SPARC #14429

jdemeyer commented Apr 9, 2013

jdemeyer commented Apr 9, 2013

jdemeyer commented Apr 9, 2013

This comment has been minimized.

jdemeyer commented Apr 9, 2013

This comment has been minimized.

jdemeyer commented Apr 9, 2013

vbraun commented Apr 9, 2013

vbraun commented Apr 9, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 11, 2013

jdemeyer commented Apr 11, 2013

jpflori commented Apr 11, 2013

Singular omalloc requires 8-byte alignment on SPARC #14429

Singular omalloc requires 8-byte alignment on SPARC #14429

Comments

jdemeyer commented Apr 9, 2013

jdemeyer commented Apr 9, 2013

jdemeyer commented Apr 9, 2013

This comment has been minimized.

jdemeyer commented Apr 9, 2013

This comment has been minimized.

jdemeyer commented Apr 9, 2013

vbraun commented Apr 9, 2013

vbraun commented Apr 9, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jdemeyer commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jpflori commented Apr 10, 2013

jdemeyer commented Apr 11, 2013

jdemeyer commented Apr 11, 2013

jpflori commented Apr 11, 2013