Adding support to an ARM processor

[SnarkBoojum]

The compilation of sage 5.0.beta9 fails on toshiba ac100 netbooks, with an arm processor running an ubuntu distribution.

Relevant tickets:

#10328 (the one needed for successful building of 5.0.beta9, positive review)

Ones which are fixing bugs:

#12371 (fixing C types related bug in vertex_separator, positive review)

#12586 (maxima-related floating point stuff, needs work)

#12449 (log(gamma) for floating point related stuff, needs work)

Component: build

Keywords: ARM

Reviewer: Jean-Pierre Flori

Issue created by migration from https://trac.sagemath.org/ticket/10285

[SnarkBoojum]

Log from the failed build

[SnarkBoojum]

comment:1

Attachment: log.gz

I had a wonderful idea this night : compiling with "make -k" instead of make, so the compilation goes through all packages and go on even on error.

That means running "make -k 2>&1 > build.log" has given me a log with the failed build of the only problematic packages!

So here is the full list of things which don't build on this box (grep build.log -B1 -e Finished) : atlas-3.8.3.p16, boehm_gc-7.1.p6, flint-1.5.0.p5, singular-3-1-1-4.p3, symmetrica-2.0.p5,tachyon-0.98beta.p11.

[SnarkBoojum]

comment:2

W. Hart from FLINT saw something very interesting in the logs : the errors about Thumb support happen on assembly files which come from C-files -- so in fact the compiler which is generating wrong assembly.

I reported that as : https://bugs.launchpad.net/ubuntu/+bug/677360

Getting that right won't fix all packages, but should definitely help.

[sagetrac-michaelh]

comment:3

Hi there. I found this ticket via:
https://bugs.launchpad.net/ubuntu/+bug/677360

There's a bunch of things going on here and they look like the package has been configured incorrectly. I don't think the compiler is at fault.

This code:

/home/jpuydt/sage/spkg/build/atlas-3.8.3.p16/ATLAS-build/../src//CONFIG/src/atlconf_misc.c:306: warning: the use of `tmpnam' is dangerous, better use `mkstemp'
2690	/home/jpuydt/sage/spkg/build/atlas-3.8.3.p16/ATLAS-build/../src//CONFIG/src/backend/cpuid.S: Assembler messages:
2691	/home/jpuydt/sage/spkg/build/atlas-3.8.3.p16/ATLAS-build/../src//CONFIG/src/backend/cpuid.S:30: Error: bad instruction `subl $8,%esp'
2692	/home/jpuydt/sage/spkg/build/atlas-3.8.3.p16/ATLAS-build/../src//CONFIG/src/backend/cpuid.S:31: Error: bad instruction `movl %ebx,(%esp)'
2693	/home/jpuydt/sage/spkg/build/atlas-3.8.3.p16/ATLAS-build/../src//CONFIG/src/backend/cpuid.S:32: Error: bad instruction `movl %edi,4(%esp)'
2694	/home/jpuydt/sage/spkg/build/atlas-3.8.3.p16/ATLAS-build/../src//CONFIG/src/backend/cpuid.S:36: Error: bad instruction `movl 16(%esp),%eax'

is x86 assembler.

This line:

	cd /home/jpuydt/sage/spkg/build/atlas-3.8.3.p16/ATLAS-build ; ./xprobe_comp -v 0 -o atlconf.txt -O 1 -A 0 -Si nof77 0  -Fa ic '-fPIC' -Fa sm '-fPIC' -Fa dm '-fPIC' -Fa sk '-fPIC' -Fa dk '-fPIC' -Fa xc '-fPIC' -C if '/home/jpuydt/sage/local/bin/sage_fortran' -Fa if '-fPIC'  -b 32 > config1.out
2754	cc1: error: unrecognized command line option "-m32"

fails due to the ARM compiler being passed the x86 specific -m32 flag.

This code:

 gcc -DPACKAGE_NAME=\"gc\" -DPACKAGE_TARNAME=\"gc\" -DPACKAGE_VERSION=\"7.1\" "-DPACKAGE_STRING=\"gc 7.1\"" -DPACKAGE_BUGREPORT=\"Hans.Boehm@hp.com\" -DGC_VERSION_MAJOR=7 -DGC_VERSION_MINOR=1 -DPACKAGE=\"gc\" -DVERSION=\"7.1\" -DGC_LINUX_THREADS=1 -D_REENTRANT=1 -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DNO_EXECUTE_PERMISSION=1 -DALL_INTERIOR_POINTERS=1 -DGC_GCJ_SUPPORT=1 -DJAVA_FINALIZATION=1 -DATOMIC_UNCOLLECTABLE=1 -DLARGE_CONFIG=1 -I./include -fexceptions -I libatomic_ops/src -g -O2 -MT alloc.lo -MD -MP -MF .deps/alloc.Tpo -c alloc.c  -fPIC -DPIC -o .libs/alloc.o
/tmp/cckXY2Jn.s: Assembler messages:
/tmp/cckXY2Jn.s:1314: Error: selected processor does not support Thumb mode `swp ip,r2,[r3]'

is due to a swp instruction being used in inline assembly instead of the correct ldrex. I know that Boehm GC 7.2 does this correctly and suspect that 7.1 does as well, so you might have another configuration problem.

[SnarkBoojum]

comment:4

Let me summarize what has been found so far :

• atlas does use x86 code, and this is a probably a bug in atlas ;

• atlas also uses the -m32 switch, and that is probably a bug there too ;

• tachyon has the same -m32 problem but its author is working on it ;

• I tested the compilation of gc-7.2alpha4 (Boehm gc), and it compiled well, while compiling gc-7.1 failed with the same problem as in sage -- so upgrading the version will fix the package ;

• I tried to find out the problem in flint-1.5.2 and failed -- but trying flint2, I get "error: impossible constraint in ‘asm'" ; which probably means there's something fishy in the code too ;

• singular just refuses to build because it doesn't know the arch ;

• symmetrica seems to trigger a compiler problem.

[SnarkBoojum]

comment:5

I found where the problem is introduced in flint-1.5.2's source code -- waiting for feedback from upstream (W.Hart already answered me and thought it may have been a compiler problem -- I just found out it isn't).

[SnarkBoojum]

comment:6

I don't know why I came to own that bug :-/

[SnarkBoojum]

comment:7

For symmetrica, I discovered the problem was that the compiler went out of memory : adding more swap, I was able to get it to compile in sage.

So as things stand :

(1) atlas-3.8.3.p16 doesn't build for several reasons (I haven't investigated yet) ;
(2) boehm_gc-7.1.p6 doesn't build, but the 7.2alpha4 version does ;
(3) flint-1.5.0.p5 has a problem in its assembly code -- reported upstream and waiting for feedback ;
(4) singular-3-1-1-4.p3 detects an unsupported platform (I haven't investigated yet) ;
(5) tachyon-0.98beta.p11 doesn't build because the Makefile target adds a switch which is unsupported on this platform -- reported upstream and waiting for feedback.

[SnarkBoojum]

Attachment: build.log

Build log from the failed build with all problems listed

[SnarkBoojum]

comment:8

I pushed things further -- it's taking shape :

1. atlas -- reported upstream, no feedback, no further investigation yet ;

2. Boehm's GC -- doesn't build in version 7.1.p6(sage), but version 7.2alpha4(upstream) does ;

3. flint -- reported and discussed with upstream, ideas of solutions found ; which solution gets put into practice is still not sure ;

4. singular -- reported upstream, no feedback, partial fixes found ;

5. tachyon -- reported upstream, upstream answered once, a fix has been found, waiting for further feedback from upstream.

[SnarkBoojum]

comment:9

I opened ticket #10328 for the flint package, giving everything to make it just work.

[SnarkBoojum]

comment:10

Why am I again the owner!? Sigh.

Singular-3-1-3 will have support for the tegra2 processor (yes, that means I had good feedback :-) )

[SnarkBoojum]

comment:11

Ok, as far as I know, atlas is the last hurdle (and what a hurdle).

I have a strange problem with Boehm' GC version 7.2 though : the spkg I made doesn't build as-is. If I use my usual shell, export SAGE_LOCAL, then unpack my spkg and ./spkg-install, all is fine. If I use sage -sh, then the resulting shell isn't able to compile the package! I first get strange warnings in the configure step :

checking if gcc supports -c -o file.o... rm: cannot remove `conftest*': No such file or directory

then it goes on trying to build (which is already surprising) :

/bin/bash ./libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I./include -I./include -I./libatomic_ops/src -I./libatomic_ops/src -fexceptions -g -O2 -MT allchblk.lo -MD -MP -MF .deps/allchblk.Tpo -c -o allchblk.lo allchblk.c
rm: cannot remove allchblk.o': No such file or directory rm: cannot remove .libs/allchblk.o': No such file or directory
rm: cannot remove allchblk.lo': No such file or directory rm: cannot remove allchblk.loT': No such file or directory
rm: cannot remove allchblk.lo': No such file or directory rm: cannot remove allchblk.loT': No such file or directory
rm: cannot remove .libs/allchblk.o': No such file or directory rm: cannot remove ': No such file or directory

Could someone lend me a hand?

[vbraun]

comment:12

Often, errors during ATLAS build are no problem: The tuning process builds different assembly implementations to benchmark on your particular hardware and its OK if these fail. Of course some errors, on the other hand, are genuine bugs ;-)

You could try #10226, although I doubt that it would make any difference on ARM.

[SnarkBoojum]

comment:13

Replying to @vbraun:

Often, errors during ATLAS build are no problem: The tuning process builds different assembly implementations to benchmark on your particular hardware and its OK if these fail. Of course some errors, on the other hand, are genuine bugs ;-) You could try #10226, although I doubt that it would make any difference on ARM.

Those errors are not during the build when ATLAS tries to optimize. They happen during the configuration, when ATLAS decides what it will try. This step definitely doesn't work with that processor, and their organisation is so special I haven't understood how it works yet. The only documentation I found was how to use it when it works -- not how to extend and fix it.

I reported the issue upstream as : https://sourceforge.net/tracker/index.php?func=detail&aid=3111857&group_id=23725&atid=379483

I'm now fully stuck... :-(

[SnarkBoojum]

comment:14

I could push sage build much further those last days :

• atlas is ok ; I have a patch attached to my sourceforge report to upstream (!https://sourceforge.net/tracker/?func=detail&atid=379483&aid=3111857&group_id=23725) ;
• boehm is about ok ; I still need to build it by hand since the same error occurs when going through sage -sh (and hence toplevel Makefile or spkg/install way) -- do I need to mention some feedback would be nice?
• flint is mostly ok, I opened another ticket about it
• numpy is ok ; I have a patch attached to a trac report to upstream (http://projects.scipy.org/numpy/ticket/1694)
• singular is ok ; I have a patch attached to a trac report to upstream (http://www.singular.uni-kl.de:8002/trac/ticket/299)
• tachyon is mostly ok (I know how to fix it, but private discussion is going on with upstream on how to better do things -- basically remove the -m32 flag from the switches when on ARM).

For the rest, I'm still building...

[SnarkBoojum]

comment:15

So, yesterday I wrote that it was still building ; this is what I have to say this morning :

jpuydt@hecke:~/sage-4.6$ tail -n 2 build.log 
build succeeded.
Build finished.  The built documents can be found in /home/jpuydt/sage-4.6/devel/sage/doc/output/html/fr/tutorial

Yes, that means that my latest list of problems was the last : I have sage up on a toshiba AC100 -- a netbook running an ARM processor.

[vbraun]

comment:24

Every Sage release passes all doctests on the supported platforms. Unless you added a patch that causes the pickling/deprecation errors the doctest should pass. It is possible that you end up in a different code path while unpickling...

[SnarkBoojum]

comment:25

I tried to track the issue of gamma(float(6)) not being equal enough to 120 (sic).

The ptestlong log points to devel/sage-main/sage/functions/other.py ; where I learn gamma is a GinacFunction 'tgamma' ; ginac is implemented in the pynac-0.2.1 package. Notice that if I call gamma(float(6), prec=10) then I get to call the mpmath version which returns the correct result.

In the pynac-0.2.1 package, things get hairy, but it seems I end up calling Number_T::tgamma, which is just py_funcs.py_tgamma. Of course, py_funcs is only declared in the pynac package, so hunting where it is wasn't that simple.

I managed to find the trail again by checking which packages depend on pynac in spkg/standard/deps : none. So I turned to sage-4.6, where I found : py_funcs.py_tgamma = &py_tgamma, implemented in ./sage/symbolic/pynac.pyx, where as far as I understand, since float(6) is a float, I guess the type checking leads me to sage_tgammal, which is defined in ./sage/symbolic/pynac_cc.h ; since I'm not running cygwin, this function is just calling tgammal... which is in math.h.

And I checked that this tgammal returns 120 with a printf ("%Lg\n", tgammal(6.)); ... so I'm a little at loss to where the error can come from.

Where did I fail in my search?

[vbraun]

comment:26

Are you using tgammal from C or Cython? A good test would be (run in notebook):

%cython

cdef extern from "math.h":
    long double tgammal(long double x)

cdef gammatest():
    cdef float x = 6
    print tgammal(x)
    printf("%.20Lf", tgammal(x))
    
gammatest()

[SnarkBoojum]

comment:27

This cython code gives a good answer.

Notice that it leaves open the question : where did I fail in my seach?

[dimpase]

comment:28

Replying to @vbraun:

Some failures are just time outs because your processor is slower than what the doctest writer took into account. I think thats the origin of the "0 doctests failed" case.

The only worrying ones are the pickling error (no idea whats going on there) and the two floating point rounding errors, for example

File "/home/jpuydt/sage-4.6/devel/sage-main/sage/functions/other.py", line 497:
    sage: gamma1(float(6))
Expected:
    120.0
Got:
    119.99999999999997

I think this is using GSL to compute the gamma function. Definitely needs to be sorted out why it fails on ARM.

as far as floating point goes, 119.99999999999997 is definitely equal to 120...
I don't see why this can be classified as a bug. To me, it's just a feature of this particular FPU, nothing
more than that. (e.g. it can present 6 as 5.9999999999999, nothing is wrong with this...)
I think you can just modify the doctests accordingly.

Regarding the pickling, is this even a stable failure, i.e. when you re-run this particular test, do you see exactly the same error
again and again?

[SnarkBoojum]

comment:29

Well, I have two questions : (1) if those floating point numbers are supposed to be considered equal, why isn't the test designed to pass up to an epsilon? (2) how do you explain the test fails, but I'm unable to pinpoint exactly how that test-failing result is found?

Yes, the errors are stable. The fact that there are "DeprecationWarning" all over is strange.

[kiwifb]

comment:30

I'd say the "DeprecationWarning" are expected by the test.

Expected:
    x...
    Successfully unpickled ... objects.
    Failed to unpickle 0 objects.

I think the dots means that you can match anything between before the dots and after the dots. Here is the results of that test on a machine where it passes:

sage: print "x"; sage.structure.sage_object.unpickle_all()
x
/home/francois/Work/sandbox/sage-4.6.2.alpha4/local/bin/sage-ipython:1: DeprecationWarning: Your word object is saved in an old file format since FiniteWord_over_OrderedAlphabet is deprecated and will be deleted in a future version of Sage (you can use FiniteWord_list instead). You can re-save your word by typing "word.save(filename)" to ensure that it will load in future versions of Sage.
  #!/usr/bin/env python
/home/francois/Work/sandbox/sage-4.6.2.alpha4/local/bin/sage-ipython:1: DeprecationWarning: Your word object is saved in an old file format since AbstractWord is deprecated and will be deleted in a future version of Sage (you can use FiniteWord_list instead). You can re-save your word by typing "word.save(filename)" to ensure that it will load in future versions of Sage.
  #!/usr/bin/env python
/home/francois/Work/sandbox/sage-4.6.2.alpha4/local/bin/sage-ipython:1: DeprecationWarning: Your word object is saved in an old file format since Word_over_Alphabet is deprecated and will be deleted in a future version of Sage (you can use FiniteWord_list instead). You can re-save your word by typing "word.save(filename)" to ensure that it will load in future versions of Sage.
  #!/usr/bin/env python
/home/francois/Work/sandbox/sage-4.6.2.alpha4/local/bin/sage-ipython:1: DeprecationWarning: Your word object is saved in an old file format since Word_over_OrderedAlphabet is deprecated and will be deleted in a future version of Sage (you can use FiniteWord_list instead). You can re-save your word by typing "word.save(filename)" to ensure that it will load in future versions of Sage.
  #!/usr/bin/env python
/home/francois/Work/sandbox/sage-4.6.2.alpha4/local/bin/sage-ipython:1: DeprecationWarning: ChristoffelWord_Lower is deprecated, use LowerChristoffelWord instead
  #!/usr/bin/env python
Successfully unpickled 586 objects.
Failed to unpickle 0 objects.

S the only worrying thing is the failure to unpickle.

[dimpase]

comment:32

Replying to @SnarkBoojum:

Well, I have two questions : (1) if those floating point numbers are supposed to be considered equal, why isn't the test designed to pass up to an epsilon?

Sorry, I think I was too quick here. The precision achieved is less than 2e-14, which seems to be too coarse to me.
Indeed, it's a good idea to see where this comes from.
In fact, #8847 looks quite relevant - they reported precisely the same failure of gamma(float(6)) on Core 2 Duo, and tweaked with a pynac wrapper...

(2) how do you explain the test fails, but I'm unable to pinpoint exactly how that test-failing result is found?

dig deeper... :-)

Yes, the errors are stable. The fact that there are "DeprecationWarning" all over is strange.

can you check whether it's pickling or unpickling that is failing?
I mean, you can pickle an object on one machine, copy the file it is in to another one, and try unpickling it there.

[dimpase]

comment:33

Replying to @SnarkBoojum:

I managed to find the trail again by checking which packages depend on pynac in spkg/standard/deps : none. So I turned to sage-4.6, where I found : py_funcs.py_tgamma = &py_tgamma, implemented in ./sage/symbolic/pynac.pyx, where as far as I understand, since float(6) is a float, I guess the type checking leads me to sage_tgammal, which is defined in ./sage/symbolic/pynac_cc.h ; since I'm not running cygwin, this function is just calling tgammal... which is in math.h.

can you insert printfs in sage/symbolic/pynac_cc.h to see how sage_tgammal is actually called, with which parameters, what it returns, etc? (don't forget to run "sage -b" after the change...)

[SnarkBoojum]

comment:34

I added :
printf("sage_tgammal (%.20Lf)=%.20Lf\n", x, tgammal(x));

and then got :
sage: gamma(float(6))
sage_tgammal (6.00000000000000000000)=119.99999999999997157829
119.99999999999997

Thanks to fbissey for the help.

[SnarkBoojum]

comment:35

Gasp! I thought I already checked my -lm for correctness, but now I tried :

jpuydt@hecke:/tmp$ cat test.c
#include <stdio.h>
#include <math.h>

int
main (int argc,
char* argv[])
{
long double x = 6.0;
printf("tgammal (%.20Lf)=%.20Lf\n", x, tgammal(x));
return 0;
}
jpuydt@hecke:/tmp$ ./test
tgammal (6.00000000000000000000)=119.99999999999997157829

[dimpase]

comment:36

Replying to @SnarkBoojum:

Gasp! I thought I already checked my -lm for correctness, but now I tried :

jpuydt@hecke:/tmp$ cat test.c
#include <stdio.h>
#include <math.h>

int
main (int argc,
char* argv[])
{
long double x = 6.0;
printf("tgammal (%.20Lf)=%.20Lf\n", x, tgammal(x));
return 0;
}
jpuydt@hecke:/tmp$ ./test
tgammal (6.00000000000000000000)=119.99999999999997157829

OK, so this is a libc bug -- well, assuming all the 53 bits of the fractional part of the IEEE double should be correct in this case
(long double is the same as double on ARM, afaik). And we get only 34 bits right.
Something to report upstream. Which version of libc do you run?

[SnarkBoojum]

comment:37

Replying to @dimpase:

OK, so this is a libc bug -- well, assuming all the 53 bits of the fractional part of the IEEE double should be correct in this case
(long double is the same as double on ARM, afaik). And we get only 34 bits right.
Something to report upstream. Which version of libc do you run?

I feel very annoyed by this : when I tracked the call stack, I found a working implementation, and now it seems I missed something... I must have checked "double" instead of "long double" :-(

The version is : eglibc (2.12.1-0ubuntu10.2).

My main box' is : eglibc (2.11.2-11).

I'll see with upstream.

So that means only the pickling problems are left. Thanks!

[SnarkBoojum]

comment:38

The libc issue is reported as : https://bugs.launchpad.net/ubuntu/+source/eglibc/+bug/713985

[SnarkBoojum]

comment:39

With sage-4.7rc2, there are only two build problems left : atlas and singular. From the respective bug reports, I think one will be fixed in 4.7, but the other will have to wait.

[SnarkBoojum]

comment:40

With sage 4.7.1, there are problems with atlas (bug #10808), boehm (huh... no bug report from me, I'll have to investigate), flint (bug #10328) and singular (bug #10810).

[SnarkBoojum]

comment:41

With sage 4.7.2alpha4, there are three problems left (or four with ubuntu) :

• boehm-gc (no report from me but I commented in fix boehm-gc so it builds on OS X Lion #11883)
• flint (Flint-1.5.2 with ARM 32-bits #10328)
• singular (Update Singular to 3-1-3-3 #10903)

and you can add a fourth when the distribution is ubuntu : readline (#11970).

[SnarkBoojum]

comment:42

The link I gave for singular is obsolete : I opened bug #12110 instead.

[jpflori]

comment:46

I think we can close this one...

[jpflori]

Reviewer: Jean-Pierre Flori