libGAP! -- create a Cython library interface to gap #6391

williamstein · 2009-06-23T21:59:35Z

There's libSingular, there's libPari, and next we need to create libGAP.

To try it out,

Install http://boxen.math.washington.edu/home/dima/packages/gap-4.5.6.spkg from Upgrade GAP to 4.5.7 #13211
Install http://www.stp.dias.ie/~vbraun/Sage/spkg/libgap-4.5.6.spkg
Apply attachment: trac_6391_libGAP.patch to the Sage library
Apply attachment: trac_6391_libGAP_root.patch to the root repository

Code in sage/libs/gap/test/ demonstrates an interface to libgap from C.

Depends on #13415
Depends on #13123
Depends on #13211

CC: @burcin @sagetrac-nborie @robertwb @sagetrac-jeremywest @gvol @sagetrac-davidm @kini @kcrisman @nthiery @jpflori @simon-king-jena

Component: group theory

Reviewer: Dmitrii Pasechnik, Ivan Andrus, Volker Braun, William Stein

Issue created by migration from https://trac.sagemath.org/ticket/6391

The text was updated successfully, but these errors were encountered:

williamstein · 2009-06-23T22:04:57Z

comment:1

To try out the first demo libgap.a use of GAP as a library:

Install this spkg: http://sage.math.washington.edu/home/wstein/patches/gap-4.4.10.p13.spkg
Apply the patch trac_6391.patch attached to this ticket and do "sage -br".

Then

sage: sage.libs.gap.gap.command()
...
> NormalSubgroups(SymmetricGroup(4));
[ Group(()), Group([ (1,4)(2,3), (1,3)(2,4) ]), Group([ (2,4,3), (1,4)(2,3), (1,3)(2,4) ]), Sym( [ 1 .. 4 ] ) ]
sage: sage.libs.gap.gap.command()
> a := 8383;
8383
sage: sage.libs.gap.gap.command()
> a^20;
2937577432790013740156622649268134038711666398939150844563902439935080138297601

williamstein · 2009-06-23T22:17:09Z

comment:2

Before the above, be sure to do

import sage.libs.gap.gap

williamstein · 2009-06-25T00:52:11Z

comment:3

Attachment: trac_6391-part2.patch.gz

Make sure to use the latest spkg, which I'll keep in the ticket description.

williamstein · 2009-06-25T18:24:11Z

Attachment: trac_6391-part3-error_handling.patch.gz

williamstein · 2009-06-27T03:49:38Z

Attachment: trac_6391-part4.patch.gz

williamstein · 2009-09-28T21:07:27Z

Attachment: trac_6391-part1.patch.gz

williamstein · 2009-09-28T21:09:17Z

comment:9

This log pretty much explains what/why/what next:

sage: import sage.libs.gap.gap as g
sage: a = g.libgap('10')
sage: a
10
sage: type(a)
<type 'sage.libs.gap.gap.GapElement'>
sage: a*a
100
sage: timeit('a*a')
625 loops, best of 3: 209 ns per loop
sage: b = gap('10')
sage: b
10
sage: b*b
100
sage: type(b)
<class 'sage.interfaces.gap.GapElement'>
sage: !ps ax |grep gap
 5497 pts/110  Ss+    0:00 /home/bober/sage-4.1/local/lib/gap-4.4.10/bin/x86_64-unknown-linux-gnu-gcc/gap -m 24m -l /home/bober/sage-4.
19751 pts/84   SNs+   0:00 /home/SimonKing/SAGE/sage-4.1.alpha1/local/lib/gap-4.4.10/bin/x86_64-unknown-linux-gnu-gcc/gap -m 24m -l /ho
27563 pts/109  Ss+    0:00 /scratch/steinz/sage/local/lib/gap-4.4.10/bin/x86_64-unknown-linux-gnu-gcc/gap -m 24m -l /scratch/steinz/sag
27572 pts/98   S+     0:00 sh -c ps ax |grep gap
27574 pts/98   S+     0:00 grep gap
sage: b.name()
'$sage1'
sage: timeit('b*b')
625 loops, best of 3: 292 µs per loop
sage: a = g.libgap('0')
sage: a
0
sage: b = g.libgap('10')
sage: b/a
hit stderr

williamstein · 2009-09-28T21:24:31Z

comment:10

Overview:

Modified version of GAP, which is built by doing

sage -f -m  http://sage.math.washington.edu/home/wstein/patches/gap/gap-4.4.10.p17.spkg
wait...
cd sage/spkg/build/gap-4.4.10.p17

To rebuild:

steinz@sage:~/scratch/sage/spkg/build/gap-4.4.10.p17/src$ make
steinz@sage:~/scratch/sage/spkg/build/gap-4.4.10.p17/src/bin/x86_64-unknown-linux-gnu-gcc$ gcc -shared *.o -o libgap.so

Cython code to use the Gap library:

In here:
   sage/devel/sage/sage/libs/gap/gap.pyx

If you change gap.pyx, do sage -br to build the changed version and start Sage.

sagetrac-davidm · 2010-04-10T23:32:02Z

comment:14

I get an error when importing libgap on sage 4.3.5 on OSX 10.6.3:

sage: import sage.libs.gap.gap
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

/Users/ayeq/sage-devel/<ipython console> in <module>()

/Users/ayeq/sage-devel/gap.pyx in init sage.libs.gap.gap (sage/libs/gap/gap.c:6286)()

/Users/ayeq/sage-devel/local/lib/python2.6/site-packages/sage/misc/cachefunc.pyc in __init__(self, f, classmethod)
     54 
     55         """
---> 56         self._common_init(f, ArgumentFixer(f,classmethod=classmethod))
     57         self.cache = {}
     58 

/Users/ayeq/sage-devel/local/lib/python2.6/site-packages/sage/misc/function_mangling.pyc in __init__(self, f, classmethod)
    107         """        
    108 
--> 109         defaults = f.func_defaults
    110         if defaults is None:
    111             defaults = []

AttributeError: 'builtin_function_or_method' object has no attribute 'func_defaults'

The cached_function decorator works at the interpreter level so I really do not see what is happening. When the line is removed the import works fine and from what I understand about cache_function this should be okay.

--- a/sage/libs/gap/gap.pyx	Sat Jun 27 05:49:14 2009 +0200
+++ b/sage/libs/gap/gap.pyx	Sat Apr 10 16:30:27 2010 -0700
@@ -71,8 +71,8 @@
 
 from sage.structure.element cimport ModuleElement, RingElement
   
-from sage.misc.cachefunc import cached_function
-@cached_function
+#from sage.misc.cachefunc import cached_function
+#@cached_function
 def gap_root():
     """
     Find the location of the GAP root install which is stored in the gap

dimpase · 2011-01-14T18:34:03Z

comment:19

Replying to @sagetrac-davidm:

I get an error when importing libgap on sage 4.3.5 on OSX 10.6.3:

 sage: import sage.libs.gap.gap
 ---------------------------------------------------------------------------
 AttributeError                            Traceback (most recent call last)
 
 /Users/ayeq/sage-devel/<ipython console> in <module>()
 
 /Users/ayeq/sage-devel/gap.pyx in init sage.libs.gap.gap (sage/libs/gap/gap.c:6286)()
 
/Users/ayeq/sage-devel/local/lib/python2.6/site-packages/sage/misc/cachefunc.pyc in __init__(self, f, classmethod)
      54 
      55         """
 ---> 56         self._common_init(f, ArgumentFixer(f,classmethod=classmethod))
      57         self.cache = {}
      58 
 
 /Users/ayeq/sage-devel/local/lib/python2.6/site-packages/sage/misc/function_mangling.pyc in __init__(self, f, classmethod)
     107         """        
     108 
 --> 109         defaults = f.func_defaults
     110         if defaults is None:
     111             defaults = []
 
 AttributeError: 'builtin_function_or_method' object has no attribute 'func_defaults'

I see exactly the same with the new spkg on OSX 10.6.
And the patch to gap.pyx you suggest works for me too.

dimpase · 2011-01-14T21:37:06Z

comment:20

Replying to @dimpase:

Replying to @sagetrac-davidm:

I get an error when importing libgap on sage 4.3.5 on OSX 10.6.3:

I see exactly the same with the new spkg on OSX 10.6.
And the patch to gap.pyx you suggest works for me too.

moreover, I get the same problem (and solition(?)) on Linux (on boxen). So this is certainly not OSX-specific.

dimpase · 2011-01-17T14:10:42Z

Attachment: 6391.6.patch.gz

cumulative patch for sage 4.6.1

dimpase · 2011-01-17T14:15:46Z

comment:21

Replying to @dimpase:

Replying to @dimpase:

Replying to @sagetrac-davidm:

I get an error when importing libgap on sage 4.3.5 on OSX 10.6.3:

I see exactly the same with the new spkg on OSX 10.6.
And the patch to gap.pyx you suggest works for me too.

moreover, I get the same problem (and solition(?)) on Linux (on boxen). So this is certainly not OSX-specific.

it turns out that cached_function won't work in Cython at all, so indeed this needs to be changed.
The cumulative patch 6391.6.patch fixed this, as well, as it supersedes the previous 4 patches.

To install for Sage 4.6.1, install the updated spkg, and this patch.

Dima

dimpase · 2011-01-18T13:34:33Z

comment:22

Perhaps someone who understands libs/gap/gap.pyx better than me can comment on few sticking points I encounter:

class Gap(ParentWithBase) is instantiated only once, for base=ZZ. Is it meant that it should be instantiated for other bases, too? And if so, which ones? (I must say it's very puzzling to me piece of code...)

in the class GapElement, _div_ and other member functions can potentially fail (as demonstrated in the comment "This log pretty much explains what/why/what next:..." above), e.g. due to division by 0. It is meant that the corresponding checks are to be implemented in these member functions? I guess not, it's rather a matter of properly catching the corresponding GAP interrupts, right?

What _dealloc_ in this class meant to do?

williamstein · 2011-01-18T16:32:51Z

comment:23

Replying to @dimpase:

class Gap(ParentWithBase) is instantiated only once, for base=ZZ. Is it meant that it should be instantiated for other bases, too? And if so, which ones? (I must say it's very puzzling to me piece of code...)

No, only once for base=ZZ.

in the class GapElement, _div_ and other member functions can potentially fail (as demonstrated in the comment "This log pretty much explains what/why/what next:..." above), e.g. due to division by 0. It is meant that the corresponding checks are to be implemented in these member functions? I guess not, it's rather a matter of properly catching the corresponding GAP interrupts, right?

Yes. I don't know if GAP uses "interrupts" though. You are basically above asking: "Here is a problem nobody has solved yet, which is why libgap hasn't moved forward. What is the solution?"

What _dealloc_ in this class meant to do?

See {{dealloc}} at http://docs.cython.org/src/reference/extension_types.html for the definition of dealloc. It is like __del__ in python.

dimpase · 2011-01-18T22:19:04Z

comment:24

Replying to @williamstein:

Replying to @dimpase:

in the class GapElement, _div_ and other member functions can potentially fail (as demonstrated in the comment "This log pretty much explains what/why/what next:..." above), e.g. due to division by 0. It is meant that the corresponding checks are to be implemented in these member functions? I guess not, it's rather a matter of properly catching the corresponding GAP interrupts, right?

Yes. I don't know if GAP uses "interrupts" though. You are basically above asking: "Here is a problem nobody has solved yet, which is why libgap hasn't moved forward. What is the solution?"

From printing contents of the GAP interpreter output_buffer (introduced in the patch of GAP's scanner.c), I see that during the fatal a.div(g.libgap('0'))_ it starts getting the contents of the usual GAP error message, like Entering break read-eval-print loop (i.e. it does a "GAP interrupt"). And then the crash happens in the Python interpreter. So I suppose more reverse engineering/understanding of the GAP interpreter is needed here, in order to avoid replicating it in Python...

vbraun · 2011-03-22T01:27:38Z

comment:25

I updated the patch to work with Sage-4.7.alpha1. I also added a bunch of doctests while trying to understand what is going on and replaced the deprecated ParentWithBase with Parent.

I'll be doing some more work on libgap during Sage Days 29, so let me know if you have any suggestions.

miguelmarco · 2012-10-29T20:13:04Z

comment:148

So the only hope for my project is to rewrite kbmag to be able to communicate with shared libraries? I am afraid that is way beyond my skills now. Anyways, if you can help me a little bit i can try to learn about it.

nbruin · 2012-10-30T07:01:15Z

comment:149

I don't think you need to do anything with shared libraries. As far as I can see, the package communicates with external stand-alone executables via Exec calls. I'm a little surprised that libgap doesn't support Gap's Exec when it does support the rest of the interpreted language. Perhaps it's just a matter of initializing the environment of the Gap interpreter more fully?

Anyway, if you can't get the Exec approach to work, it should be possible to rewrite the kbmag routines that interface with the executable so that they produce the output that needs to go into a temporary communication file in a way that does work. Then you can call the executable with whatever means are appropriate and you parse its output back. It may not be pleasant to do, but doesn't require particular skills either. You'd better be very sure it really has a benefit over using a separate GAP process, because it'll be quite a bit of work.

Grep for Exec to see where the interfacing takes place.

miguelmarco · 2012-10-30T10:24:11Z

comment:150

How hard would it be to make Gap's Exec work through libgap?

vbraun · 2012-10-30T11:33:29Z

comment:151

If kbmag only uses stdin/out to communicate with its binary component then it should work. The Exec GAP function basically works when calling it via libGAP

sage: libgap_exec = libgap.eval("Exec")
sage: libgap_exec
<Gap function "Exec">
sage: libgap_exec('echo $HOME')
/home/vbraun
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

/home/vbraun/<ipython console> in <module>()

/home/vbraun/opt/sage-5.4.rc1/local/lib/python2.7/site-packages/sage/libs/gap/element.so in sage.libs.gap.element.GapElement_Function.__call__ (sage/libs/gap/element.c:7929)()

ValueError: libGAP: Segmentation fault

There is a segfault when returning the result, I'll fix that in another ticket (maybe #13588).

How did you install kbmag, I don't think its one that we ship. Is there a spkg for it?

dimpase · 2012-10-30T13:44:47Z

comment:152

Replying to @vbraun:

How did you install kbmag, I don't think its one that we ship. Is there a spkg for it?

the googledocs link above will give you the spkg...
That's certainly not the best way to host files.
There is http://code.google.com/p/spkg-upload/
which would make much more sense.

Or, perhaps, even easier, a bitbucket or github account.

vbraun · 2012-10-30T14:09:51Z

comment:153

And open a ticket to make kbmag an experimental spkg so we have a place to discuss it. This issue is really done so nobody should post to it.

miguelmarco · 2012-10-30T17:35:50Z

comment:154

I have opened #13673.

sagetrac-tfeulner · 2012-11-09T12:03:59Z

comment:155

Thanks for this great contribution!

I just want to mention a problem with the assertion `strlen(libGAP_stdin_buffer) < length' which
leads to an unhandled SIGABRT. Could this be avoided?

sage: S = SymmetricGroup(10000)
sage: s = S.random_element()
sage: libgap(s)

vbraun · 2012-11-09T15:48:59Z

comment:156

Will be done in #13588

miguelmarco · 2012-12-26T11:06:17Z

comment:158

Now that it seems that gap upgrading has moved to 4.5.7, will this need to change?

miguelmarco · 2012-12-26T11:26:33Z

comment:159

Indeed, the actual version of libgap is incompatible with the gap-4.5.7 spkg:


sage: libgap.FreeGroup(4)
An error occurred, but libGAP has no handler set.
Error message: 
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)

/home/mmarco/sage-5.5/<ipython console> in <module>()

/home/mmarco/sage-5.5/local/lib/python2.7/site-packages/sage/misc/lazy_import.so in sage.misc.lazy_import.LazyImport.__getattr__ (sage/misc/lazy_import.c:1877)()

/home/mmarco/sage-5.5/local/lib/python2.7/site-packages/sage/misc/lazy_import.so in sage.misc.lazy_import.LazyImport._get_object (sage/misc/lazy_import.c:1291)()

/home/mmarco/sage-5.5/libgap.pyx in init sage.libs.gap.libgap (sage/libs/gap/libgap.c:4157)()

/home/mmarco/sage-5.5/libgap.pyx in sage.libs.gap.libgap.Gap.__init__ (sage/libs/gap/libgap.c:2716)()

/home/mmarco/sage-5.5/local/lib/python2.7/site-packages/sage/libs/gap/util.so in sage.libs.gap.util.initialize (sage/libs/gap/util.c:3489)()

RuntimeError: libGAP initialization failed


You are running a GAP kernel which does not fit with the library.
Probably you forgot to apply the kernel part or the library part
of a bugfix?

If you are using Windows, make sure you installed the file
wbin4r5n7.zoo (or .zip),
Macintosh users make sure the file
bin4r5n7-PPC.sit (or -68k.sit) is installed,
Unix users please recompile.

Error before error-handling is initialized: Update to correct kernel version!

dimpase · 2012-12-26T12:01:11Z

comment:160

Replying to @miguelmarco:

Indeed, the actual version of libgap is incompatible with the gap-4.5.7 spkg:

SPKG.txt contains brief instructions about upgrading. You'll need to increase the version and rerun autotools to produce new spkg. I never tried this myself though.

vbraun · 2012-12-26T12:23:07Z

comment:161

I'm working on it.

dimpase · 2012-12-26T12:30:39Z

comment:162

Replying to @vbraun:

I'm working on it.

I've built the new package from your bitbucket repo, and it at least installs OK.

dimpase · 2012-12-26T12:48:29Z

comment:163

Replying to @dimpase:

Replying to @vbraun:

I'm working on it.

I've built the new package from your bitbucket repo, and it at least installs OK.

http://boxen.math.washington.edu/home/dima/packages/libgap-4.5.7.p1.spkg

It also passes the tests in sage/libs/gap

To create the package on MacOSX, I had to install GNU findtools, for the native find has different options.

vbraun · 2012-12-26T13:06:53Z

comment:164

The libgap source repository is of course at the state of #13588. I've forward-ported all changes there. Since this ticket is now superseded by the new GAP version, it should be closed as duplicate. One of you guys should review #13588 :-)

miguelmarco · 2012-12-26T13:14:28Z

comment:165

So the patches to the sage library and the root repository should be moved to #13588, right?

I am techically unable to review the code, all i can do is test it and report.

vbraun · 2012-12-26T13:18:16Z

comment:166

Yes, I've moved the patches from here to #13588 to make Jeroen's life easier.

jdemeyer · 2013-01-21T09:34:35Z

Changed author from Dima Pasechnik, Ivan Andrus, Volker Braun, William Stein to none

jdemeyer · 2013-01-21T09:34:35Z

Changed reviewer from Dima Pasechnik, Ivan Andrus, Volker Braun, William Stein to Dmitrii Pasechnik, Ivan Andrus, Volker Braun, William Stein

williamstein added this to the sage-5.4 milestone Jun 23, 2009

williamstein added c: interfaces labels Jun 23, 2009

williamstein self-assigned this Jun 23, 2009

This comment has been minimized.

Sign in to view

williamstein added t: enhancement and removed t: bug labels Jun 25, 2009

This comment has been minimized.

Sign in to view

jdemeyer added this to the sage-5.7 milestone Dec 21, 2012

jdemeyer removed the pending label Dec 21, 2012

vbraun removed this from the sage-5.7 milestone Dec 26, 2012

jdemeyer removed the s: positive review label Dec 27, 2012

jdemeyer closed this as completed Dec 27, 2012

jdemeyer added r: duplicate and removed p: major / 3 labels Dec 27, 2012

miguelmarco mentioned this issue May 14, 2012

Free Groups #12339

Closed

vbraun mentioned this issue Dec 26, 2012

libGAP! -- create a Cython library interface to gap #13588

Closed

miguelmarco mentioned this issue Dec 4, 2012

Parents for groups #13687

Closed

libGAP! -- create a Cython library interface to gap #6391

libGAP! -- create a Cython library interface to gap #6391

Comments

williamstein commented Jun 23, 2009

williamstein commented Jun 23, 2009

williamstein commented Jun 23, 2009

williamstein commented Jun 25, 2009

This comment has been minimized.

williamstein commented Jun 25, 2009

This comment has been minimized.

This comment has been minimized.

williamstein commented Jun 27, 2009

This comment has been minimized.

williamstein commented Sep 28, 2009

This comment has been minimized.

williamstein commented Sep 28, 2009

williamstein commented Sep 28, 2009

sagetrac-davidm mannequin commented Apr 10, 2010

This comment has been minimized.

dimpase commented Jan 14, 2011

dimpase commented Jan 14, 2011

dimpase commented Jan 17, 2011

dimpase commented Jan 17, 2011

dimpase commented Jan 18, 2011

williamstein commented Jan 18, 2011

dimpase commented Jan 18, 2011

vbraun commented Mar 22, 2011

miguelmarco commented Oct 29, 2012

nbruin commented Oct 30, 2012

miguelmarco commented Oct 30, 2012

vbraun commented Oct 30, 2012

dimpase commented Oct 30, 2012

vbraun commented Oct 30, 2012

miguelmarco commented Oct 30, 2012

sagetrac-tfeulner mannequin commented Nov 9, 2012

vbraun commented Nov 9, 2012

miguelmarco commented Dec 26, 2012

miguelmarco commented Dec 26, 2012

dimpase commented Dec 26, 2012

vbraun commented Dec 26, 2012

dimpase commented Dec 26, 2012

dimpase commented Dec 26, 2012

vbraun commented Dec 26, 2012

miguelmarco commented Dec 26, 2012

vbraun commented Dec 26, 2012

jdemeyer commented Jan 21, 2013

jdemeyer commented Jan 21, 2013