"How to call C code from Sage" thematic tutorial

[nathanncohen]

This branch adds to Sage's thematic tutorial an explanation of how to interface a C code (or a compiled library) with Sage. It does not replace the developer's manual section entitled "Packing third-party code" as it is only a way to make it work "on your own compputer", without the complications that are only cause by Sage's infrastructure.

I have been asked the same instructions several times, and with this we will have a link to provide instead of the same explanation and the example files.

Nathann

CC: @sagetrac-azi @jm58660 @vbraun @kcrisman @anneschilling

Component: documentation

Author: Nathann Cohen

Branch/Commit: 2f37b6f

Reviewer: Karl-Dieter Crisman, David Coudert

Issue created by migration from https://trac.sagemath.org/ticket/17605

[nathanncohen]

Branch: public/17605

[sagetrac-git]

Commit: 462b6e2

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

462b6e2

trac #17605: "How to call C code from Sage" thematic tutorial

[kcrisman]

comment:3

I don't know enough about this to say whether it is all correct, but it looks very helpful (at least, to me!). Did you stop early, though?

+Calling a function defined in a library
+---------------------------------------
+
+  ~/a$ gcc -c -fPIC double_vector.c -o libdoublevector.o

I guess you have compiled the library file, but where is the calling part?

[nathanncohen]

comment:4

I am an idiot. I made many attempts to link this .pyx file with a compiled library and failed, and this is just the beginning of the section that I do not know how to write. I will update my patch in a second to remove that.

I would love to write it, but I just don't know how. There are quite some things that I know how to do by modifying sage, but not without that :-P

Nathann

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

a846240

trac #17605: "How to call C code from Sage" thematic tutorial

[sagetrac-git]

Changed commit from 462b6e2 to a846240

[nathanncohen]

comment:6

I don't know enough about this to say whether it is all correct, but it looks very helpful (at least, to me!).

Well, the point is actually that with this you can give it a try and witness that it does work ;-)

Volker? Do you know how to link to a compiled library while outside of Sage? Can you teach me? And I'll write the doc!

Nathann

[vbraun]

comment:8

Note: An object file (.o) is not a library (.a / .so).

You can link compiled C/C++ code with a stand-alone Cython project like in https://github.com/vbraun/lattice_phi4. Then just import as module in Sage.

[nathanncohen]

comment:9

Hello Volker,

You can link compiled C/C++ code with a stand-alone Cython project like in https://github.com/vbraun/lattice_phi4. Then just import as module in Sage.

I tried it but I was not able to compile it. It seems that I am missing some headers.

I kept on whipping my test files and go to something new, though: I was able to make it work provided that I copied libdoublevector.so to SAGE_ROOT/local/lib/.

Now, I am trying to make it all run from my working directory but I meet the following problem:

• When calling "%runfile double_vector_sage.pyx" there are two calls to GCC
• The one which contains -ldoublevector is not the one that contains -L/home/ncohen/a

As a result the second call always fails, because the library is not found.

sage: %runfile double_vector_sage.pyx
Compiling ./double_vector_sage.pyx...
...
RuntimeError: Error compiling ./double_vector_sage.pyx:
running build
running build_ext
building '_home_ncohen_a_double_vector_sage_pyx_5' extension
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -fPIC
 -I/home/ncohen/.Sage/local/include/csage -I/home/ncohen/.Sage/local/include
 -I/home/ncohen/.Sage/local/include/python2.7 
 -I/home/ncohen/.Sage/local/lib/python/site-packages/numpy/core/include 
 -I/home/ncohen/.Sage/src/sage/ext -I/home/ncohen/.Sage/src 
 -I/home/ncohen/.Sage/src/sage/gsl -I. -I/home/ncohen/.Sage/local/include/python2.7 
 -c _home_ncohen_a_double_vector_sage_pyx_5.c 
 -o build/temp.linux-x86_64-2.7/_home_ncohen_a_double_vector_sage_pyx_5.o 
 -w -O2 -L/home/ncohen/a/
gcc -pthread -shared -L/home/ncohen/.Sage/local/lib build/temp.linux-x86_64-2.7/_home_ncohen_a_double_vector_sage_pyx_5.o 
 -L/home/ncohen/.Sage/local/lib/ -L/home/ncohen/.Sage/local/lib -ldoublevector 
 -lmpfr -lgmp -lgmpxx -lstdc++ -lpari -lm -lec -lgsl -lgslcblas -latlas -lntl 
 -lcsage -lpython2.7 -o build/lib.linux-x86_64-2.7/_home_ncohen_a_double_vector_sage_pyx_5.so 
 -L/home/ncohen/.Sage/local/lib

The header of my .pyx file is:

#clib doublevector
#cargs -L/home/ncohen/a/

Thanks to you if you can give me some hint. In the meantime I'll be trying again to make it work.

Nathann

[sagetrac-git]

Changed commit from a846240 to 2ad9b51

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

2ad9b51

trac #17605: Make it work for libraries

[nathanncohen]

comment:11

Okay found a way ;-)

Needs review again !

Nathann

[vbraun]

comment:13

There is probably some alternate reality where the new description is useful for others to read/search, I just lack the imagination for it.

Once you install the shared library into Sage you might just as well go through module_list.py as well.

[nathanncohen]

comment:14

There is probably some alternate reality where the new description is useful for others to read/search, I just lack the imagination for it.

There is probably one in which comments like that don't make people uselessly angry. Are we in the reality in which you also review the branch ?

Once you install the shared library into Sage you might just as well go through module_list.py as well.

I do not understand this comment.

Nathann

[vbraun]

comment:15

Replying to @nathanncohen:

Once you install the shared library into Sage you might just as well go through module_list.py as well.

I do not understand this comment.

Presumably you want to build a stand-alone C/C++ and Cython project without having to install stuff into Sage. By definition, this should exclude steps that modify Sage directly. I mentioned the code in my github repo, this is the only way to do this currently: Write Makefile that calls gcc and cython directly.

Also, you shouldn't build shared libraries the way you describe. Its very platform dependent and your steps won't work under OSX, say.

[nathanncohen]

comment:16

Presumably you want to build a stand-alone C/C++ and Cython project without having to install stuff into Sage. By definition, this should exclude steps that modify Sage directly.

I do not understand. In the tutorial no instruction advises the user to change anything inside of Sage. What did you see that I missed?

I mentioned the code in my github repo, this is the only way to do this currently: Write Makefile that calls gcc and cython directly.

I want to make it as simple as possible, and the current instructions avoid that.

Also, you shouldn't build shared libraries the way you describe. Its very platform dependent and your steps won't work under OSX, say.

Oh. Could you tell me how to build this library under OSX ? I will update the tutorial to explain both situations.

Nathann

[vbraun]

comment:17

Copying files to SAGE_ROOT/local/lib/ is modifying Sage in my book. E.g. on a global installation (like SMC) you just don't have permissions to do that.

[nathanncohen]

comment:18

Copying files to SAGE_ROOT/local/lib/ is modifying Sage in my book. E.g. on a global installation (like SMC) you just don't have permissions to do that.

The tutorial does not do that.

Nathann

[kcrisman]

comment:19

Well, the point is actually that with this you can give it a try and witness that it does work

Sadly, not today :( though it would be more fun than what I have to do. Though I would recommend removing the ncohen references to something more generic, no doubt an oversight.

Copying files to SAGE_ROOT/local/lib/ is modifying Sage in my book.

The tutorial does not do that.

I think Volker is referring to the earlier comment

I kept on whipping my test files and go to something new, though: I was able to make it work provided that I copied libdoublevector.so to SAGE_ROOT/local/lib/.

which however I think you are right is not in the tutorial, you seem to have taken a different approach.

As for the OS X, I think the .so files are called .dylib (at least sometimes) and on Cygwin .dll... but I am not an expert in such things. Good luck! This would indeed be a good tutorial to have.

[nathanncohen]

comment:20

Hello,

Sadly, not today :( though it would be more fun than what I have to do. Though I would recommend removing the ncohen references to something more generic, no doubt an oversight.

Okay I can do that... I saw Jeroen's home path in several documents of the developer's manual, so I thought it did not matter.

I think Volker is referring to the earlier comment

Yes, probably.

As for the OS X, I think the .so files are called .dylib (at least sometimes) and on Cygwin .dll... but I am not an expert in such things. Good luck! This would indeed be a good tutorial to have.

HMmm.. Yep, but I need help to write that. I never used those architectures.

Nathann

[kcrisman]

comment:21

I saw Jeroen's home path in several documents of the developer's manual, so I thought it did not matter.

Probably not that much, but might as well make it more generic if you have the opportunity.

[nathanncohen]

comment:22

Probably not that much, but might as well make it more generic if you have the opportunity.

http://www.sagemath.org/doc/developer/doctesting.html

I'll fix them all...

Nathann

[nathanncohen]

comment:25

Replying to @kcrisman:

I'll fix them all...

Well, I'd make that a separate ticket

I changed those introduced by this branch.

Because it could be that several other people, including yourself, disagree with me about this

For this kind of stuff, let's say that I prefer to make the changes and get my ticket reviewed.

My personal feeling is that usually it's best to say something early on, as we do in some tutorials, like "Here we assume that your home directory and username are username/ or something analogous to that.

I hope that just changing the prompt will make it (more) obvious in this case.

Nathann

[kcrisman]

comment:26

I hope that just changing the prompt will make it (more) obvious in this case.

Yes, this does a good job.

[kcrisman]

Reviewer: Karl-Dieter Crisman

[kcrisman]

comment:27

Issues I found.

• My compiler gives this.

$ gcc hello.c -o hello; ./hello
hello.c: In function ‘main’:
hello.c:7: warning: return type of ‘main’ is not ‘int’
Hello World

• Inconsistent names here:

+  def my_interface_function():
+      hello_world() # This is the C function from hello.c
...
+  sage: my_bridge_function()
+  Hello World

• Compiler error:

sage: %runfile hello_sage.pyx
Compiling ./hello_sage.pyx...
Error compiling cython file:
Error converting ./hello_sage.pyx to C:


Error compiling Cython file:
------------------------------------------------------------
...
include "interrupt.pxi"  # ctrl-c interrupt block support
include "stdsage.pxi"  # ctrl-c interrupt block support

include "cdefs.pxi"
cdef extern from "hello.c":
void hello_world()
^
------------------------------------------------------------

_Users_..._hello_sage_pyx_0.pyx:7:0: Expected an increase in indentation level

• Even after fixing this:

sage: %runfile hello_sage.pyx
Compiling ./hello_sage.pyx...
Error compiling cython file:
Error compiling ./hello_sage.pyx:
running build
...
_Users_..._hello_sage_pyx_0.c:349:19: fatal error: hello.c: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1

Maybe my file needs to be in my PATH? But that didn't seem to help.

• And since presumably I needed to compile double_vector.c first, I tried that, but...

$ gcc double_vector.c -o double_vector
double_vector.c: In function ‘sum_of_two_vectors’:
double_vector.c:4: warning: incompatible implicit declaration of built-in function ‘malloc’
Undefined symbols for architecture x86_64:
  "_main", referenced from:
      start in crt1.10.6.o
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status

SO tells me there are two errors - need to include stdio.h and then compile with -c or provide a main function. Still get the weird error about it not existing when I try to %runfile, though.

• Similar problems with the final library example.

[nathanncohen]

comment:28

Yo ! Thank you for looking at this.

• My compiler gives this.

$ gcc hello.c -o hello; ./hello
hello.c: In function ‘main’:
hello.c:7: warning: return type of ‘main’ is not ‘int’
Hello World

My compiler does not say a word. I changed the return type of main to int and added a 'return 0'. I would prefer to remove this return 0 (my compiler does not complain either if I do) for that is irrelevant information to what I want to illustrate. Can you tell me what it gives for you ?

• Inconsistent names here:

Fixed.

• Compiler error:

I fixed the indentation level.

• Even after fixing this:

sage: %runfile hello_sage.pyx
Compiling ./hello_sage.pyx...
Error compiling cython file:
Error compiling ./hello_sage.pyx:
running build
...
_Users_..._hello_sage_pyx_0.c:349:19: fatal error: hello.c: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1

This I do not understand. Is hello_sage is in the same folder as hello.c ? O_o I really do not understand that.

Maybe my file needs to be in my PATH? But that didn't seem to help.

No no, you should not have to. And I would prefer to avoid asking the user to change the PATH anyway O_o

• And since presumably I needed to compile double_vector.c first, I tried that, but...

Nono, you do not need to compile it by itself (and when you do it complains because no 'main' function is defined, which is correct). It gets compiled when you call "runfile". Actually, you do not need to compile 'hello.c' at first. It is just to give an example. If it gets complicated we can remove this compilation of "hello world", actually. It is not necessary.

• Similar problems with the final library example.

When you say "similar problem", is it also about 'main' ? O_o

Nathann

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

bd0dae3

trac #17605: Errors...

[sagetrac-git]

Changed commit from 96177d8 to bd0dae3

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

23139b5

trac #17605: Remove useless (and problematic) compilation (+indentation fixes)

[sagetrac-git]

Changed commit from bd0dae3 to 23139b5

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

5f20f0a

trac #17605: Remove useless (and problematic) compilation (+indentation fixes)

[sagetrac-git]

Changed commit from 23139b5 to 5f20f0a

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

2f37b6f

trac #17605: Syntax highlighting

[sagetrac-git]

Changed commit from 5f20f0a to 2f37b6f

[dcoudert]

Changed reviewer from Karl-Dieter Crisman to Karl-Dieter Crisman, David Coudert

[dcoudert]

comment:35

Hello,

I tried with my MBA/OSX Yosemite.
The documentation is clear and useful.
I'm able to reproduce all examples without error/warning.

Unless others have difficulties on other systems, I vote for positive review.
Let me know if I can change the status.

David.

[nathanncohen]

comment:36

Yo,

I tried with my MBA/OSX Yosemite.
The documentation is clear and useful.
I'm able to reproduce all examples without error/warning.

Oh, so it works on Mac too ? Cool !

Unless others have difficulties on other systems, I vote for positive review.
Let me know if I can change the status.

Well I'd vote for that.. And if there is something wrong with some distribution, we can add a note later. And hopefully in the meantime the guy will figure out how to make it work.

Nathann

[dcoudert]

comment:37

So let's go ;)

[kcrisman]

comment:38

This also works for me now, though I have no idea why. Maybe because we removed the original compilation and Sage compiled it wherever Sage compiles things.

[nathanncohen]

comment:39

This also works for me now, though I have no idea why. Maybe because we removed the original compilation and Sage compiled it wherever Sage compiles things.

Ahahah. Well, remember? "Theory is when you know everything but nothing works, practice is when everything works but nobody knows why" :-P

Nathann

[vbraun]

Changed branch from public/17605 to 2f37b6f