Python bindings

[traverseda]

Noticed that there isn't currently a ticket for python bindings. I for one would find that useful.

[rcmz]

Hey,
I'm trying to create python bindings.
Is there some place to discuss how libfive works ? Or someone who could write a bit on how everything is structured ?
I'm still quite a beginner programmer so this may be a little out of my league.

[traverseda]

Probably right here is the best place to discuss this for the time being.

[ghost]

If you haven't seen this website:

https://www.libfive.com/

it is a good place to start. Have a look at the stack diagram on the front page.

A language binding involves re-implementing the Scheme bindings and Standard library layers, at least. To view models, you can either export to STL and use a STL viewer or re-implement the Studio layer.

The libfive.h file in the C API layer has the C functions you call to implement your language binding:

https://github.com/libfive/libfive/blob/master/libfive/include/libfive.h

The Libfive Scheme bindings call functions from this API to implement primitives:

https://github.com/libfive/libfive/blob/master/libfive/bind/libfive-guile.cpp

The standard library builds upon these primitives to build shape, transforms, CSG operations, etc.:

https://github.com/libfive/libfive/blob/master/libfive/bind/shapes.scm
https://github.com/libfive/libfive/blob/master/libfive/bind/transforms.scm
https://github.com/libfive/libfive/blob/master/libfive/bind/csg.scm

Libfive also uses sandboxing, which complicates things a little.

For a simplified model viewer, there is Inspekt3d, written in Scheme, which could be ported to Python:

https://github.com/sjm-tl-gh/inspekt3d

Other ideas for model viewing with Python bindings:

PyInventor: https://github.com/TheHubbit/PyInventor
Pivy: https://github.com/FreeCAD/pivy
Jupyter notebooks: https://jupyter.org/

Currently, doing a Libfive language binding involves a lot of code reading. Libfive does not have a mailing list or wiki, as far as I know, so this Github issue tracker is the place to get help.

[rcmz]

Wow thanks a lot ! This for sure gives a clear view of how everything is structured.
I thing I'm starting to get a good grasp on this library.
I'm having a bit of a difficult time understanding the Scheme bindings because I'm not familiar with this language but having to learn new programming paradigms is always a good thing ^^

Your project is really cool. I think more advanced users will prefer using their own editor and view the model in a separate window.

[mkeeter]

That's a great overview, thanks @sjm-tl-gh!

Let me clarify a few details:

• There are two approaches to writing a binding:
  • You can write the binding directly in C / C++, which is what I did for Scheme, using Guile's C API. In this binding, Scheme files (e.g. transforms.scm) are pasted into the source at compile-time and built into a single shared library. This is easy to distribute, but requires more C / C++ knowledge.
  • Alternatively, you could write the binding purely in the target language, using its FFI interface to libfive (e.g. ctypes or cffi in Python). In this case, you would work with functions from libfive.h, calling them directly from Python. This is probably an easier way to get started.
• The sandboxing is part of the Scheme bindings for the GUI, not inherent to libfive itself. You don't need to do that for Python bindings (and in fact, sandboxing Python is notoriously difficult), but that means that scripts will be able to execute arbitrary code.
• Right now, you'll need to re-implement whichever parts of the standard library you want to use (e.g. the shape definitions). I'd like to develop a language-independent format for the standard library, but am still working out a few design challenges.

I've added a bunch of API docs in 59a7c99, which should help explain what functions in libfive.h actually do.

[rcmz]

Hey guys, I've done a bit of work: https://gitlab.com/rcmz0/libfivepy
This is a simple implementation using ctypes.
I know this is still pretty rough but do you think this is going in the good direction ?

I have a couple of questions:

• Most of the time, basic functions have been implemented using libfive. For example if you type cos(3)
  it uses the libfive kernel. But, I cannot overwrite basic operators for basic types, so 4 - 3 uses python and not libfive. Is this a big dill ?
  You may be able to get around that with some sort of wrapper but the the syntax would be less elegant.
  Also note that I have overwritten min and max which is pretty strange and might cause problems.

• Right now every cgs operation or transform is implemented using functions, so to do a union you use union(shape_a, shape_b). Do you think it would be a good idea to create a Shape class and overload operators so that for example union(shape_a, shape_b) would become shape_a + shape_b.
  I think this might be simpler for beginners but in the end it is pretty confusing for more advanced users.

[ghost]

@rcmz When I try https://gitlab.com/rcmz0/libfivepy, I get "An error occurred while loading commit signatures" at the top of the page in a red box. There is no source code shown.

Re: your second question, for an infix expression language like Python, it might be more elegant to use infix as much as possible. Libfive uses F-rep, which is functional representation, and allows adding new functions. I may be wrong, but Python doesn't support adding infix operators as a language feature--but there are some hacks like: http://code.activestate.com/recipes/384122/.

Unless you can come up with a method of generally extending infix notation, at some point users are going to see prefix function calls. That's an advantage of Lisp for the modeling language, which uses prefix notation uniformly. Of course, Python developers are used to looking at mixed expressions, so they might not be too confused.

[kovasb]

Also looking forward to python bindings to this project :)

[rcmz]

@sjm-tl-gh Sorry about the gitlab error it should be fixed now.

The trick you showed for infix operators in python is pretty cool. It is quite hacky but it is an idea to keep in mind.

[ghost]

@rcmz Your repository is now visible. Looks like you have a good start! Re: your previous question 1, at about:

https://github.com/libfive/libfive/blob/master/libfive/bind/libfive-guile.cpp#L380

in Libfive, you can see where Matt has implemented generic arithmetic operations for shapes with function overloading. (Note that at this point, a lower level tree becomes a higher level <shape> class). A Python number has to be converted to a Libfive const tree (shape) before using it with the library. The ensure-shape function does this in the generic arithmetic code. See how each operator and function is overloaded?

When you get to adding vectors, the Libfive vector code also uses generic arithmetic. I'm not sure what Python has built-in for vector support, but I found this helpful:

https://github.com/libfive/libfive/blob/master/libfive/bind/vec.scm

when writing my Common Lisp binding to Libfive:

https://common-lisp.net/viewvc/tovero/trunk/src/tovero/

I also factored out the generic arithmetic code for shapes into a separate file:

https://common-lisp.net/viewvc/tovero/trunk/src/tovero/generic.lisp?view=markup

I took the second approach @mkeeter mentioned earlier, using dynamic FFI rather than a C++ code extension library like Libfive Scheme. In general, though, I translated the Libfive binding code fairly closely, as it is well organized (and works!).

[mkeeter]

@rcmz I'd recommend wrapping the math tree object (libfive_tree) in a Python class. Inside this class, you could override all of the basic math functions (e.g. __add__ and __radd__) so that they dispatch to libfive_tree_binary, etc. This class would also be responsible for free the pointer (with libfive_tree_delete).

Here's a very minimal example (that only overloads addition), with a sample interaction at the bottom.

The only issue, as @sjm-tl-gh mentioned, is prefix operations (like min or max). These can't be overloaded by a class. A few options:

• Clobber them in the global namespace (by telling people to do from libfive import *)
• Trust people to call libfive.min(...) when the arguments could contain Tree objects
• Encourage people to use functions with libfive-specific names (like union(...) instead of min).

[ghost]

@mkeeter, your code for shape addition illustrates in Python what I was trying to say in my last post. However, you are overloading the built-in operators "inside" the class, rather than "externally" with generic functions like GOOPS/CLOS. Your code also illustrates the equivalent of ensure-shape in the operator in case you get a Python number as an operand.

It looks @rcmz has started doing it similarly, except with a Tree class rather than Shape I think it does make sense at this level of abstraction to add the "shape" concept for users--it took me a while to understand this though.

I don't know what approach would be best for issues like with min. In my own binding, I had an issue with built-in symbols conflicting with Libfive function names, and I just renamed them--but there probably would be a better solution if I had more time to look at it.

As for the @rcmz 's question:

4 - 3 uses python and not libfive. Is this a big dill ?

I think you want to use Python for 4 - 3 (as well as cos(3))--you might want the actual Python number as a result. On the other hand, shape1 - shape2 or cos(shape3) would use Libfive--the dispatching of these is what @mkeeter was talking about in his last post.

[mkeeter]

Oh yeah, it looks like @rcmz is already past the point of my example – I hadn't looked at their code before writing out my reply.

I agree that you should use Python for basic arithmetic, only returning libfive objects if the result is a Tree; doing operator overloading will accomplish this. If you want to clobber min, you could do the dispatching in the binding's function, e.g. constructing a Tree if either of the arguments are trees, and otherwise calling Python's built-in min function.

[rcmz]

Thanks for your answers, I have renamed Tree to Shape and started the work to use libfive functions only when dealing with libfive objects. I also made sure that pointers where freed correctly (But their might probably be some memory leaks somewhere ).

I added/changed quite a bit of stuff but the main one is that you can use transforms and cgs operations OOP style. So for example move(sphere(1), Vector(1,0,0)) can be written as sphere(1).move(1,0,0).
This is pure syntax sugar and you can still use the functional style if you prefer.
All you have to do to change you functions into methods is to add the @shape_method decorator on top of your function. For example :

@shape_method
def move(shape, delta):
        return shape.remap(x-delta.x, y-delta.y, z-delta.z)

[rcmz]

Quick update: You can now use | (or) and & (and) as union and intersection (which makes sense), and also ^ (xor) as difference (which doesn't make much sense but is useful)

[traverseda]

Personally I'd go for

+ being union, - being difference, and probably / being intersection. They fit just about as well as the current symbols, but are more common. I'd save the more esoteric symbols |&%^ etc for the more esoteric operations, like minkowski sums.

Really - representing difference makes a lot of sense, you're literally subtracting one shape from another.

[mkeeter]

I don't think it makes sense to overload arithmetic operators for CSG; because shapes are math expressions under the hood, those operators already have meaning as arithmetic.

For example, sphere(r=1) is just lambda x, y, z: x**2 + y**2 + z**2 - 1. This means that sphere(r=1) + 2 should become lambda x, y, z: x**2 + y**2 + z**2 - 1 + 2, not lambda x, y, z: min(x**2 + y**2 + z**2 - 1, 2)

[traverseda]

Well python is a duck typed language. It should be pretty easy to have it subtract number as in your example, but subtract shapes as a CSG operation?

While your CAD kernal is a lot more powerful then something like openscad, I really don't expect most of the people who use it from python to be using it like that, and I think it should be optimized for the most common use case.

When you're doing an operation on a shape you're doing an operation on a shape. A shape is an abstraction for a math expression, they're not the math expression them self. We need to be talking about shapes or math, they are not the same level of abstraction.

I'd recommend not mixing your level of abstraction on this, and that you treat shapes as shapes for the most part. Don't leak the math up to the easier to use abstraction, especially when there's a lovely lisp environment for the people who like math-based paradigms.

[omgitsraven]

but then what WOULD you recommend for an offset operation? That's one I use a lot, and I'd be expecting to think of it as addition to the distance field...

I second | and &, they actually describe what's going on. You aren't "literally subtracting one shape from another", that would mean decreasing the first shape by the entirety of the second shape, rather than just decreasing it by the amount that they overlap by!

[traverseda]

I would recommend something like object.move([0,0,2]) or (object-object.move([0,0,1])).move([0,1,0]).

But, I mean, duck-typing. You can have it do different things for operations on numbers or on shapes.

I guess my question is, if you want it to work like that, why not use the scheme bindings instead? It seems like the scheme bindings would better fit how you like to program, where as the python bindings should be pythonic.

[mkeeter]

It's not number-vs-shape, it's "math expression that represents a shape" vs "math expression that represents a math expression", which is a distinction that's subtle-verging-on-nonexistent.

For example, should Shape(lambda x, y, z: x) be treated as a math expression or a shape, if you do arithmetic with it? (using lambda syntax as shorthand here)

You could potentially add a flag to the Shape class that determines whether addition and subtraction act as arithmetic or CSG operations, but then you run into issues like @omgitsraven described.

The most consistent behavior is to avoid that distinction altogether: have + and - always do arithmetic, and | and & always do CSG. You can also define functions with user-friendly names like dilate, union, difference, so that folks don't need to think about the underlying math.

[traverseda]

Alright, how would it work is the shapes weren't math expressions at all? If there were just platonic (heh) ideal shapes handed down from the gods, with the intent of matching people's intuition about what shapes are?

[omgitsraven]

If that's what people wanted, why would they even be using libfive?

[mkeeter]

If folks want to treat shapes as purely abstract objects, then they can use union / intersection / dilate / other names that are agnostic to the underlying representation.

If you want to do arithmetic on shapes, it makes sense to think about the underlying representation. It also makes it easier to build shapes using Python-like syntax, where you can say

x = Shape(lambda x, y, z: x)
y = Shape(lambda x, y, z: y)
circle = x**2 + y**2 - 1

(since the arithmetic operations are overloaded to do the same thing as Python's arithmetic, but producing math trees)

[rcmz]

I like the idea of shapes as objects but one of the great thing about libfive is that it enables you to use and understand shapes as functional representation, which is much more powerful.
And using | and ^ is not that hard to grasp if you understand + and -.

[traverseda]

Because it's a lot easier for libraries to implement things like fillet unions, chamfers, pipes, shape-blending, etc.

But my expectation is that most of the people using it for CAD aren't operating at that level, they're operating at a level closer to openscad. I'd expect libraries like the above operate at that lower-level. Meeting the expectations of the 90% of people who pick it up to use it like openscad is, in my opinion, more important then meeting the expectations of people who are going to dive into libfive in depth.

I think that it should meet the expectations of low-skill users first, having a smooth learning curve for high-skill users.

Ultimately I'd like to see it as part of the scientific python ecosystem, along side conda (on windows), numpy, scipy, etc. My hope is that this could enable truly reusable CAD modules, something openscad never managed to pull off, along side things like finite-element-analysis and geometric constraint solvers. This could allow the building of specialized CAD tools for things like architecture or circuit-board fabrication, as python libraries.

[traverseda]

See also: We need less powerful languages.

Every increase in expressiveness brings an increased burden on all who care to understand the message.

[rcmz]

I tried earlier to create a Shape class with arrhythmic operator overloading.
But the syntax needed was a lot more heavy, you had to do some aggressive casting and access instance members all the time.
So I don't think it is worth it just to have arithmetic operators when we can have &, | and ^.

But if you really wanna try it I encourage you do so the project is open-source.

[traverseda]

But if you really wanna try it I encourage you do so the project is open-source.

I'm currently getting a segfault whenever I try to run example.py, so it'll take me a while to sort that out.

[rcmz]

Sorry this project is still really rough.
You need to have libfive.so somewhere in your path (or in your libfivepy directory).

[traverseda]

It loads the library fine, but it segfaults. I haven't done a lot with ctypes before, so I'm not sure how to go about debugging that.

[rcmz]

I have committed the latest changes so I'm sure this build is working on my machine.
You need to use python3.
Does libfive studio works ?

[traverseda]

I tried earlier to create a Shape class with arrhythmic operator overloading.
But the syntax needed was a lot more heavy, you had to do some aggressive casting and access instance members all the time.

My immediate thought is to provide two interfaces, a pythonic holds-your-hand interface that matches user expectations, and an object.raw interface for accessing the mathematical primitives.

Honestly there are much more important things then this, which is an ideological thing. It's not a point I'm going to fight very hard on until things are a fair bit more mature.

You need to use python3.

Python 3.7.0

Does libfive studio works ?

Yep, I've loaded the tutorial shape successfully.

I am using archlinux, which means my libfive-studio is

Branch: master
Revision: 47a8454

[ghost]

Re: Libfive language design in Python

For comparison, it might help to look at an example of a 3D modeling system already written in Python. Here's a model in CadQuery:

https://github.com/dcowden/cadquery/blob/master/examples/FreeCAD/Ex032_3D_Printer_Extruder_Support.py

It integrates into Juypter notebooks:

https://mybinder.org/v2/gh/RustyVermeer/tryCQ/master

At the Jupyter notebook level, there are GUI fields for model parameters which might appeal more to non-programmers. It becomes "declarative modeling". (It would be nice to have both Guile and Python versions of this for Libfive.)

CadQuery apparently uses the "concatenate function calls" syntax like (object-object.move([0,0,1])).move([0,1,0]).

[traverseda]

(gdb) run
Starting program: /usr/bin/python example.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Loaded <CDLL 'libfive.so', handle 555555638890 at 0x7ffff7115198>

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6db08de in libfive_tree_remap () from /usr/lib/libfive.so
(gdb) info stack
#0  0x00007ffff6db08de in libfive_tree_remap () from /usr/lib/libfive.so
#1  0x00007ffff6e4e1c8 in ffi_call_unix64 () from /usr/lib/libffi.so.6
#2  0x00007ffff6e4dc2a in ffi_call () from /usr/lib/libffi.so.6
#3  0x00007ffff70de6b5 in _ctypes_callproc () from /usr/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so
#4  0x00007ffff70df040 in ?? () from /usr/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so
#5  0x00007ffff7b7485c in _PyObject_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#6  0x00007ffff7bc0c97 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#7  0x00007ffff7b482fb in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#8  0x00007ffff7bbbb92 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#9  0x00007ffff7b00ee9 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#10 0x00007ffff7b484a2 in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#11 0x00007ffff7bbbb92 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#12 0x00007ffff7b00ee9 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#13 0x00007ffff7b484a2 in _PyFunction_FastCallKeywords () from /usr/lib/libpython3.7m.so.1.0
#14 0x00007ffff7bbbd3d in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.7m.so.1.0
#15 0x00007ffff7b00ee9 in _PyEval_EvalCodeWithName () from /usr/lib/libpython3.7m.so.1.0
#16 0x00007ffff7b01de4 in PyEval_EvalCodeEx () from /usr/lib/libpython3.7m.so.1.0
#17 0x00007ffff7b01e0c in PyEval_EvalCode () from /usr/lib/libpython3.7m.so.1.0
#18 0x00007ffff7c2d8b4 in ?? () from /usr/lib/libpython3.7m.so.1.0
#19 0x00007ffff7c2f4be in PyRun_FileExFlags () from /usr/lib/libpython3.7m.so.1.0
#20 0x00007ffff7c30745 in PyRun_SimpleFileExFlags () from /usr/lib/libpython3.7m.so.1.0
#21 0x00007ffff7c32337 in ?? () from /usr/lib/libpython3.7m.so.1.0
#22 0x00007ffff7c32810 in _Py_UnixMain () from /usr/lib/libpython3.7m.so.1.0
#23 0x00007ffff7d90223 in __libc_start_main () from /usr/lib/libc.so.6
#24 0x000055555555505e in _start ()

[traverseda]

Thanks for the help.

I've got libfive recompiled with the build_type=Debug flag set (I think). I just edited my distributions pkgbuild to include the -DCMAKE_BUILD_TYPE=Debug argument.

I've added prints around line 103 of kernel.py, where the Shape(libfive.libfive_tree_remap(self.ptr, shape_x.ptr, shape_y.ptr, shape_z.ptr)) was called, and it does indeed segfault on that call.

I'm not sure what gdb list/gdb print does. When I gdb list I get 1 /build/gcc/src/gcc/libgcc/libgcc2.c: No such file or directory.

Loaded <CDLL 'libfive.so', handle 5593935e0230 at 0x7f65237ced68>
pre tree_remap
<kernel.Shape object at 0x7f652389c128> <kernel.Shape object at 0x7f652382d9b0> <kernel.Shape object at 0x7f652382da20> <kernel.Shape object at 0x7f6523906f98>
Segmentation fault (core dumped)
#This is where post tree_remap would be

[traverseda]

self.ptr=  0x55cd9f72e5a0  shape_x.ptr =  0x55cd9f72e6b0  shape_y.ptr =  0x55cd9f72e6d0  shape_z.ptr = 0x55cd9f72e6f0

No nulls there, I'll see about getting it actually building as debug

[ghost]

@traverseda Re: segfault, let's move this to an issue on https://gitlab.com/rcmz0/libfivepy if you are still having trouble. I deleted my posts because I think the debugging session is cluttering up the original issue here. Thanks. (Github really needs threading in issues, like we expect even in plain old email.)

[ghost]

I used PyInventor to view the sample model from @rcmz pylibfive project:

https://gitlab.com/rcmz0/libfivepy/issues/1

I'm looking converting the PyOpenGL GLUT shader example into a simple viewer like Inspekt3D.

[rcmz]

There seems to be a pretty big issue with remap in libfivepy: https://gitlab.com/rcmz0/libfivepy/issues/2

[rcmz]

Ok it has been fixed 🎉
The issue was with the sphere function:

It was: x**2 + y**2 + z**2 - (diameter/2)**2
And it should be : sqrt(x**2 + y**2 + z**2) - diameter/2

I thought those two expressions were equivalent but apparently no !
Do you know some place where I could read about the maths of functional representation to be able to understand it ?

[ghost]

Here are two links that might help you with F-rep:

https://christopherolah.wordpress.com/2011/11/06/manipulation-of-implicit-functions-with-an-eye-on-cad/

http://iquilezles.org/www/articles/distfunctions/distfunctions.htm

[rcmz]

If you open an libfivepy generated stl file with fstl and set it to auto-reload; every time you regenerate the stl (with libfivepy) fstl say it is corrupted. Which is strange because it display the stl fine and it doesn't occur when you load the file manually.

This is probably a libfivepy issue but here is a post anyway.

Link to the thread on libfivepy: https://gitlab.com/rcmz0/libfivepy/issues/4

[bradodarb]

Sorry for posting on such an old thread, but it seems like sorting out python arithmetic ops vs lib5 ops is a perfect use case for single-dispatch function

[SemMulder]

Just posting a message here to let you know I started new Python bindings here: https://github.com/SemMulder/libfive-python.

The main difference compared to libfivepy is that I'm using Cython instead of ctypes, this should be more performant and robust. And would allow us to start linking libfive statically in the future, that way users can just pip install libfive without making sure they have libfive installed first.

Right now it only exposes the raw functionality of the libfive C API, but if you are familiar with F-rep that should be enough to get going! Let me know what you think!

[traverseda]

Python bindings are now part of core, although they're not documented yet. Closing this issue, that's some very exciting stuff!