Using CasADi from Julia

[jaeandersson]

Julia is steaming ahead and becoming an inceasingly attractive alternative for scientific computing and we should ponder how to support it. There is currently no good way of calling (realistic) c++ from Julia, but there is advanced support for calling Python. So an approach which might be easy to get working is simply to call the Python bindings.

cf. http://conference.scipy.org/scipy2013/presentation_detail.php?id=203

[jaeandersson]

It turned out CasADi/Python can already be used from Julia, though the syntax is maybe not ideal:

@pyimport casadi as c
# x = c.MX.sym("x",2,2) # not (yet) working in current Julia
MX = c.pymember("MX") # workaround 1
x = MX[:sym]("x",2,2) # workaround 2
c.gradient(c.det(x),x)
# PyObject MX((det(x)*inv(x)'))

Pretty impressive.

They are apparently working on adding support for overloading ".".

[ghorn]

that is extremely impressive

[jaeandersson]

Ping @toivoh.

[jaeandersson]

This appears to be the Julia issue for allowing ".":
JuliaLang/julia#1974

[mlubin]

👍

[toivoh]

It would definitely be nice to be able to access CasADi from Julia, and I agree that using PyCall.jl seems like the best near-term solution. Overloading of the . operator has been discussed for a long time and it's very hard to predict when it might come about.

It might be worthwhile to create a shallow Julia wrapper based on PyCall in the meantime; if there's mostly a few things that are awkward to write with straight PyCall syntax, convenience functions could be provided for them. The MX.sym syntax will however not work until the . operator can be overloaded, so one would have to find a more julian way to call that convenience function (go back to msym? mxsym?).

[jaeandersson]

It might be worthwhile to create a shallow Julia wrapper based on PyCall in the meantime; if there's mostly a few things that are awkward to write with straight PyCall syntax, convenience functions could be provided for them. The MX.sym syntax will however not work until the . operator can be overloaded, so one would have to find a more julian way to call that convenience function (go back to msym? mxsym?)

I think a reasonable convention would be to replace '.' with '_'. That is 'MX_sym' instead of 'MX.sym', 'Sparsity_diag' instead of 'Sparsity.diag' etc.

[jaeandersson]

@toivoh Do you really think that it will take so long before '.' becomes available? It is in Milestone 0.4 due 1 October.

[toivoh]

Oh, I didn't know that. All I know is that there's been a lot of talk back
and forth but nothing has happened yet, but maybe we will have it this year
then. Julia releases tend to come when they are ready though, not at
pre-decided dates.

I agree that replacing . with _ seems like a reasonable convention.

[toivoh]

Another idea might be to create a wrapper in the direction of one of the AD packages in Julia, but I don't know at all if that would make sense. @mlubin: Any thoughts?

[mlubin]

There's currently no unified interface to AD in Julia, and I don't expect one for some time until the various packages mature. I would suggest first creating a wrapper that's natural for CasADi.

@jaeandersson The 0.3 milestone is three months past due, so I wouldn't realistically expect 0.4 to be released by October. That said, it's possible that the '.' syntax will be available in development builds shortly after the 0.3 release.

[mlubin]

There's now an experimental C++ interface from julia: https://github.com/Keno/Cxx.jl. See this thread: https://groups.google.com/forum/#!searchin/julia-dev/cxx/julia-dev/4kUu2ieWZ9Q/HgJqQFSQbeMJ (and this pretty cool demo: http://www.youtube.com/watch?v=Wkw2gmzfm1Q). It's still pre-alpha quality, but I've played around with it with some pretty impressive results. This is likely the best way forward once it stabilizes a bit.

[jaeandersson]

Interesting! For CasADi, anything not SWIG-based would be a showstopper. We use SWIG for Python, Haskell and MATLAB (cf. #176), so we cannot maintain a different approach for Julia.

I wonder however it this could be used as a basis for a Julia module for SWIG, without any C gateway. @ghorn - your Haskell stuff doesn't have any C gateway, right? Do you think something similar could be employed for Julia?

[ghorn]

Haskell only supports the C calling convention, not C++, so I generate C++ code where every function is extern "C"'d. This means no passing classes, structs or templates - always pointers to them or types which work in C. This also means handling memory management using Haskell's ForeignPtr external memory management.

[jaeandersson]

Ok, then that doesn't really help us further.

[mlubin]

Well if there's already a C API, it would be pretty straightforward to call it from Julia (moreso than calling C++).

[jaeandersson]

Sure. The way SWIG typically works is that you generate a C API to the C++ code and then generate proxy classes that calls the C API from the target language. That's the way SWIG's Python module works and the Matlab module (not yet part of mainline SWIG). The Octave module works differently, but that's the exception.

I see no reason why you wouldn't be able to write a Julia module for SWIG the same way. I guess the key question is how Julia's object-oriented programming works, if there is an intuitive mapping from C++ to Julia. But I can't see any show stopper. On the other hand, my Julia knowledge is very shallow.

One really nice thing with SWIG is that it deals with a lot of issues before it gets to the code generation step. Things like templates, polymorphism etc. are handled before you get to the language-specific generator code. There are also support for changing the API of certain functions or classes. For example renaming them (e.g. avoiding a keyword in the target language) or making functions return-by-value instead of return-by-reference. For example, the prototype for QR factoring a matrix in CasADi is:

template<typename DataType>
void qr(const Matrix<DataType>& A, Matrix<DataType>& Q, Matrix<DataType>& R);

But the syntax, calling it from Python or MATLAB is:

[Q,R] = qr(A)

So, to reiterate, what I would really recommend would be write a Julia module for SWIG. Like the one I have been working on for MATLAB. Given the similarity between MATLAB and Julia, one might based the module on the MATLAB module.

[jaeandersson]

After the refactoring of CasADi performed to accommodate MATLAB, the lack of the "." overloading is now much less of an issue. Don't think there are any fundamental showstoppers hindering extending SWIG to Julia and then use this extension to generate CasADi bindings.
I won't take the lead on this, but if someone feels motivated, I would be happy to provide directions. The amount of work corresponds roughly to a (quite interesting) master thesis.

[mlubin]

I recognize the convenience of generating all interfaces through SWIG, but I'm not sure if a SWIG-Julia interface would be well received by the Julia community for philosophical reasons. If someone is willing to dedicate the effort equivalent to master's thesis, I'd personally say that there are more effective uses of that time, e.g., a reimplementation of CasADi in Julia or investigation of AD based on source-code transformation (e.g. https://github.com/JuliaDiff/ReverseDiffSource.jl).
At the same time, I think we're overestimating the effort it would take to wrap CasADi. Ignoring the maintenance issue, I can't imagine that it would take more than a few days of work to get a basic usable wrapper through PyCall or Cxx.jl. Maintenance of the interface could be community driven.

[jaeandersson]

"would be well received by the Julia community for philosophical reasons" - what do you mean by this?

[mlubin]

Well... pragmatically people might question why it's worth putting the effort into developing a SWIG-Julia wrapper when Julia already has very good C and Python FFIs which don't require running a wrapper generator. The C++ FFI interface is still under development at https://github.com/Keno/Cxx.jl, and contributions here would be more than welcome. At a technical level Cxx.jl would be a much more interesting Master's thesis than "just another" SWIG frontend because it makes extensive use of LLVM to instantiate C++ templates at runtime, on demand, in a way that I've never seen done before. The result will be a much more lightweight and transparent interface between Julia and C++ with no intermediate wrappers needed.
From the perspective of existing projects with SWIG wrappers, I completely understand the motivations for wanting a SWIG-Julia interface. By "not well received by the Julia community" I don't mean that there would be opposition in any sense other than people asking "why would you want to do that?". I think it's fair to say that it would be viewed as a legacy interface and not the recommended way to call C or C++ from Julia. Admittedly this does leave a gap in functionality for authors of C/C++ projects who want to write a single interface specification and have wrappers generated for multiple languages.

[jaeandersson]

OK. Personally, I think that it will be very difficult for Julia C++ to ever catch up with SWIG in terms of functionality. SWIG is not just a proof-of-concept, it essentially maps the whole C++ language to a range of target languages. (Re)implementing SWIG is essentially implementing a C++ compiler, including standard libraries and mapping a large set of features. When I've been working on extending SWIG (to MATLAB), I've just been working on the code generator backend, getting all those other features for free. I agree with you that it would be nice to avoid explicitly instantiating all template classes, but it's not really a huge issue. Maybe it could also be solved in SWIG by adding some JIT capabilities. In SWIG, available for all target languages, not just for Julia.
Anyway, all this is besides the point. What this issue is about is making CasADi available to Julia users, in whatever way is more convenient. I think that the by far most convenient way is to extend SWIG.

[jgillis]

Note that the newest matlab allows you to interact with python

[jaeandersson]

Note that the newest matlab allows you to interact with python

How is this related to the issue... ?

[jgillis]

The relation between our python interface and julia is now the same as our python interface and matlab.

[jaeandersson]

Yes. But using Matlab's ability to interact with Python makes about as much sense as using Julia's ability to interact with Python. In fact, a lot less, since we now have a working Matlab interface.

[jaeandersson]

The Julia interface has gotten more interesting now thanks to the refactoring of CasADi that aligns it with MATLAB's multiple dispatch mechanism.

The Julia syntax would be similar to MATLAB, with a major difference being the treatment of the static "factory" methods.

E.g. instead of:

x = SX.sym('x');
z = 4*x + SX.eye(10);
F = SXFunction('F', {x},{z, sin(z)}); % Could change to SX.Function(...)

the Julia syntax (if I've understood things correctly) could look something like:

x = sym(SX, "x")
z = 4*x + eye(SX, 10)
F = SXFunction("F", (x),(z, sin(z))) # Could change to Function(SX, ...)

Cf. https://groups.google.com/forum/#!topic/julia-users/SdWfq9YSM30

[haianos]

Dear all,

I am feel like reviving this very interesting discussion, and provide my honest opinion in favor of an FFI approach despite the SWIG approach.
Resuming the advantages:

• much smaller overhead, performance close to original C/C++ implementation
• the required work is minimal wrt SWIG
• most of the modern languages provide a FFI-ish approach
• portability
  Disadvantage:
• it requires a major/novel development effort to import those changes

Regarding some previous comments

The problem is not that it won't work - the problem is that it isn't a pragmatic approach. It would be extremely difficult to motivate people to learn a new interface system and customize the approach so that it works only for Julia. We now have MATLAB and Python being generated for CasADi, and we also use SWIG to interface with JAVA and Haskell (hey, that's a statically typed language!).

I disagree. It is definitely a pragmatic approach.
Python has a (kind of) FFI inspired by LuaJIT FFI, Haskell has native FFI, Julia as FFI (even ongoing work against C++, not only C)..also MATLAB has its FFI (and it is not Mex....but I am not familiar with it to judge the maturity).
SWIG is there because no alternatives were available when SWIG project came out...

Of course, I definitely understand that this would drastically change part of the CasADi approach, with a large impact on the current implementation and workflow. However, I do think that CasADi will benefit a lot from it, especially in the long-term. Priorities may be somewhere else for CasADi, however, due to the previous motivations, I would consider this stepchange in a future, major release - it is always pain to get rid of some legacy code, but I believe it is worth in this case...but yet, I am not a CasADi developer!

[jgillis]

Hi,

Do you have a pointer to Python FFI for C++?
I could only find https://pypi.python.org/pypi/CFFIpp/ which is not so mature..

[haianos]

Sure!
I am aware of few other efforts, but I was referring to ctypes and cffi
https://docs.python.org/2/library/ctypes.html
https://cffi.readthedocs.io/en/latest/
I found some useful hints/examples here
http://eli.thegreenplace.net/2013/03/09/python-ffi-with-ctypes-and-cffi/

Hope this helps!

Update The former are C-only, no C++ ... (I read too quickly before)

[jaeandersson]

I am feel like reviving this very interesting discussion, and provide my honest opinion in favor of an FFI approach despite the SWIG approach.

There must be some sort of misunderstanding here. SWIG is not an alternative to a FFI approach, it's an abstraction layer on top of the target language native FFI approach. The current MATLAB interface uses MEX and a single entry point per SWIG module. Most other SWIG modules (as well as a failed, previous attempt to write a MATLAB module) uses multiple entry points. In MATLAB, this means using "loadlibrary", as I said this failed. Not because the approach didn't work, but because MEX turned out to be more maintainable. But this is a decision that has to be made for each backend.

What SWIG does is that it reduces the problem of interfacing C++ to a problem of interfacing C. For a C++ project, it will (almost always afaik) generate C bindings, compile to a shared library and then have the target language talk to these C bindings via whatever way is more natural and maintainable. For MATLAB, it turned out to be MEX, although loadlibrary was also an option. For Julia it would most likely be ccall.

For the MATLAB module, in addition to the C wrapper, a MATLAB class hierarchy is built up that matches the original C++ hierarchy. Other modules for object-oriented languages (e.g. Python) work the same way. If you want to get an idea of what it looks like for CasADi C++, have a look at https://gist.github.com/jaeandersson/8f3644488c69b9642c8b1a851830b846. This is the autogenerated MATLAB user class (casadi.Function) corresponding to one class in CasADi (casadi::Function). The docstrings are autogenerated from doxygen. For Julia, it would work in a similar way, only instead of classes you would build up Julia types that most closely resembles the C++ classes and the member functions would be defined outside the class (right? I'm not too familiar with Julia syntax).

[jaeandersson]

MATLAB has its FFI (and it is not Mex....but I am not familiar with it to judge the maturity)

This is incorrect. Both loadlibrary (MATLAB's way of calling C functions in shared libraries) which you refer to) and MEX are "FFI", by any reasonable definition. And they are related, MEX is really nothing but a shared library (with an entry point called "mexFunction") and IIRC, you can even mix the two approaches.

[haianos]

No, not really...IMHO, you can't really talk about FFI solutions if you end up to compile something.
That's what SWIG-approach does - it is no-less, no-more that passing through a third-common language to bridge other two.
A "real" FFI solution calls directly the function symbol, no overhead.
It may seems a tiny, technical difference, but It is not...both conceptually and performance-wise.
And yet, MEX is not exactly plan FFI, for the same reason...(isn't it?)
(The misunderstanding may be from the wording - to me, FFI and bindings are two different things).

Of course, the fact that you deal with C++ code (and not C) makes everything more tricky, and somehow you end-up with a manual wrapping that SWIG does for you.

Indeed, the real question is: the effort is worth for CasADi, especially considering you already have something that does the job? Maybe yes, maybe no. For sure, it is not the main priority of the project! :-)

Now, getting back to the Julia topic, the idea of having: C++ -> SWIG -> Python -> SWIG -> Julia sounds more an hack solution (that works!) than a "real" solution to me. Of course, if that is "for free" and it does the job, why not, but I won't consider it as a "final" solution...
This has been already mentioned in this issue topic (see @mlubin ).
I understand the pragmatic reason about don't invest time on something different for the sake of it, but also missed occasions on not doing that because of "the current solution already works", or at least it is how I interpreted the following:

For CasADi, anything not SWIG-based would be a showstopper.

while instead SWIG (as any other dependency) is just a tool that serves a purpose, and It should not impose a limitation...I think...

[jaeandersson]

No, not really...IMHO, you can't really talk about FFI solutions if you end up to compile something.
That's what SWIG-approach does - it is no-less, no-more that passing through a third-common language to bridge other two.

There must be a misunderstanding. SWIG is not a tool that replaces a FFI. It's just a tool for generating bindings. Or rather, automate as much as possible of the process of writing language bindings in a systematic way. Because no bindings could ever be completely automatic, if you want them to feel natural in the target language. There will be no dependency on SWIG in the generated code. The generated code is just plain C/C++ as well as (usually) code in the target language. Code that can be distributed either in source form or compiled form.

the idea of having: C++ -> SWIG -> Python -> SWIG -> Julia sounds more an hack solution (that works!) than a "real" solution to me

No one is proposing that though. The proposal is to have C++ -> C (auto-generated by SWIG) -> Julia (auto-generated by SWIG). Where the communication is done using ccall.

[asprionj]

Any news on this topic? Could Cxx.jl be an option / shortcut?

[jgillis]

Planned for 2019..

…

On Sun, Dec 30, 2018, 21:09 asprionj ***@***.*** wrote: Any news on this topic? Could Cxx.jl <https://github.com/Keno/Cxx.jl> be an option / shortcut? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUFSKXxkDAwNxMfZkXOWekbFl-mu4zLks5u-R1rgaJpZM4B3wvB> .

[asprionj]

Great! Already a fixed idea how to approach it? If PyCall.jl by now allows attribute/method access by point operator, this could already be an almost perfect solution... don't know that, though. Otherwise, I'd be interested in helping; planning on getting more into Julia in 2019 but want to keep CasADi...

[jaeandersson]

Had a quick look at the dot operator overloading in Julia. Looks very similar to MATLAB's approach so I'm pretty sure that you can use a similar design. A Julia back-end to SWIG would be really great and benefit not only CasADi but also a lot of other projects.

[asprionj]

I just saw that Cxx.jl has quite some issues getting consistent Julia 1.0 support. You think SWIG is the way to go then? There was no response at all to your post in the SWIG mailing list back in 2014 -- maybe Julia's gain on popularity changed things in the meantime. Where to ask whether anyone is interested in joining the development of a Julia module for SWIG?

[jaeandersson]

I do think SWIG is the way to go. Unless there is a better alternative available, which I doubt. I think the main issue is a desire in the Julia community to develop everything from scratch - in the case of C++ it doesn't really make much sense IMO. An approach like SWIG which allows you to simultaneously support multiple target languages, including Julia, is more useful than an approach that works with Julia alone.
In terms of the coding, I think that it's much easier than most people think. I saw that with the MATLAB module. It really just took a couple of weeks for me to get a working module, the one we use for MATLAB/Octave bindings. Since Julia's object oriented programming paradigm is so similar to MATLAB's, I assume that the situation would be very similar: A couple of weeks (e.g. a good master student) to get a fully functional Julia back-end for SWIG.
Once there is a SWIG module in place, I don't think you need much work to get CasADi interfaced. We had to refactor CasADi's interface a lot in order to get MATLAB interfaced in an intuitive way. This included a lot of inline friend functions in the C++ API which are nice for math (regardless of language). Since MATLAB member functions are quite similar to C++ friend functions, you get a natural mapping. For a Julia interface, all of this is already in place.

[jaeandersson]

Oh, and @asprionj: Did you try using CasADi-Python from Julia as-is? Given Julia's pretty seamless interaction with Python, I'd think that would work pretty well now. IIRC, I tried it a few years ago and then it worked fine out of the box, but the syntax was horrible without dot operator overloading.

[asprionj]

Haven't tried Julia -> PyCall -> CasADi. As I said, if dot operator is used in PyCall for attribute/method calls by now, this would be a more than acceptable workaround. If not, it still is a viable option that's already available.

No idea how urgent the desire for disposing the Python detour / dependency is for the common Julia user. How easily and overhead-free can code-generated CasADi problems be interfaced from Julia (when compiled to a shared library)? If that works, I guess most needs are already fulfilled.

I'll be back home tomorrow; can try some stuff then.

[jaeandersson]

How easily and overhead-free can code-generated CasADi problems be interfaced from Julia (when compiled to a shared library)?

There is a separate issue for that: #1730. I don't think you'd see hardly any overhead. #1730 is a pretty simple extension, maybe one day or so to implement.

[asprionj]

Just to let everyone know, the PyCall-workaround still works fine (Julia 1.0.3, CasADi 3.4.5). Basically, you'd do:

@pyimport casadi as c
x = c.MX[:sym]("x")
c.jacobian(c.sin(2*x) + 3, x)

Which will return PyObject MX((2.*cos((2.*x)))). You can wrap frequently used stuff such as MX:

MX = pywrap(c.MX)
x = MX.sym("x")

Also, by specifying a method for built-in functions on argument type PyObject, and by assigning functions to Julia variables, you can get rid of the c.:

import Base.sin
sin(x::PyObject) = c.sin(x)
jacobian = c.jacobian
jacobian(sin(2*x) + 3, x)

This will produce the same output as the original version. Of course, you have to manually make sure that there's no conflicting variables etc. Also, if using other Python stuff via PyCall, overloading for type PyObject is dangerous...

[ChrisRackauckas]

I wouldn't recommend a macro here. Instead, it might be a good idea to just namespace all of the functions you want into some Casadi submodule, so Casadi.sin (not an extension of Base.sin, but a new one which has things for PyObject defined), etc. so that the working code is:

Casadi.jacobian(Casadi.sin(2*x) + 3, x)

Then make a macro that does a find/replace on these known functions to put the namespace on them automatically, i.e.

@casadi jacobian(sin(2*x)+3,x)

would then be safe and clean. The other thing you could do is wrap the PyObjects in a small struct and dispatch on that.

[kdheepak]

The dot operator in PyCall has also been overloaded in this commit, so that should make it a lot nicer to use going forward.

That said, I'm also very interested in having a SWIG module for Julia working. It would be really nice to have just a few interface files and have bindings for Python, MATLAB, R, Java AND Julia all be generated programmatically from those interface files; instead of having to maintain separate bindings just for Julia alone. I've started to hack on this here to see what would be required to get it going. It does seem like someone familiar with swig modules and Julia's C API can get it up and running fairly quickly.

[jaeandersson]

I've started to hack on this

Fantastic, @kdheepak! With a concentrated effort it should be doable in couple of weeks. And it could have a major impact, given how popular Julia has become. Keep us updated.

[asprionj]

Just learned that PyCall 1.90 was released recently:

In preparation for PyCall 2.0, this deprecates the old o[:foo] access in favor of o.foo.

That is, one could do, for the example used earlier:

c = pyimport("casadi") # (@pyimport is deprecated)

x = c.MX.sym("x")
c.jacobian((2*x).sin() + 3, x)

[jgillis]

Miles, Jul '19: "We have no current plans to add support for vector-valued expressions to JuMP. I’d recommend supporting the effort to get casadi running in Julia"
Perhaps I should finally do it..

[frapac]

A Julia interface for Casadi would be a game changer to me. Maybe I could help on that.

On another extent, the JuMP development team is planning for a deep refactoring of JuMP's NLP interface. I think it could be interesting to have a discussion with the Casadi team about your own design!

[jgillis]

As I see it, SWIG is just a very handy tool to generate artifacts from C++ headers.
The output will be a bunch of Julia datatypes and methods.

The implementation of those methods could be with ccall to a C wrapper or Cxx.jl
While the latter looks more direct, I'm not convinced that it is desirable.
You'd need to guarantee binary C++ compatibility of CasADi binaries with Julia's JIT compiler.

In the 10 years we've been using CasADi I've never seen wrapper code be a computational bottleneck for optimization use case.

[raphaelchinchilla]

Has there been any progress on porting CasADi to Julia?

[nxtruong]

There is a Julia interface to CasADi here. It uses PyCall for accessing CasADi, so basically one must have Python and CasADi installed in Python, then use this unregistered package to conveniently access certain features of CasADi from Julia. It's not ideal, as it doesn't directly interface with CasADi, but I have found it sufficient for my work.