Curious about making Plots.jl a conditional dependency?

[dehann]

As a first post -- thanks for putting this package out there! Would it be useful to the community if the Plots.jl dependency and surrounding code in Manopt.jl were conditional?

E.g.:

using Plots
using Manopt

# continue as before

Which allows other cases:

using Manopt

# Plots.jl doesn't even need to be installed

Requires.jl does the job pretty well:
https://github.com/JuliaPackaging/Requires.jl

I'm looking to integrate Manopt.jl as a dependency (to replace similar features) in ApproxManifoldProducts.jl (AMP) which extends the idea to particle sets and combines it with a Bayesian belief product construct:
https://github.com/JuliaRobotics/ApproxManifoldProducts.jl

FYI, a lot of work will be coming to AMP in the future and Manopt.jl and Manifolds.jl obviously worth it for greater community effort. I will add though that performance and thread safety will be major drivers for that push (and may well include a few other wide ranging packages).

PS, this might be a useful mechanism to prepare for other plotting backends including:
https://github.com/JuliaPlots/Makie.jl
https://github.com/JuliaPlots/WGLMakie.jl
https://github.com/rdeits/MeshCat.jl

[kellertuer]

Cool, that you are interested in Manifolds and thanks for pointing that out. I was quite new when I started the dependencies here, so yes that should be changed to abstract plotting. But for now there is also no proper plotting yet. I will check an d remove that.

For your dependency, would it be better to depend on ManifoldsBase.jl (https://github.com/JuliaManifolds/ManifoldsBase.jl) instead? What do you need for your package feature wise?

[dehann]

Great thanks, the integrated plotting is nice. We have done some work on the JuliaRobotics/Caesar.jl side to make plotting "separate" from functional code -- so however best to keep that concept going. We ended up with RoME.jl and RoMEPlotting.jl as example, but that was before Requires.jl became mainstream.

I will have to look at the breakdown to ManifoldsBase.jl to be sure -- thanks for pointing that out.

Robotic navigation / sensor fusion use cases would be a major user of manifolds (already is). The current solver and interface code is bringing various concepts together, manifolds being one of them. Currently we decided to focus on 2D in order to get the design right but then rapidly expand to 3D and other strange non-standard manifolds of own design. The biggest cases right now are SE(2) and E(3) + Euler angles (but want to replace that with Quaternion or SE(3) or SP(3/4?) once other major design architectures are sufficiently resolved). The system have been functional for a while, but now we taking it to the next level, so to speak. Also, while SE(3) represents positions and orientations, we would need to expand so that velocities are included in state representations too.

We are currently using a blend of CoordinateTransformations.jl and dehann/TransformUtils.jl (bad name ... to be updated); and then as stopgap doing things like Pose2 (as in robot position in time) which has manifolds (:Euclid, :Euclid, :Circular). This is obviously a terrible use of dispatch, so just moving to the formal definitions provided by JuliaManifolds will already be a huge boost (we've wanted to do that for a while). As example, we have run into some problems with RoME.Point2 = (:Euclid, :Euclid) being overloaded in GeometryTypes.jl.

I guess the major features we'd look for is firstly design within the existing Julia ecosystem, how is JuliaManifolds going to interact with Optim.jl, NLsolve.jl, CoordinateTransformations.jl, GeometryTypes.jl, etc.. Example here is something cringe worthy on what we currently doing for SE(2) and S(1) -- Manopt.jl is way way cleaner:
https://github.com/JuliaRobotics/ApproxManifoldProducts.jl/blob/bc0464eeafe6ff6c19e32fbf380d2e98407fd326/src/Circular.jl#L27-L33

and
https://github.com/JuliaRobotics/ApproxManifoldProducts.jl/blob/bc0464eeafe6ff6c19e32fbf380d2e98407fd326/src/CommonUtils.jl#L4-L9

These examples above are where we run gradient on a sphere for a slew of applications further downstream.

Thats a half answer, but maybe helps scope our steps as we start this process -- and my expectation is it will take a few months to complete. So changes will be high level abstract, but simple. The hope is that Manifolds become well standardized through the ecosystem like Colors.jl or Arrays are today.

I guess the major quirk which might influence you guys more early is the ability for users to define their own weird manifolds. And documentation around that please... The best example i have is something called inertial sensor preintegration (inertial odometry). Take a gyroscope and accelerometer triad and integrate the outputs for a few seconds -- so that gravity and sensor calibration errors are included into that integration process. Now retroactively try remove the gravity and biases earth rotation etc from the integrand, using assumption of slow changing gravity, biases, etc. To do this we numerically follow the submanifolds on which to subtract these "errors" from the integrands: This can be developed via differential equations using a multidimensional Taylor expansion of around at least 15-17 dimensions (positions, velocities, orientations, biases, etc.) with Taylor terms as high as 4th order. We know how to write this down as a operation that looks like a manifold exponential map (approx since no closed form expression has yet been found). So finally the punchline is, how can we use Manifolds.jl in a sensible way to combine and glue together our own definitions so that some exotic user "manifold" can be created and then used in the ecosystem without resorting to bad hacks like RoME.Pose2=(:Euclid, :Euclid, :Circular) as we are currently doing.

[dehann]

xref JuliaRobotics/ApproxManifoldProducts.jl#32

[kellertuer]

Thanks for the explanation, that looks very promising for manifolds!

Manifolds gets you covered for the manifolds you mentioned, for example directly, see https://juliamanifolds.github.io/Manifolds.jl/latest/manifolds/group.html#Special-Euclidean-group-1, or for other cases using the Product (or even Power) Manifold. I think what you mention in Pose2 would be ProductManifold(Euclidean(2),Circle()).

The changes for optimization are quite some. There is first steps in Optim.jl towards using Manifolds (not yet with ManifoldsBase.jl but from before that), but algorithms get a little more involved on manifolds; one of the next ones Manopt.jl will cover is L-RBFGS. The gradient for example is not that different (but just uses a few retractions in Optim currently), but also available here – and you can use exp/log or any retraction you like – see https://juliamanifolds.github.io/Manifolds.jl/latest/interface.html#ManifoldsBase.retract-Tuple{Manifold,Any,Any} on how we cover that.

To implement your own manifold, you just have to cover a few functions (all required for your application) from https://juliamanifolds.github.io/Manifolds.jl/latest/interface.html#Types-and-functions-1 and you can use all algorithms here. We could discuss that for your example either in an issue over at manifolds and provide that manifold to all other users of Manifolds.jl, too, or on your package. But I would love to see more manifolds coming into Manifolds.jl. So if we can help there, feel free to reach out.

[mateuszbaran]

That's a really cool application of manifolds 👍 . I could add a few things to what Ronny already wrote.

FYI, a lot of work will be coming to AMP in the future and Manopt.jl and Manifolds.jl obviously worth it for greater community effort. I will add though that performance and thread safety will be major drivers for that push (and may well include a few other wide ranging packages).

We are already taking performance seriously. For example I'm working on being able to use methods from StaticArrays wherever possible. I didn't think about thread safety yet but we can work on that together.

This can be developed via differential equations using a multidimensional Taylor expansion of around at least 15-17 dimensions (positions, velocities, orientations, biases, etc.) with Taylor terms as high as 4th order. We know how to write this down as a operation that looks like a manifold exponential map (approx since no closed form expression has yet been found). So finally the punchline is, how can we use Manifolds.jl in a sensible way to combine and glue together our own definitions so that some exotic user "manifold" can be created and then used in the ecosystem without resorting to bad hacks like RoME.Pose2=(:Euclid, :Euclid, :Circular) as we are currently doing.

There was some effort on exponential integrators in the DifferentialEquations ecosystem which I was thinking about integrating with our manifolds: JuliaManifolds/Manifolds.jl#128 but there is not much done yet. These exponential integrators could in principle use any of our retractions (approximate exponential maps) and you can define many different retractions on the same manifold and then use these retractions via multiple dispatch -- not only for integration but also for optimization, vector transport etc.

[dehann]

Thanks, that helps guide things this side too!

[kellertuer]

I just checked and there is only an example using plots but no internal functions for plotting yet, so I resolved this.

Still – if you want to discuss more on manifolds or manopt – feel free to stay in contact.

[dehann]

Will do and thanks!