Sort out what version of numpy (and other packages) to require for TuLiP

[murrayrm]

As part of commit ce7d868, we bumped the requirement for NumPy up to 1.7+. Some stable distributions of linux (and Mac OS X?) use NumPy 1.5. We should decide what we want to support. Opening this issue to create a place for discussion.

[johnyf]

A suggestion is to bring back adj_update and within it try using np.pad and return. If this fails, just pass and use the old adj_update code. In the future after numpy versions progress and linux distributions update to post 1.7, we can remove adj_update.

20f3318 was another issue related to numpy @1.8, but it concerned indexing a 1d array and shouldn't cause any problems with older versions.

[necozay]

I think recent MAC OS X comes with numpy 1.6.

One example of how we support compatibility with early versions of certain
packages in TuLiP can be seen at _block_diag2 function in
discretization.py.

For functions created for backward compatibility, I prefer names closer to
the name used in the future version of the package or mentioning the
corresponding function in the docstrings.

On Mon, Dec 30, 2013 at 2:56 PM, Ioannis Filippidis <
notifications@github.com> wrote:

A suggestion is to bring back adj_update and within it try using np.padand
return. If this fails, just pass and use the old adj_update code. In the
future after numpy versions progress and linux distributions update to
post 1.7, we can remove adj_update.

20f331820f331806be50b82506af67d6315955d82d0cbbewas another issue related to numpy
@1.8, but it concerned indexing a 1d array and shouldn't cause any
problems with older versions.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/48#issuecomment-31364782
.

[johnyf]

After introducing support for PLY, PyParsing code exists but is slow, so probably we need to consider replacing the dependency on PyParsing with PLY.

[johnyf]

7b4243b and acb3f8e replace PyParsing with PLY. Minimal PLY version set to 3.4, though it may also work with earlier versions. This version is present in debian wheezy.

[necozay]

What is the minimum gr1c version compatible with the master branch?

[johnyf]

I am using gr1c @0.6.6. This is actually a very good point: we need to check the minimal versions of external solvers (now gr1c, in the future more). A check should probably be added to interfaces.gr1c, because the interface should be responsible for this, not tulip - which should in principle be agnostic of what lies in the path. There are two reasons for this:

• Users interested in something independent, e.g., subclass spec.LTL for their fancy new logic, or use transys because they want to play around with automata should not need to install everything, especially things that don't install automatically and involve compilation.
• Checking at install time is not sufficient. One may upgrade or downgrade arbitrarily gr1c, in which case the interface will break (we have already observed this a couple of times with users who needed to update their gr1c installation).

[slivingston]

The minimum version of gr1c to target for TuLiP v1.1a is 0.7.3. There is still an outstanding bug (issue tulip-control/gr1c#13), but it concerns an error that occurs when an obviously flawed input is given. If I am able to fix this before release of TuLiP 1.1a, then we can bump the minimum required gr1c version number.

[slivingston]

The minimum version of gr1c for TuLiP is now 0.7.4.

[johnyf]

cvxopt is not a tulip dependency any more, because only polytope depends on it, so it has become a polytope dependency instead. For this reason it will be removed from setup.py.

Note that cvxopt QP is still used in tulip.abstract.find_controller. However that is not essential tulip functionality yet, for 2 reasons:

1. simulation facilities are not substantially developed within tulip
2. even when developed, this is a Receding Horizon Control module, which may as well be moved to python-control.

Note the first divide-and-conquer benefit of modularizing by extracting polytope from tulip: separation of concerns. Users can now install tulip for discrete synthesis only, if they are not interested in abstraction of dynamics, i.e., if they wish to only use it as a python interface to gr1c or jtlv, since standalone pygr1c or pyjtlv packages don't exist.

For this reason I suggest that polytope become an optional dependency of tulip. This will allow new users, even if they are interested in dynamics abstraction, to install a pure-python tulip, with no non-standard compilation needed besides numpy and scipy, which will always be well supported and compile w/o problems when installed using some standard distribution mechanism on any major platform (so we don't have to worry about them).

This separates the concerns for LP and QP, because currently polytope requires an LP interface only, whereas Receding Horizon Control requires the QP interface. Therefore we shift user installation headaches to polytope, where I plan to replace cvxopt (make it optional) with one of the multiple other packages that are available to interface to GLPK. This shouldn't be a significant change.

Then it only remains to either let tulip depend optionally to cvxopt for RHC, or replace also that with some python binding for a QP solver. In any case this essentially liberates us from cvxopt.

[necozay]

I think this type of separation requires a bit more discussion. This might turn tulip into a lightweight wrapper and a collection of data structures with no content that has research value (which might be fine if this is what we want tulip to be). Couple of arguments/questions to consider:

• What is the "core" tulip functionality we are aiming at?
• If I want to only do discrete synthesis, why should I use tulip? I can just get gr1c or jtlv. Pure-python tulip could be useful for python users interested in discrete synthesis but from my observations so far, I can tell that many new tulip users are not primarily python users (they use python for tulip - which would not be required if they only care about discrete synthesis).
• If I am developing abstraction algorithms, is tulip the right home for them or not?
• If abstraction is to be a major part of tulip, some abstraction algorithms require convex optimization. Also, observer design might require convex optimization. I am not particularly satisfied with cvxopt and GLPK but I think for many interesting problems at the intersection of continuous and discrete states, a good optimization interface is important. If optimization will definitely be required in the future, maybe we can just leave it as is (is the idea to take it off now and put it back in when the relevant functionality is better developed?).
• I don't think abstraction of continuous dynamics is particularly useful if we do not provide a means to implement the discrete controller in continuous level.
• LP-QP distinction is kind of superficial in that in implementing the discrete-controller feasibility is important for correctness and the choice of the cost function is kind of arbitrary (with quadratic costs being traditionally better fits for certain tasks).

[johnyf]

Before answering, just a note that still polytope is required, therefore whatever dependency it has on optimization solvers is indirectly required, but it becomes clear by locating the dependencies in each package what exactly uses linear programming. This comment conflicts a bit with some comments below, unless polytope is turned into an optional dependency as suggested above.

The motivation is practical, namely to get as many users as possible by lowering the threshold for them to install and use the tool. Those both familiar enough with the subject and/or convinced that they need to use our tool (in other words quite familiar already with the field) will invest in it. others who come across it and might be interested in doing some discrete synthesis for their particular application, which may not involve dynamics, will have to go through the cvxopt installation difficulties and may be led to quit in the process. Even the experts may find it more friendly to get a basic installation up and running at no time, then worry about getting everything working.

One viewpoint would be that data structures also have research value, besides their educational value. Many of our data structures are classes with methods, so they include algorithms packaged with them.

Strictly for gr1c and jtlv the functionality tulip provides is to a large extent a python wrapper, that provides an environment that makes using them friendlier. This refers to the interfaces, spec and transys sub-packages and the synth module.

I definitely agree with several of the comments above, but I think that requiring dependencies for things that will be supported in the near (or more distant) future can be delayed until those dependencies are in the code. Addressing specific points:

• This assumes that the core functionality is defined by what exists in the HEAD of master branch. So wrapping game synthesis as described above and abstraction algorithms (simulation in the near future, but currently not well supported).

• If a user is interested in discrete synthesis and they directly use gr1c or jtlv, then they will just have to either:

  • duplicate all our effort in writing some (miniature or bigger versions) of spec, transys, synth, which can take quite some of their time, or
  • produce the input to the solvers and parse the results manually for each specific problem they use the solvers for.

  If tulip is easy to install, considering that python is such a friendly language with increasing popularity in the scientific world, why not install it and use python. Our advantage minimally is having more users testing part of the code and reporting bugs (Linus's Law).

• Yes, tulip can be the right place for abstraction algorithms if they fit in its architecture and the core development team agrees about including those particular algorithms. The algorithms might be implemented in python or wrap some lower-level implementation that lives in either tulip's repository or need be installed separately. This actually hints to how the issue arises in the first place. There can be several algorithms and approaches shipped with tulip, but we can minimize the non-pure-python dependencies required for an installation, those that need compilation.

• The current abstraction algorithms depend on convex optimization only via geometric operations implemented in polytope, so polytope requires them present, but tulip does not, instead it requires polytope to be installed.

• As commented above, I suggest introducing dependencies together with functionality, not ahead.

• Yes, optimization will be a direct dependency in tulip again. But I hope that cvxopt will be circumvented. Though I am not familiar with cvxopt history, its maintenance and compilation seem to have issues. For LP there are available several packages that directly interface to GLPK and other solvers. Taking advantage of this can make polytope a package "easy" to install (vs using cvxopt).

• For quadratic or other solvers, more searching is needed for the available python alternatives. Since it is not essential to tulip @1.1a it might as well be skipped as a dependency. Any user who knows how to use tulip.abstract.find_controller and write the necessary simulation wrapper around a MealyMachine that couples it with an AbstractSwitched system, most likely won't find it significantly difficult to also install cvxopt.

• I totally agree that missing simulation (and integration of RHC) is a big gap in functionality. However, if RHC is eventually moved to python-control and imported from there, then strictly speaking again tulip will not directly depend on quadratic solvers, unless some other new algorithm is added that does.

• The distinction between LP and QP is intended for implementation reasons. Good support for LP solvers seems to be more abundant than for QP solvers. Thus polytope can restrict itself to a robust interface to one or more LP solvers, with cvxopt an optional alternative. If QP is needed outside polytope, then we need worry about it only there.

Reducing the difficulty of installation may also avoid some user questions sent to the mailing list or posted as issues here.

[slivingston]

In c462b39 , I rolled back the removal of cvxopt. Besides the comments that @necozay makes, it is still used in tulip/abstract/find_controller.py.

Alternatively, we can regard cvxopt as an optional dependency, though I will not have time to do this before the upcoming release. We can have a vote about what to do if dissent remains.

[slivingston]

I am going to enforce in the examples that jtlv is a required dependency and that gr1c is optional. We can revisit this after releasing version 1.1a.

[slivingston]

The minimum version of gr1c is 0.9.0 now.

[slivingston]

As of da82dcb, there is a requirements.txt file that lists known working and current versions of Python dependencies. In the installation page of the User's Guide, at least version 0.9.0 of gr1c is requested. The required versions of other external tools are not documented, but I think it is enough to say try the most recent and then document known failures as they are discovered.