Connecting choco with PuLP

[pchtsp]

Good day!

I'm interested on integrating choco with python's PuLP library to test it with the models that I've modeled so far.
This library (PuLP) lets the user build one single model and then communicates with several linear programming solvers: CBC, GUROBI, CPLEX, XPRESS, glpk.

I guess there are multiple ways of doing the integration. One very simple way pulp communicates with some of these solvers is the following:

1. it creates a .mps or .lp file
2. it then calls the solver as a command line with the file as input,
3. the solver generates the output as a .sol file
4. PulP reads the output and converts it into the corresponding python variables.

I'm wondering if this could be done with choco. That is:

1. Is it possible to use choco as a command line application?
2. Is it possible for choco to read .mps or .lp files?

I'm aware that much of the functionality of the CP solver is lost sometimes doing these kinds of translations, specially if the models come from the LP / MIP world. Still I think it's useful to be able to at least test the solver for lazy people like me.

Of course, I'd be more than glad to edit the necessary parts in PuLP if needed (although I'm not the maintainer of the repo).

thanks!

Franco

[cprudhom]

Hi Franco,

We do have a .mps file parser, but I didn't push changes yet.
I'm running out of time to do it, but I will try to push by the end of the month (no guarantee !)

I let you know,

Best

[cprudhom]

I just pushed the code. You can have a look at the readme file.
Beware, the command line may change in the near future (for simplification though).

I did not plan any release yet.

Best,
CP

[pchtsp]

Nice, thanks Charles!

So what would be the steps to build and test it via command line? I'm not at all versed in the arts of java.

(by the way I think the MPS.md has an error in the example command)

java -cp .:/path/to/choco-parsers-4.0.4-with-dependencies.jar org.chocosolver.parser.xcsp.ChocoMPS [<options>] [<file>]

should be:

java -cp .:/path/to/choco-parsers-4.0.4-with-dependencies.jar org.chocosolver.parser.mps.ChocoMPS [<options>] [<file>]

?

Things I've already done, watching the instructions in the READMEs:

1. downloaded the choco-solver and choco-parser repos.
2. Tried to build the choco-solver (not sure if needed): it appears to have worked.
3. tried to build the choco-parser: not worked, got the following error when runnin mvn clean install -DskipTests in the root dir of the repo:

[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building A FlatZinc parser for Choco3 4.0.5-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for org.choco-solver:choco-solver:jar:4.0.7-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.771 s
[INFO] Finished at: 2018-03-19T22:45:53+01:00
[INFO] Final Memory: 14M/303M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project choco-parsers: Could not resolve dependencies for project org.choco-solver:choco-parsers:jar:4.0.5-SNAPSHOT: Could not find artifact org.choco-solver:choco-solver:jar:4.0.7-SNAPSHOT -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

thanks again!

Franco

[cprudhom]

Hi,

Thank you, you are right about the readme.

Here are the required steps to work with master branch (no need to download choco first):

1. download the source (make sure this is the last version, I pushed something this morning)
2. unzip the archive
3. go to the directory and open a terminal
4. run the maven command : mvn clean install -DskipTests
5. run the command java -cp .:/path/to/choco-parsers-4.0.5-SNAPSHOT-with-dependencies.jar org.chocosolver.parser.mps.ChocoMPS /path/to/instance.mps

Note that .mps.gz extensions are allowed.

Best,
CP

[pchtsp]

Salut Charles!

Thank you for the help. I think I got it working now, although I had to still do some weird things:

Even with the new version I was getting the same error about choco-solver not being available. From what I got from the error and what I could search for, maven does not have that version 4.0.7-SNAPSHOT available while choco-parser has the dependency on this specific version of choco-solver in its pom.xml file.

I tried changing the version in the pom.xml file to 4.0.6; which is the latest one in maven and, though it found it, apparently it still got some other building errors.

Finally, what I did was to copy the choco-solver-4.0.7-SNAPSHOT.jar from the choco-solver I built from the choco-solver repo and paste it together with its pom file in my maven repository directory, does 'installing it manually'.

Then apparently it built ok and now works with at least one example.

I'll keep you updated with what I do.

thanks,

Franco

[cprudhom]

That is strange because the snapshot directory is declared in the pom.xml and version 4.0.7-SNAPSHOT is release on that directory.

But, downloading the last version of choco from github is ok too.

Best,
CP

[pchtsp]

Hello once again!

I've been playing with pulp calling choco and apparently the normal solving works correctly. That's good!

There are a couple of things that are yet to work / I have doubts about:

1. Is there any command line option to have the result be written to a file? Right now I'm just rederecting the output (stdout) to a file but it would be a little nicer if it'd be done via a command line option.
2. I get an error when trying the execute the solving with multiple processors.
3. Finally, and this is just curiosity: is there too much difference between performance of choco when run via the mps version of a problem vs building the model by oneself using the CP functions offered by choco? In other words: is it fair to compare choco and its MPS parser with other solvers? or should I really build the model using the java API for the comparison to really be fair?

For number 2, here's the response I get from choco. Taking out the -p 3 option makes it work. I've tried with different examples and got the same error. I even tried forcing all variables of my problem to be integer but still got the error.

what I run:

java -cp .:/home/pchtsp/Documents/resources/choco-parsers/target/choco-parsers-4.0.5-SNAPSHOT-with-dependencies.jar org.chocosolver.parser.mps.ChocoMPS -p 3 /home/pchtsp/Documents/resources/choco-parsers/src/test/resources/mps/example1.mps 

what I get in response:

c [-p, 3, /home/pchtsp/Documents/resources/choco-parsers/src/test/resources/mps/example1.mps]
c 3 solvers in parallel
c Unimodular: false
c Unimodular: false
c Unimodular: false
Exception in thread "main" java.lang.UnsupportedOperationException: ParallelPortfolio cannot deal with real variable objective optimization problems
    at org.chocosolver.solver.ParallelPortfolio.check(ParallelPortfolio.java:412)
    at org.chocosolver.solver.ParallelPortfolio.prepare(ParallelPortfolio.java:254)
    at org.chocosolver.solver.ParallelPortfolio.solve(ParallelPortfolio.java:188)
    at org.chocosolver.parser.mps.MPS.manyThread(MPS.java:181)
    at org.chocosolver.parser.mps.MPS.solve(MPS.java:150)
    at org.chocosolver.parser.mps.ChocoMPS.main(ChocoMPS.java:27)
s UNKNOWN

PC details: I'm using Ubuntu 16.04 LTS 64 bits with an i7 processor and 16GB RAM.

Thanks for your help!

F.

[cprudhom]

Hi Franco,

1. There is no such option. Can you create a new issue ?

2. That’s a limitation of the current portfolio, it doesn’t manage real variable as objective variable. That should be modified too. Can you create another issue on that point too (related to choco, not parser) ? Btw, thank you for the MWE.

3. I think this is pretty fair. You will pay the cost of parsing the file but it is most of the time negligible in comparison to the resolution time.

Thank you for your contribution.

Best,
CP

[jgFages]

Hi,

Just a remark on point 3) it should be fair compared to building exactly the same model using choco api, but it is possible that a cp user would build a different formulation of the problem (more adapted to cp). At least, it is very common for us to specify a search heuristic (telling which variables to branch on) and this usually helps a lot the solver.

[pchtsp]

thanks to both: I'll create the issues as proposed.

I'm not really sure about (2) though because I at, some point, forced all my variables to be Integer, and even then I got the same error about real variables.

And @jgFages, I'm guessing it's not possible to assign branching priorities via the command line right now. Not sure if this would be hard to implement.

thanks!

F.

[pchtsp]

Well the connection has been working for some time in a branch in my fork of pulp: https://github.com/pchtsp/pulp-or/tree/CHOCO

As you can see here, I've embedded the choco parser jar file inside the repo to be able to distribute it together with the pulp library.

I recently received a response from the pulp project manager that has make me wonder: does the choco license permit this kind of distribution? from what I've read in the license file, it does. But I just wanted to confirm this with you.

Besides this question, I guess we could close this ticket because I do not think there is much else to do.

regards,

Franco

[cprudhom]

Yes, BSD is unrestrictive. Expect if we change the license, I can see no limitation to embed choco in PuLP.
That's very nice btw.
This is though limited to pure integer FO, right ?

[pchtsp]

FO = Fonction objectif ?

pulp is built to formulate MIP problems since it, until now, only connected to mixed linear programming solvers such as gurobi, cplex, etc. I have not tried to build any weird non-linear constraints to see if they're correctly passed to CHOCO but I'd expect it to be possible.

I'm using the mps standard from choco-parsers to send the problem to CHOCO so I'd guess the formulation would also be constrained by the limitations on the file type.

Finally, pulp has no support for CP constraints such as "all different" or "cardinality constraint" so everything would have to go via "algebraic equations".

As a summary, what I've tested is:

• Forced all variables as integers with a bounded, discrete, range.
• All constraints as linear relations between the variables.
• An objective function which is a linear combination of variables.
• Only used one core in CHOCO (portfolio: manage real variable as objective variable choco-solver#582).

F.