Python 3 support for CyLP #28

jjhelmus · 2015-10-02T21:37:35Z

Changes needed for CyLP to work in Python 3.4.
A single code base is used which supports Python 2.7, 3.3, and 3.4.
Unit test pass in all of the above Python version.

closes #15

Python 3 does not differentiate betten int and long so the PyInt_ functions must be replaced with PyLong_. Macros from cx_Oracle project.

The izip function fron itertools was removed from Python 3, use of zip is recommended.

TypeError and NotImplementedError are build ins

long was removed in Python 3, use int in its place.

The build in reduce function was removed in Python 3 and must be imported from functools.

Due to the different methods that Python 2 and 3 represent strings it is necessary to encode the filename passed to the readMps function which can then be converted into a char* pointer for use in C. Since strings do not share a type in Python 2 and 3 the filename argument of the readMps function must not be typed.

sebastroy · 2015-10-30T18:15:22Z

1b798eb has a mistake: it's missing a parenthesis at the end of the line:

         print(0.5 * x * (x * G).T + np.dot(c, x) - self.objectiveOffset

in file QP.py

in the same commit in QPGen.py on the very first line it's missing a "t" to import:

         from __future__ impor print_function

I also get an error when executing "python setup.py install" (but I have it with pip install too...):
writing cylp.egg-info/PKG-INFO
Traceback (most recent call last):
File "setup.py", line 418, in
package_data={"cylp": ['cpp/.cpp', 'cpp/.hpp', 'cpp/*.h', 'VERSION']},)
File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "setup.py", line 392, in run
install.run(self)
File "/usr/lib/python3.5/site-packages/setuptools/command/install.py", line 65, in run
orig.install.run(self)
File "/usr/lib/python3.5/distutils/command/install.py", line 551, in run
self.run_command(cmd_name)
File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib/python3.5/site-packages/setuptools/command/install_egg_info.py", line 33, in run
self.run_command('egg_info')
File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib/python3.5/site-packages/setuptools/command/egg_info.py", line 173, in run
writer(self, ep.name, os.path.join(self.egg_info, ep.name))
File "/usr/lib/python3.5/site-packages/setuptools/command/egg_info.py", line 388, in write_pkg_info
metadata.write_pkg_info(cmd.egg_info)
File "/usr/lib/python3.5/distutils/dist.py", line 1107, in write_pkg_info
self.write_pkg_file(pkg_info)
File "/usr/lib/python3.5/distutils/dist.py", line 1128, in write_pkg_file
long_desc = rfc822_escape(self.get_long_description())
File "/usr/lib/python3.5/distutils/util.py", line 471, in rfc822_escape
lines = header.split('\n')
TypeError: a bytes-like object is required, not 'str'

The culprit is the s_README encoding. I just commented it out from setup.py but I guess the encoding from the getBdistFriendlyString function should be fixed instead. Thanks!

jjhelmus · 2015-10-30T19:01:18Z

Good catch on those mistakes @sebastroy, I added a commit to fix them. Investigating the install issue.

sebastroy · 2015-10-30T19:07:07Z

Great thanks!

This may break the bdist_mpkg and bdist_wininst commands

jjhelmus · 2015-10-30T20:31:03Z

Commit 13d5619 seem to fix the install issue although it might have broken the bdist_mpkg and bdist_wininst commands in setup.py. With commit 89ce697 and 7fd344c I am able to run python setup.py bdist_mpkg on my machine in Python 2.7 and Python 3.5 but it fails with lines concerning pax failing:

pax: Cpio header field is too small to store file ./cylp/VERSION
pax: Cpio header field is too small to store file ./cylp-0.7.4-py2.7.egg-info
pax: Cpio header field is too small to store file ./cylp-0.7.4-py2.7.egg-info/dependency_links.txt
pax: Cpio header field is too small to store file ./cylp-0.7.4-py2.7.egg-info/not-zip-safe
pax: Cpio header field is too small to store file ./cylp-0.7.4-py2.7.egg-info/PKG-INFO
pax: Cpio header field is too small to store file ./cylp-0.7.4-py2.7.egg-info/requires.txt
pax: Cpio header field is too small to store file ./cylp-0.7.4-py2.7.egg-info/SOURCES.txt
pax: Cpio header field is too small to store file ./cylp-0.7.4-py2.7.egg-info/top_level.txt
error: command '/bin/pax' failed with exit status 1

Unfortunately my knowledge of bdist_mpkg is limited and I have no idea where to begin to debug this. I get the same error messages on the current master so this may be limited to my machine.

I have no way of easily testing the bdist_wininst command.

sebastroy · 2015-10-31T00:28:34Z

I'm on linux myself... If I replace the call to getBdistFriendlyString and use plain "u" like the other two lines, install works but it possibly breaks the 2.7 version... In a nutshell, the issue is under python 2, str = bytes and under 3, bytes = bytes.

jjhelmus · 2016-02-09T17:04:45Z

Wanted to bump this PR and see if there was anything blocking this from being merged. I am very interested in having a Python 3 version of CyLP and am willing to fix up any outstanding issues.

sebastroy · 2016-02-09T17:09:02Z

Fine by me

scollis · 2016-11-29T13:33:09Z

Ping.. I am almost entirely 3 now.. Would be great to see this in the master branch

RhysU · 2017-04-18T03:25:29Z

Ditto on a desire to see this in master.

Guzzii · 2017-05-02T15:20:02Z

It would be really great to see python3 support from CyLP

scollis · 2017-07-13T16:32:46Z

Are we going to see action on this?

SRserves85 · 2017-08-08T18:43:03Z

Bump, Is this ever happening?

tkralphs · 2017-08-11T03:45:12Z

I would still like to get back to CyLP and spend some time addressing the backlog of PRs, but I have obviously lost credibility in claiming this will happen. I will try to find a weekend soon when I can play with it. Obviously, it is not difficult to merge the PR. The time consuming part is testing. If those who have been using CyLP with Python 3 can post and say what their experience has been, it would be helpful in getting a level of comfort.

sschnug · 2018-01-29T18:07:58Z

I did not use this pull-request extensively, but i never approached problems!

While documentation and (mostly) missing development is depressing, it's still a very interesting project and official python3 support would be great!

(Two examples: footrule-ranking (Clp), kemeny-ranking (Cbc))

scollis · 2018-01-29T18:25:45Z

We are now just redirecting folks for Py-ART to use the @jjhelmus branch. Having master fork updated with jjhelmus/py3 would be great

scollis · 2018-02-10T02:07:27Z

All, and specifically @tkralphs , We are at the point now that I am considering creating a fork with @jjhelmus 's work, making his py3 branch master and using that for Py-ART. This is a great project by we need a Py3 compatible master branch

tkralphs · 2018-02-21T01:07:13Z

I don't think a fork would be in anyone's best interest. Although this is not exactly my project, I do have the ability to add collaborators and I think I said at some point that I would be happy to hear from anyone who is willing to help, though maybe that got lost in the fray. The implication of the above comment is that there are people who would be happy to help. I would want to have some discussions with those people to make sure we're on the same page, but this seems to be the way forward.

scollis · 2018-02-21T15:54:46Z

Ok! I can help.. It looks like we have some conflicts.

scollis · 2018-02-21T20:02:14Z

@jjhelmus was it just Py3 compatibility issues or did you refactor? I am wondering if I should start with the current master and work from that or try to resolve the conflicts.

scollis · 2018-02-21T20:02:39Z

Actually now I look at the commits it looks quite involved.

jjhelmus · 2018-02-21T21:30:04Z

I created this patch over two years, I do not recall what all was done.

Unfortunately, I do not have the time to re-work the patches or otherwise move this forward. I'd be happy for someone else to pick up this work but I cannot offer any help myself.

scollis · 2018-02-21T21:31:45Z

No worries and understand @jjhelmus

tkralphs · 2018-02-24T20:26:34Z

As there haven't been very many commits since this PR was created and such commits were not very involved, I hope that resolving the conflicts would be not be too difficult. @scollis, perhaps we can schedule a face-to-face meeting and discuss a strategy? It would certainly be helpful to have someone who's using CyLP regularly and has a little bandwidth to push this forward. I'll actually be using CyLP with students in the next month or two and I'm sure they will be wanting to use Python 3, so between the two of us, we should be able to get this going.

SteveDiamond · 2019-03-13T16:47:09Z

Hi everyone, is CyLP a live project? I'm a developer of cvxpy, and we use CyLP to interface to Cbc. It's an important tool for us, so if we can help out in modernizing CyLP let us know. I've posted here because Python 3 support is essential.

sebastroy · 2019-03-13T16:55:03Z

My rule of thumb for "dead" projects is last commit more than 4-6mo ago.

For what it's worth, we are using ortools now. It was a bit clunky a few years ago but it came a long way since then and is very stable. It does not have 100% all the cbc features (neither does CyLP for that matter) but it had enough for us. Unlike pulp, it does interface with the coin-or library rather than launch cbc in a shell wrapped in python code.

scollis · 2019-03-13T17:11:03Z

Hey @SteveDiamond We still use it.. I have been very slack in taking @tkralphs on his offer.. Having enough trouble with Py-ART which uses CyLP.

scollis · 2019-03-13T17:11:28Z

What would be amazing is a LP code in Scipy.optimize or suchlike

tkralphs · 2019-03-13T18:11:17Z

I'm still here and still hoping for a co-maintainer :). Although I have zero credibility left in terms of promises, I am going to be using CyLP myself in one of my classes this semester and will be trying to do some maintenance work over the next couple of weeks.

tkralphs · 2019-03-22T04:30:47Z

I just re-generated the Cython files and updated this PR in #67.

tkralphs · 2019-03-26T04:50:14Z

I posted a beta version with Python 3 support to Pypi:

pip install --pre cylp

There is a binary wheel for Windows, but you still need to build Cbc before installing on Linux and OS X.

tkralphs · 2019-03-28T03:09:07Z

I seem to have figured out bug that prevented Cython 0.29 from properly re-generating the files. I re-generated the files once again with Cython 0.29 and just pushed a binary wheel up for Python 3.7 on Windows. With a little more testing, I'll be ready to merge and release an official Python 3 version.

jjhelmus added 18 commits October 2, 2015 10:56

PY3: Explicit decoding of location in setup.py

e414ab6

PY3: Use relative imports inside cylp

cd36bcb

PY3: define PyInt_* macros for Python 3.x

347d8c0

Python 3 does not differentiate betten int and long so the PyInt_ functions must be replaced with PyLong_. Macros from cx_Oracle project.

PY3: Use print function

1b798eb

STY: Remove excess whitespace

25e903a

PY3: Use range in place of depreciated xrange

3d8e1fc

STY: Additional whitespace removal

dc4ec88

PY3: Explicit list creation from iterators

81c7a98

PY3: Remove tuple parameter unpacking

2ba9596

PY3: Use zip in place of izip when not available

06d8b8d

The izip function fron itertools was removed from Python 3, use of zip is recommended.

MAINT: Remove unneeded import of exceptions

15736c9

TypeError and NotImplementedError are build ins

PY3: Use int for long in Python 3

b33d815

long was removed in Python 3, use int in its place.

PY3: Import reduce from functools

d4da697

The build in reduce function was removed in Python 3 and must be imported from functools.

PY3: Add __mul__, __add__, __sub__ methods to CyLPExpr

017e6ae

DEV: Add regenerate_cpp_files.sh script

f10af07

PY3: Cast str to byte when passing to C functions

3513909

MAINT: Regenerate cpp files using Cython

288afdf

jjhelmus mentioned this pull request Oct 2, 2015

Phase processing in python3 ARM-DOE/pyart#389

Closed

BUG: Fix typos from commit 1b798eb

b7346ef

jjhelmus added 3 commits October 30, 2015 14:46

PY3: Do not encode README in getBdistFriendlyString

13d5619

This may break the bdist_mpkg and bdist_wininst commands

STY: Retab fixBinaries.py

89ce697

PY3: Explicit decode of library name in fixBinaries

7fd344c

tkralphs mentioned this pull request Mar 6, 2017

byte vs str breakage in setup #46

Closed

sschnug mentioned this pull request Apr 16, 2018

Breaks with python-3.6 #62

Closed

tkralphs self-assigned this Mar 13, 2019

tkralphs mentioned this pull request Mar 22, 2019

Python 3 support #67

Merged

tkralphs merged commit 7fd344c into coin-or:master Jun 30, 2019

tkralphs mentioned this pull request Nov 30, 2021

BUG: Error when using CyLP ARM-DOE/pyart#1022

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3 support for CyLP #28

Python 3 support for CyLP #28

jjhelmus commented Oct 2, 2015

sebastroy commented Oct 30, 2015

jjhelmus commented Oct 30, 2015

sebastroy commented Oct 30, 2015

jjhelmus commented Oct 30, 2015

sebastroy commented Oct 31, 2015

jjhelmus commented Feb 9, 2016

sebastroy commented Feb 9, 2016

scollis commented Nov 29, 2016

RhysU commented Apr 18, 2017

Guzzii commented May 2, 2017

scollis commented Jul 13, 2017

SRserves85 commented Aug 8, 2017

tkralphs commented Aug 11, 2017

sschnug commented Jan 29, 2018

scollis commented Jan 29, 2018

scollis commented Feb 10, 2018

tkralphs commented Feb 21, 2018

scollis commented Feb 21, 2018

scollis commented Feb 21, 2018

scollis commented Feb 21, 2018

jjhelmus commented Feb 21, 2018

scollis commented Feb 21, 2018

tkralphs commented Feb 24, 2018

SteveDiamond commented Mar 13, 2019

sebastroy commented Mar 13, 2019

scollis commented Mar 13, 2019

scollis commented Mar 13, 2019

tkralphs commented Mar 13, 2019

tkralphs commented Mar 22, 2019

tkralphs commented Mar 26, 2019

tkralphs commented Mar 28, 2019

Python 3 support for CyLP #28

Python 3 support for CyLP #28

Conversation

jjhelmus commented Oct 2, 2015

sebastroy commented Oct 30, 2015

jjhelmus commented Oct 30, 2015

sebastroy commented Oct 30, 2015

jjhelmus commented Oct 30, 2015

sebastroy commented Oct 31, 2015

jjhelmus commented Feb 9, 2016

sebastroy commented Feb 9, 2016

scollis commented Nov 29, 2016

RhysU commented Apr 18, 2017

Guzzii commented May 2, 2017

scollis commented Jul 13, 2017

SRserves85 commented Aug 8, 2017

tkralphs commented Aug 11, 2017

sschnug commented Jan 29, 2018

scollis commented Jan 29, 2018

scollis commented Feb 10, 2018

tkralphs commented Feb 21, 2018

scollis commented Feb 21, 2018

scollis commented Feb 21, 2018

scollis commented Feb 21, 2018

jjhelmus commented Feb 21, 2018

scollis commented Feb 21, 2018

tkralphs commented Feb 24, 2018

SteveDiamond commented Mar 13, 2019

sebastroy commented Mar 13, 2019

scollis commented Mar 13, 2019

scollis commented Mar 13, 2019

tkralphs commented Mar 13, 2019

tkralphs commented Mar 22, 2019

tkralphs commented Mar 26, 2019

tkralphs commented Mar 28, 2019