Python3 compatibility #1673

SteVwonder · 2018-09-23T18:48:51Z

This PR builds off of @trws's PR and updates the python bindings to work for both python 2 & 3.

Major changes include:

Any strings returned by the python bindings are now unicode strings. Most char*s coming out of Flux are assumed to be utf-8 minus a few exceptions (e.g., job states from the JSC).
Any strings passed to the bindings are checked to see if they are unicode strings, if so, they are encoded as utf-8. If they are raw bytes, they are passed straight through.
Moves the python bindings tests to t/python from src/bindings/python/tests.
Switches the tests from using the sideflux to using the subflux module
Adds tests for the jsc bindings and unicode support.

Choosing which python to build against

The current automake macro searches automatically for a python interpreter >= 2.7. AFAIK, it searches from the newest, stable python version (3.7) and works its way down (to 2.1). To override this behavior, users can set the PYTHON or PYTHON_VERSION env variables when running configure (these two vars are documented in the output of configure --help). After some searching online, I didn't see any pre-built solutions for compiling the bindings with both python2 and python3 simultaneously with autotools. It seems like a lot of projects that do this treat their bindings as separate project or a sub-project housed in-tree. I am starting a discussion on how to handle this in #1448.

Closes #1669
Closes #1448
Closes #1046
Closes #805

coveralls · 2018-09-23T19:14:53Z

Coverage decreased (-0.04%) to 79.961% when pulling 8d88e62 on SteVwonder:python3-compatibility into 8b195da on flux-framework:master.

codecov-io · 2018-09-29T03:38:25Z

Codecov Report

Merging #1673 into master will decrease coverage by 0.02%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #1673      +/-   ##
==========================================
- Coverage    79.3%   79.28%   -0.03%     
==========================================
  Files         186      186              
  Lines       35059    35063       +4     
==========================================
- Hits        27804    27798       -6     
- Misses       7255     7265      +10

Impacted Files	Coverage Δ
src/modules/pymod/py_mod.c	`0% <0%> (ø)`	⬆️
src/common/libflux/mrpc.c	`85.49% <0%> (-1.15%)`	⬇️
src/common/libflux/reactor.c	`91.78% <0%> (-0.92%)`	⬇️
src/broker/broker.c	`76% <0%> (-0.45%)`	⬇️
src/cmd/flux-module.c	`85.28% <0%> (-0.31%)`	⬇️
src/broker/module.c	`83.78% <0%> (-0.28%)`	⬇️
src/modules/connector-local/local.c	`74.51% <0%> (+1.35%)`	⬆️

SteVwonder · 2018-09-29T03:46:45Z

Finally all green! Ready for review once you are back, @trws.

Sorry about the order in which the commits show up in GitHub. I guess it orders them by the commit date rather than their order in the git history. Unfortunately, that might make this PR a little confusing to review, if you prefer to go through PRs commit-by-commit.

grondo · 2018-09-29T18:32:03Z

Cool! I'm really wondering now what is going in Travis. The builds on master keep failing with some kind of output truncation error, but your PR on top of current master worked. 😕

trws · 2018-10-02T07:59:21Z

This looks great to me. The only thing that occurs to me, are the flake tests back to running? I don't see anything that should tickle it but I remember @grondo mentioning they were partially deactivated at some point.

SteVwonder · 2018-10-02T20:56:28Z

@trws: that is a good point. I think pylint tests were running under the old travis config but have been turned off during the switch to Docker. I'll look into turning them back on. I believe they were turned off due to lots of errors from newer pylint versions.

grondo · 2018-10-02T22:18:19Z

I believe they were turned off due to lots of errors from newer pylint versions.

Yeah, I had to disable pylint because it wasn't passing :-)

Co-authored-by: Stephen Herbein <herbein1@llnl.gov>

- Uses `six` package for most of the py2 and py3 support - Also uses `from __future__ import print_function` Co-authored-by: Stephen Herbein <herbein1@llnl.gov>

SteVwonder · 2018-10-03T06:41:34Z

Pylint has been re-enabled, and the resulting errors fixed.

Bad first argument given to super class

The biggest change was switching super(self.__class__, self) to actually directly reference the class. self.__class__ works when the class is at the front of the MRO, but as soon as another class inherits from that class, an infinite recursion happens in the __init__. Most of the uses were in a class that was inside another class, so I doubt anyone would actually subclass them, but at least now pylint is happy.

no-else-return

For this error, in most cases, I obliged the linter, but in one particular case, I think the if/else is sufficiently large that the way it's currently written is easier to read/parse. So I overrode that case.

no-value-for-parameter

pylint really hates the ffi and lib modules being passed in as arguments to Wrapper.__init__. I don't see a better way to do it, so I just manually overrode each instance.

useless-object-inheritance

pylint3 doesn't like class WrapperBase(object). Disabled universally since this is required for py2 backwards compatibility.

trws · 2018-10-03T08:18:20Z

Wow, nice work. I'm happy, and would say LGTM.

trws · 2018-10-04T08:49:47Z

It may be possible to fix some of this, but let me lay out the reasoning for the current pattern for context.

python file copying

This was part of the reason for breaking out the built extensions into _flux. The python searchpath only uses the first module that matches, this is what lua does as well actually, which meant that when everything was under flux.* all of the files had to be in the same directory which had to be named flux. The split may allow pythonpath manipulation to fix that, but I hadn't tested it.

so copying out of .libs

This is mainly because libtool deals with library search paths and library generation in general in the most frustrating way conceivable. Since nothing but libtool can actually read .la files, and the .so files are invalid in the build path without explicitly adding library paths to the environment, there's a non-trivial amount of working around that broken behavior. If we used anything else, that would just build the sos as valid in the build tree then re-path them on install, none of that would be necessary. If we can avoid it, I'm all for it, but that was the best way I could think of at the time to build with the same tooling as the rest of flux and still get the job done.

garlick · 2018-10-04T13:25:09Z

Following up on a conversation @SteVwonder and I had yesterday about .libs. I had thought that its existence as an artifact of libtool was architecture dependent couldn't be portably assumed in build scripts. I remembered libtool on arm7l (32-bit) as the counter-example. However, I just checked that on a Pi and it works like other systems. I also verified 32-bit x86 works the same. Both were fairly recent debian-based.

If it isn't used on some platform or OS, I've lost track of which. Never mind?

trws · 2018-10-04T13:35:53Z

I think you're right that there is a case where it doesn't create the .libs directory, but IIRC that's only on systems that don't support shared objects, which means our modules and python's couldn't be built anyway, so I wasn't worrying about it.

SteVwonder · 2018-10-04T17:49:29Z

This was part of the reason for breaking out the built extensions into _flux. The split may allow pythonpath manipulation to fix that, but I hadn't tested it.

Embarrassingly enough, it was only after I removed the copying that the reason for the split finally "clicked" (thanks for the paving the way on that one). The split worked btw, there is no longer copying of source files over to the build tree (the exception being the __init__.py file in _flux).

Sidenote: the pymod macro that you (@trws) wrote is super handy! Thanks for that.

grondo · 2018-10-04T18:59:05Z

I can't tell off-hand, but do any of the commits in here need to be squashed together? (Not trying to make useless work for you though!)

Also, a comment on commit messages: commit messages should be written in the imperative for consistency (e.g. check that pylint... not checks that pylint...), and the body should be wrapped at about 72 cols, or else various tools (and git long on my terminals 😉) will make weird line breaks.

You might only want to address the commit message comments if you have to squash and rebase anyway.

SteVwonder · 2018-10-04T19:10:38Z

do any of the commits in here need to be squashed together

Yeah, I can squash some of the latter changes to the automake files. I did a squashing pass on the python changes earlier this week.

commit messages should be written in the imperative

Will do. Thanks!

the body should be wrapped at about 72 cols

Let me see if I can tweak my emacs config to enforce this automatically (I think the default is ~80)

- moves generated shared libraries to a _flux directory - adds *.py and *.c gitignores to _flux directory Done so that ultimately setuptools can be used for building the python bindings. Co-authored-by: Stephen Herbein <herbein1@llnl.gov>

- removes the user-passable flux handle to RPC which has changed from a flux_handle_t to a flux_future_t - removes a use of flux_rpc_multi - improves error checking in KVS API calls

Also includes minor cleanup of several of the tests

- all input strings are checked to determine if they need to be encoded all - returned strings are decoded as 'utf-8', there are a few exceptions where the strings are decoded as 'ascii' (like getting the state of a job from the JSC)

add source tree to PYTHONPATH in AM_TEST_ENVIRONMENT to remove need to copy the source py files into the build tree add BUILT_SOURCES primary and use `nodist_` prefix to prevent distribution of auto-generated c files replace copying of python .so's out of .libs directory with symlinking

- use a LOG_DRIVER rather than direct execution - rename python tests to have .py extension rather than .t - tests now run under the same python interpreter that flux was configured with This commit adds a thin wrapper around the tap-driver.sh to retain test output granularity while still ensuring the python tests are executed with the python interpreter that flux was configured with.

Disable unwanted and address relevant lint warnings. Bad first argument given to super class: The biggest change was switching `super(self.__class__, self)` to actually directly reference the class. `self.__class__` works when the class is at the front of the MRO, but as soon as another class inherits from that class, an infinite recursion happens in the `__init__`. Most of the uses were in a class that was inside another class, so I doubt anyone would actually subclass them, but at least now pylint is happy. no-else-return: For this error, in most cases, I obliged the linter, but in one particular case (src/bindings/python/flux/wrapper.py:224), I think the if/else is sufficiently large that the way it's currently written is easier to read/parse. So I overrode that case. no-value-for-parameter: pylint really hates the `ffi` and `lib` modules being passed in as arguments to `Wrapper.__init__`. I don't see a better way to do it, so I just manually overrode each instance. useless-object-inheritance: pylint3 doesn't like `class WrapperBase(object)`. Disabled universally since this is required for py2 backwards compatibility.

SteVwonder · 2018-10-04T19:40:32Z

Squashed a few commits and re-wrapped several commit message bodies to be < 72 characters.

This PR should be good to go now.

grondo · 2018-10-04T19:58:03Z

Awesome!

Let me see if I can tweak my emacs config to enforce this automatically (I think the default is ~80)

Yeah, sorry, this isn't too important, but it keeps git log output looking nice with 80 char wide terminals, since the commit message is always indented at least 4 and up to 8. (probably I'm the only one this affects, if so feel free to tell me to be quiet!)

grondo · 2018-10-04T20:00:52Z

Since @trws and @garlick approved this in writing and verbally, I'm merging.

SteVwonder · 2018-10-04T22:02:45Z

(probably I'm the only one this affects, if so feel free to tell me to be quiet!)

Not a problem at all; it is a pretty easily automated thing. Now that I have my emacs config setup, it shouldn't be a problem in the future.

Maybe we should consider adding a check for things like that and coding style to the CI so that maintainers don't have to bother asking for it to be addressed.

SteVwonder requested a review from trws September 23, 2018 18:48

SteVwonder force-pushed the python3-compatibility branch 2 times, most recently from 35eb00d to 5626853 Compare September 23, 2018 20:35

SteVwonder mentioned this pull request Sep 23, 2018

writing on the wall for python 3 support #1448

Closed

SteVwonder force-pushed the python3-compatibility branch 13 times, most recently from 8310bd9 to 3871db9 Compare September 29, 2018 03:19

SteVwonder and others added 3 commits October 2, 2018 19:05

travis: update deps and enable python3 in build matrix

39776e2

python: implement py3 compat

f6bb759

Co-authored-by: Stephen Herbein <herbein1@llnl.gov>

python/test: make python binding tests py3 compatible

8715f4e

- Uses `six` package for most of the py2 and py3 support - Also uses `from __future__ import print_function` Co-authored-by: Stephen Herbein <herbein1@llnl.gov>

SteVwonder force-pushed the python3-compatibility branch from 3871db9 to 1d3bc20 Compare October 3, 2018 06:06

SteVwonder force-pushed the python3-compatibility branch 2 times, most recently from 52d0f22 to f5e05e5 Compare October 4, 2018 06:27

trws and others added 14 commits October 4, 2018 12:23

python: make flux.core loading lazy

6ec5f92

python: track KVS and RPC C API changes

111aa1f

- removes the user-passable flux handle to RPC which has changed from a flux_handle_t to a flux_future_t - removes a use of flux_rpc_multi - improves error checking in KVS API calls

t/python: move tests from /src/bindings/python/test to /t

4d1d8f4

Also includes minor cleanup of several of the tests

t/python: ensure wrapper and request tests run under subflux

ad623f7

t/python: setup and teardown handle once per unittest class

3fe87d0

python: cache wrapped functions per-handle rather than globally

01366ff

python: support utf-8 unicode strings

91699a8

- all input strings are checked to determine if they need to be encoded all - returned strings are decoded as 'utf-8', there are a few exceptions where the strings are decoded as 'ascii' (like getting the state of a job from the JSC)

python: add job_state2num to jsc

0c216f8

t/python: add tests for unicode strings and new jsc bindings

bc6585e

t/python: fix asserts for ENOENTs

f55426a

SteVwonder force-pushed the python3-compatibility branch from f5e05e5 to 33532ff Compare October 4, 2018 19:26

grondo merged commit 1c86b38 into flux-framework:master Oct 4, 2018

SteVwonder deleted the python3-compatibility branch February 16, 2019 00:22

andre-merzky mentioned this pull request Nov 25, 2019

configure picks older python version #2556

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python3 compatibility #1673

Python3 compatibility #1673

SteVwonder commented Sep 23, 2018 •

edited

Loading

coveralls commented Sep 23, 2018 •

edited

Loading

codecov-io commented Sep 29, 2018 •

edited

Loading

SteVwonder commented Sep 29, 2018

grondo commented Sep 29, 2018

trws commented Oct 2, 2018

SteVwonder commented Oct 2, 2018

grondo commented Oct 2, 2018

SteVwonder commented Oct 3, 2018

trws commented Oct 3, 2018

trws commented Oct 4, 2018

garlick commented Oct 4, 2018

trws commented Oct 4, 2018

SteVwonder commented Oct 4, 2018

grondo commented Oct 4, 2018

SteVwonder commented Oct 4, 2018

SteVwonder commented Oct 4, 2018

grondo commented Oct 4, 2018

grondo commented Oct 4, 2018

SteVwonder commented Oct 4, 2018

Python3 compatibility #1673

Python3 compatibility #1673

Conversation

SteVwonder commented Sep 23, 2018 • edited Loading

Major changes include:

Choosing which python to build against

coveralls commented Sep 23, 2018 • edited Loading

codecov-io commented Sep 29, 2018 • edited Loading

Codecov Report

SteVwonder commented Sep 29, 2018

grondo commented Sep 29, 2018

trws commented Oct 2, 2018

SteVwonder commented Oct 2, 2018

grondo commented Oct 2, 2018

SteVwonder commented Oct 3, 2018

Bad first argument given to super class

no-else-return

no-value-for-parameter

useless-object-inheritance

trws commented Oct 3, 2018

trws commented Oct 4, 2018

python file copying

so copying out of .libs

garlick commented Oct 4, 2018

trws commented Oct 4, 2018

SteVwonder commented Oct 4, 2018

grondo commented Oct 4, 2018

SteVwonder commented Oct 4, 2018

SteVwonder commented Oct 4, 2018

grondo commented Oct 4, 2018

grondo commented Oct 4, 2018

SteVwonder commented Oct 4, 2018

SteVwonder commented Sep 23, 2018 •

edited

Loading

coveralls commented Sep 23, 2018 •

edited

Loading

codecov-io commented Sep 29, 2018 •

edited

Loading