Enable Pylint checks for PyWren #144
Conversation
Also add ignore rules for special cases
Possible workaround for pylint-dev/pylint#1296
shivaram
changed the title
|
@ooq This probably needs more closer review than the last one as it actually has some code changes. The travis check on the branch passes https://travis-ci.org/pywren/pywren/builds/252271798 |
| def save_memoryview(self, obj): | ||
| """Fallback to save_string""" | ||
| Pickler.save_string(self, str(obj)) | ||
| Pickler.save_str(self, str(obj)) # pylint: disable=no-member |
There was a problem hiding this comment.
Good qn - so if you see the code paths below, we call save_memoryview only for Python 3. And if you look at the _Pickler implementation in cpython [1] then you can see that the method is called save_str. The same file in the 2.7 branch [2] uses save_string
[1] https://github.com/python/cpython/blob/3.5/Lib/pickle.py#L698
[2] https://github.com/python/cpython/blob/2.7/Lib/pickle.py#L486
| - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION numpy pytest cython nose boto3 PyYAML Click pytest numba | ||
| - source activate test-environment | ||
| - pip install glob2 | ||
| - pip install glob2 pylint tornado |
There was a problem hiding this comment.
how was tornado installed before?
There was a problem hiding this comment.
It was not installed before but we seemed to use it in cloudpickle.py[1] if it was available. The problem is lint doesn't like it when we import packages in _rebuild_tornado_coroutine.
[1]
| - RUN_LAMBDA=true | ||
| - RUN_MACROREDUCE=true | ||
| - RUN_COMMANDLINE=true | ||
| - RUN_PYLINT=true |
There was a problem hiding this comment.
Maybe it's more resource-economic to run PyLint for all configurations, and halt the actual tests if linting fails? Essentially, passing linting should be a pre-requisite.
There was a problem hiding this comment.
Ideally, it would be nice if we can run RUN_PYLINT=true first, and make its passing be a dependence for RUN_LAMBDA=true, etc. But I don't think it's feasible with Travis?
There was a problem hiding this comment.
We can form a master script that does this - like they describe in https://docs.travis-ci.com/user/customizing-the-build/#Implementing-Complex-Build-Steps ?
There was a problem hiding this comment.
Cool. I think we can make a separate issue/PR for this.
| # FIXME delete individuals | ||
| """ | ||
| for instance_name, instance_obj in instance_list: | ||
| logger.debug('Terminating instance %s', instance_name) |
There was a problem hiding this comment.
No its actually trying to use instance_name to avoid an unused variable warning. i can also make it an _
| for mod_path in list(mod_paths): | ||
| if module in mod_path and mod_path in mod_paths: | ||
| mod_paths.remove(mod_path) | ||
| if exclude_modules: |
There was a problem hiding this comment.
Should we be more explicit, with if exclude_modules is None?
There was a problem hiding this comment.
We also dont want to run this loop if exclude_modules is an empty list ?
There was a problem hiding this comment.
You're right. I'd assumed that a user would never pass in an empty list to exclude_modules, but that's not the case.
| This module was extracted from the `cloud` package, developed by `PiCloud, Inc. | ||
| <https://web.archive.org/web/20140626004012/http://www.picloud.com/>`_. | ||
There was a problem hiding this comment.
I'm not sure if we'll need to understand the cloudpickle sources and maintain them in the future. If not, maybe we should just exclude them from linting completely? Diverging them too much from the original source might make it harder to sync with updated cloudpickle code.
There was a problem hiding this comment.
yeah this is a very good point. I thought about this while making changes as well. I guess we need to pull in @ericmjonas for this :)
@ericmjonas do you have an idea how many changes we made when we forked from cloudpickle ? Also do you see us syncing their more recent changes ?
There was a problem hiding this comment.
I think we will sync their more recent changes; is there a way to tell pylint to ignore certain files? In the interest of getting this out there I think we should merge this PR asap.
There was a problem hiding this comment.
I think that's doable? @shivaram
Let's then ignore those files, revert the pylint changes and merge this PR.
There was a problem hiding this comment.
https://stackoverflow.com/a/10238540/8322280
# pylint: skip-file
There was a problem hiding this comment.
Yeah I'll do this tonight and merge after that
ooq
left a comment
ooq
left a comment
There was a problem hiding this comment.
This looks good to me. I have only one question w.r.t. how we should maintain modified external sources such as cloudpickle.py.
|
@ooq I reverted the cloudpickle.py changes (it will look like a diff here because the earlier PR had some whitespace fixes and I also reverted those). Waiting for Travis now and I guess somebody will need to approve this PR as well ! |
|
Thanks all ! This is merged now |
This PR addresses a bunch of remaining Pylint issues and for the ones which are not addressed, we add a local comment to disable Pylint. Finally this PR also adds a Pylint check into Travis.