Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Pylint checks for PyWren #144

Merged
merged 10 commits into from Aug 4, 2017
Merged

Enable Pylint checks for PyWren #144

merged 10 commits into from Aug 4, 2017

Conversation

@shivaram
Copy link
Collaborator

@shivaram shivaram commented Jul 11, 2017

This PR addresses a bunch of remaining Pylint issues and for the ones which are not addressed, we add a local comment to disable Pylint. Finally this PR also adds a Pylint check into Travis.

@shivaram shivaram changed the title Enable PyLint checks for PyWren Enable Pylint checks for PyWren Jul 11, 2017
@shivaram
Copy link
Collaborator Author

@shivaram shivaram commented Jul 11, 2017

@ooq This probably needs more closer review than the last one as it actually has some code changes. The travis check on the branch passes https://travis-ci.org/pywren/pywren/builds/252271798

@@ -158,17 +156,17 @@ def dump(self, obj):

def save_memoryview(self, obj):
"""Fallback to save_string"""
Pickler.save_string(self, str(obj))
Pickler.save_str(self, str(obj)) # pylint: disable=no-member

This comment has been minimized.

@apengwin

apengwin Jul 18, 2017
Contributor

Should this be save_string?

This comment has been minimized.

@shivaram

shivaram Jul 19, 2017
Author Collaborator

Good qn - so if you see the code paths below, we call save_memoryview only for Python 3. And if you look at the _Pickler implementation in cpython [1] then you can see that the method is called save_str. The same file in the 2.7 branch [2] uses save_string

[1] https://github.com/python/cpython/blob/3.5/Lib/pickle.py#L698
[2] https://github.com/python/cpython/blob/2.7/Lib/pickle.py#L486

@@ -34,7 +34,7 @@ install:
- conda info -a
- conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION numpy pytest cython nose boto3 PyYAML Click pytest numba
- source activate test-environment
- pip install glob2
- pip install glob2 pylint tornado

This comment has been minimized.

@ooq

ooq Jul 19, 2017
Collaborator

how was tornado installed before?

This comment has been minimized.

@shivaram

shivaram Jul 19, 2017
Author Collaborator

It was not installed before but we seemed to use it in cloudpickle.py[1] if it was available. The problem is lint doesn't like it when we import packages in _rebuild_tornado_coroutine.

[1]

def is_tornado_coroutine(func):

@@ -87,3 +88,4 @@ env:
- RUN_LAMBDA=true
- RUN_MACROREDUCE=true
- RUN_COMMANDLINE=true
- RUN_PYLINT=true

This comment has been minimized.

@ooq

ooq Jul 26, 2017
Collaborator

Maybe it's more resource-economic to run PyLint for all configurations, and halt the actual tests if linting fails? Essentially, passing linting should be a pre-requisite.

This comment has been minimized.

@ooq

ooq Jul 26, 2017
Collaborator

Ideally, it would be nice if we can run RUN_PYLINT=true first, and make its passing be a dependence for RUN_LAMBDA=true, etc. But I don't think it's feasible with Travis?

This comment has been minimized.

@shivaram

shivaram Jul 26, 2017
Author Collaborator

We can form a master script that does this - like they describe in https://docs.travis-ci.com/user/customizing-the-build/#Implementing-Complex-Build-Steps ?

This comment has been minimized.

@ooq

ooq Jul 26, 2017
Collaborator

Cool. I think we can make a separate issue/PR for this.

@@ -173,10 +173,10 @@ def terminate_instances(instance_list):
# FIXME delete individuals
"""
for instance_name, instance_obj in instance_list:
logger.debug('Terminating instance %s', instance_name)

This comment has been minimized.

@ooq

ooq Jul 26, 2017
Collaborator

This is from merging?

This comment has been minimized.

@shivaram

shivaram Jul 26, 2017
Author Collaborator

No its actually trying to use instance_name to avoid an unused variable warning. i can also make it an _

This comment has been minimized.

@ooq

ooq Jul 26, 2017
Collaborator

I see. It's cool.

for mod_path in list(mod_paths):
if module in mod_path and mod_path in mod_paths:
mod_paths.remove(mod_path)
if exclude_modules:

This comment has been minimized.

@apengwin

apengwin Jul 26, 2017
Contributor

Should we be more explicit, with if exclude_modules is None?

This comment has been minimized.

@shivaram

shivaram Jul 26, 2017
Author Collaborator

We also dont want to run this loop if exclude_modules is an empty list ?

This comment has been minimized.

@apengwin

apengwin Jul 26, 2017
Contributor

You're right. I'd assumed that a user would never pass in an empty list to exclude_modules, but that's not the case.

@@ -12,7 +12,7 @@
<https://web.archive.org/web/20140626004012/http://www.picloud.com/>`_.

This comment has been minimized.

@ooq

ooq Jul 26, 2017
Collaborator

I'm not sure if we'll need to understand the cloudpickle sources and maintain them in the future. If not, maybe we should just exclude them from linting completely? Diverging them too much from the original source might make it harder to sync with updated cloudpickle code.

This comment has been minimized.

@shivaram

shivaram Jul 26, 2017
Author Collaborator

yeah this is a very good point. I thought about this while making changes as well. I guess we need to pull in @ericmjonas for this :)

@ericmjonas do you have an idea how many changes we made when we forked from cloudpickle ? Also do you see us syncing their more recent changes ?

This comment has been minimized.

@ericmjonas

ericmjonas Aug 3, 2017
Collaborator

I think we will sync their more recent changes; is there a way to tell pylint to ignore certain files? In the interest of getting this out there I think we should merge this PR asap.

This comment has been minimized.

@ooq

ooq Aug 3, 2017
Collaborator

I think that's doable? @shivaram
Let's then ignore those files, revert the pylint changes and merge this PR.

This comment has been minimized.

This comment has been minimized.

@shivaram

shivaram Aug 3, 2017
Author Collaborator

Yeah I'll do this tonight and merge after that

Copy link
Collaborator

@ooq ooq left a comment

This looks good to me. I have only one question w.r.t. how we should maintain modified external sources such as cloudpickle.py.

@shivaram
Copy link
Collaborator Author

@shivaram shivaram commented Aug 4, 2017

@ooq I reverted the cloudpickle.py changes (it will look like a diff here because the earlier PR had some whitespace fixes and I also reverted those).

Waiting for Travis now and I guess somebody will need to approve this PR as well !

@ooq
ooq approved these changes Aug 4, 2017
Copy link
Collaborator

@ooq ooq left a comment

LGTM. Thanks @shivaram ! Let's merge it as soon as tests pass.

@shivaram shivaram merged commit ba14fa2 into master Aug 4, 2017
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@shivaram
Copy link
Collaborator Author

@shivaram shivaram commented Aug 4, 2017

Thanks all ! This is merged now

@Vaishaal Vaishaal deleted the pylint-more-checks branch Nov 28, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.