New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+2] Implements ducktyping to allow for alternative Memory implementations #9584

Merged
merged 68 commits into from Aug 30, 2017

Conversation

Projects
None yet
6 participants
@thechargedneutron
Contributor

thechargedneutron commented Aug 18, 2017

Reference Issue

Fixes #9563

What does this implement/fix? Explain your changes.

Introduced ducktyping in pipeline.py as well as in hierarchical.py

Any other comments?

Added tests as well.

@thechargedneutron

This comment has been minimized.

Show comment
Hide comment
@thechargedneutron

thechargedneutron Aug 18, 2017

Contributor

How to import joblib. Tests failed because of ImportError. Do I need to mention somewhere that I used a new module?

Contributor

thechargedneutron commented Aug 18, 2017

How to import joblib. Tests failed because of ImportError. Do I need to mention somewhere that I used a new module?

@jnothman

I think in practice we also access memory.cachedir. Seeing as this is just used as a proxy for "is there actual caching happening", we can just use getattr(memory, 'cachedir', True) in its place.

Show outdated Hide outdated sklearn/cluster/hierarchical.py Outdated

thechargedneutron added some commits Aug 19, 2017

@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman

jnothman Aug 19, 2017

Member
Member

jnothman commented Aug 19, 2017

@thechargedneutron

This comment has been minimized.

Show comment
Hide comment
@thechargedneutron

thechargedneutron Aug 19, 2017

Contributor

I removed those lines and added a check memory.cachedir condition. Can you please check the last commit I made and decide whether it works?

Contributor

thechargedneutron commented Aug 19, 2017

I removed those lines and added a check memory.cachedir condition. Can you please check the last commit I made and decide whether it works?

thechargedneutron added some commits Aug 19, 2017

@thechargedneutron

This comment has been minimized.

Show comment
Hide comment
@thechargedneutron

thechargedneutron Aug 19, 2017

Contributor

I hope this satisfies the requirements. Kindly review it and suggest changes, if any.

Contributor

thechargedneutron commented Aug 19, 2017

I hope this satisfies the requirements. Kindly review it and suggest changes, if any.

@glemaitre

This comment has been minimized.

Show comment
Hide comment
@glemaitre

glemaitre Aug 21, 2017

Contributor

You should change your title to MRG. You will get more attention from the developers :)

Contributor

glemaitre commented Aug 21, 2017

You should change your title to MRG. You will get more attention from the developers :)

@thechargedneutron

This comment has been minimized.

Show comment
Hide comment
@thechargedneutron

thechargedneutron Aug 21, 2017

Contributor

I am waiting for a review. I think it's complete but not sure ;) What do you think?

Contributor

thechargedneutron commented Aug 21, 2017

I am waiting for a review. I think it's complete but not sure ;) What do you think?

@glemaitre

You need to add check_memory in the documentation in doc/modules/classes.rst and doc/developers/utilities.rst
You need also to add an entry in the what's new.

Show outdated Hide outdated sklearn/utils/validation.py Outdated
Show outdated Hide outdated sklearn/utils/validation.py Outdated
Show outdated Hide outdated sklearn/cluster/hierarchical.py Outdated
Show outdated Hide outdated sklearn/pipeline.py Outdated
Show outdated Hide outdated sklearn/cluster/tests/test_hierarchical.py Outdated
Show outdated Hide outdated sklearn/cluster/tests/test_hierarchical.py Outdated
@@ -155,6 +156,14 @@ def _shape_repr(shape):
return "(%s)" % joined
def check_memory(memory):

This comment has been minimized.

@glemaitre

glemaitre Aug 21, 2017

Contributor

You need to test this functions in test_validation.py. Create the DummyMemory and WrongDummyMemory classes there.

@glemaitre

glemaitre Aug 21, 2017

Contributor

You need to test this functions in test_validation.py. Create the DummyMemory and WrongDummyMemory classes there.

Show outdated Hide outdated sklearn/cluster/tests/test_hierarchical.py Outdated
Show outdated Hide outdated sklearn/tests/test_pipeline.py Outdated
Show outdated Hide outdated sklearn/tests/test_pipeline.py Outdated
@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman

jnothman Aug 21, 2017

Member

I am waiting for a review. I think it's complete but not sure ;) What do you think?

WIP means: Hello everyone this is a snippet of what I'm working on. WDYT?

MRG means: I've done what I'm going to do to make this mergeable. Now I need review.

Member

jnothman commented Aug 21, 2017

I am waiting for a review. I think it's complete but not sure ;) What do you think?

WIP means: Hello everyone this is a snippet of what I'm working on. WDYT?

MRG means: I've done what I'm going to do to make this mergeable. Now I need review.

Show outdated Hide outdated sklearn/utils/validation.py Outdated
Show outdated Hide outdated sklearn/utils/validation.py Outdated
Show outdated Hide outdated sklearn/utils/validation.py Outdated
@glemaitre

LGTM, only nitpick ... @lesteve I would almost do them if I would have push rights :)

Show outdated Hide outdated sklearn/cluster/hierarchical.py Outdated
Show outdated Hide outdated sklearn/utils/validation.py Outdated
Show outdated Hide outdated sklearn/utils/validation.py Outdated
@thechargedneutron

This comment has been minimized.

Show comment
Hide comment
@thechargedneutron

thechargedneutron Aug 29, 2017

Contributor

@glemaitre Done! except the 2nd comment.

Contributor

thechargedneutron commented Aug 29, 2017

@glemaitre Done! except the 2nd comment.

@glemaitre

This comment has been minimized.

Show comment
Hide comment
@glemaitre

glemaitre Aug 29, 2017

Contributor

@glemaitre Done! except the 2nd comment.

Great. Was it a review to remove them previously. If yes, I am fine for merging.

Contributor

glemaitre commented Aug 29, 2017

@glemaitre Done! except the 2nd comment.

Great. Was it a review to remove them previously. If yes, I am fine for merging.

Show outdated Hide outdated sklearn/pipeline.py Outdated

@jnothman jnothman added this to the 0.19.1 milestone Aug 30, 2017

@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman

jnothman Aug 30, 2017

Member

I've marked this for inclusion in 0.19.1 because it may help a lot with the new feature in Pipeline.

Member

jnothman commented Aug 30, 2017

I've marked this for inclusion in 0.19.1 because it may help a lot with the new feature in Pipeline.

@thechargedneutron

This comment has been minimized.

Show comment
Hide comment
@thechargedneutron

thechargedneutron Aug 30, 2017

Contributor

@jnothman Changed the function description. Review please.

Contributor

thechargedneutron commented Aug 30, 2017

@jnothman Changed the function description. Review please.

Show outdated Hide outdated sklearn/utils/validation.py Outdated
@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman

jnothman Aug 30, 2017

Member

Fix that up and I'll merge on green.

Member

jnothman commented Aug 30, 2017

Fix that up and I'll merge on green.

thechargedneutron and others added some commits Aug 30, 2017

@jnothman jnothman merged commit e66aa6d into scikit-learn:master Aug 30, 2017

0 of 4 checks passed

ci/circleci CircleCI is running your tests
Details
continuous-integration/appveyor/pr Waiting for AppVeyor build to complete
Details
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
lgtm analysis: Python Running analyses for revisions
Details
@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman
Member

jnothman commented Aug 30, 2017

jnothman added a commit to jnothman/scikit-learn that referenced this pull request Aug 30, 2017

@thechargedneutron

This comment has been minimized.

Show comment
Hide comment
@thechargedneutron

thechargedneutron Aug 30, 2017

Contributor

@jnothman welcome!! Thanks for the help :)

Contributor

thechargedneutron commented Aug 30, 2017

@jnothman welcome!! Thanks for the help :)

@lesteve

This comment has been minimized.

Show comment
Hide comment
@lesteve

lesteve Aug 30, 2017

Member

I am not sure why we added string on top of joblib.Memory-like explicitly to the type info. As per the check_memory docstring:

joblib.Memory-like means that memory can be converted into a
sklearn.externals.joblib.Memory instance (typically a str denoting the
cachedir) or has the same interface (has a cache method).

In my mind, joblib.Memory-like was supposed to be modelled after array-like in numpy. You don't see this in numpy docstrings:

parameter : array_like or list

I would agree that being explicit and adding string makes it easier to use for user not familiar with joblib.Memory (in general the docstring does say what happens if a string is passed in though). I was hoping that just using joblib.Memory-like in the type info and either adding a link to check_memory function for more details or copy and pasting the same docstring around would work.

Member

lesteve commented Aug 30, 2017

I am not sure why we added string on top of joblib.Memory-like explicitly to the type info. As per the check_memory docstring:

joblib.Memory-like means that memory can be converted into a
sklearn.externals.joblib.Memory instance (typically a str denoting the
cachedir) or has the same interface (has a cache method).

In my mind, joblib.Memory-like was supposed to be modelled after array-like in numpy. You don't see this in numpy docstrings:

parameter : array_like or list

I would agree that being explicit and adding string makes it easier to use for user not familiar with joblib.Memory (in general the docstring does say what happens if a string is passed in though). I was hoping that just using joblib.Memory-like in the type info and either adding a link to check_memory function for more details or copy and pasting the same docstring around would work.

@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman

jnothman Aug 30, 2017

Member
Member

jnothman commented Aug 30, 2017

@lesteve

This comment has been minimized.

Show comment
Hide comment
@lesteve

lesteve Aug 30, 2017

Member

Well, we can debate it, but I strongly disagree that "joblib.Memory-like" is sufficiently helpful to a user who's never heard of joblib.

Makes sense but then I would change the wording in the check_memory docstring, probably so that joblib.Memory-like means "same interface as joblib.Memory".

Member

lesteve commented Aug 30, 2017

Well, we can debate it, but I strongly disagree that "joblib.Memory-like" is sufficiently helpful to a user who's never heard of joblib.

Makes sense but then I would change the wording in the check_memory docstring, probably so that joblib.Memory-like means "same interface as joblib.Memory".

@glemaitre

This comment has been minimized.

Show comment
Hide comment
@glemaitre

glemaitre Aug 30, 2017

Contributor
Contributor

glemaitre commented Aug 30, 2017

@thechargedneutron thechargedneutron deleted the thechargedneutron:third branch Aug 30, 2017

maskani-moh added a commit to maskani-moh/scikit-learn that referenced this pull request Nov 15, 2017

jwjohnson314 pushed a commit to jwjohnson314/scikit-learn that referenced this pull request Dec 18, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment