Created new per class accuracy function. #624

deepandas11 · 2019-11-08T12:44:18Z

Features:

new pos_label argument(got to change name, as there could be ambiguity
with positive_label in the original scoring function). Is None by
default, in which case, average accuracy is returned
uses existing _accuracy function for computing per class accuracy

Description

This PR intends to introduce a standalone per class accuracy function

Related issues or pull requests

Fixes #617

Pull Request Checklist

Added a note about the modification or contribution to the ./docs/sources/CHANGELOG.md file (if applicable)
Added appropriate unit test functions in the ./mlxtend/*/tests directories (if applicable)
Modify documentation in the corresponding Jupyter Notebook under mlxtend/docs/sources/ (if applicable)
Ran PYTHONPATH='.' pytest ./mlxtend -sv and make sure that all unit tests pass (for small modifications, it might be sufficient to only run the specific test file, e.g., PYTHONPATH='.' pytest ./mlxtend/classifier/tests/test_stacking_cv_classifier.py -sv)
Checked for style issues by running flake8 ./mlxtend

Features: * new pos_label argument(got to change name, as there could be ambiguity with positive_label in the original scoring function). Is None by default, in which case, average accuracy is returned * uses existing _accuracy function for computing per class accuracy

pep8speaks · 2019-11-08T12:44:21Z

Hello @deepandas11! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-11-22 19:01:42 UTC

coveralls · 2019-11-08T12:54:19Z

Coverage increased (+0.006%) to 92.354% when pulling d814e0d on deepandas11:per_class_accuracy into 9dc73ed on rasbt:master.

rasbt · 2019-11-08T14:40:02Z

mlxtend/evaluate/scoring.py

@@ -111,10 +120,10 @@ def scoring(y_target, y_predicted, metric='error',
    elif metric == 'error':
        res = _error(targ_tmp, pred_tmp)
    elif metric == 'average per-class accuracy':


I think there was a misunderstanding. The per-class accuracy would not be used to replace the average-per-class-accuracy. Maybe have a look at the brief comment in #615

Yes! This discussion makes it much clearer now. Thanks!

This reverts commit 4ee5d6a.

deepandas11 · 2019-11-13T10:35:20Z

Hello, @rasbt !
Please let me know if there are any corrections/recommendations!

rasbt · 2019-11-13T16:31:01Z

mlxtend/evaluate/per_class_accuracy.py

+    pos_label : str or int, None by default.
+        The class whose accuracy score is to be reported.
+        'binary' has to be set to True.
+    binary : bool


What happens if binary is set to False. Will this be a regular accuracy computation then? I think it looks like that. It would be nice to document that.

I have another suggestion, though. How about

calling this function just "accuracy" and computing the "average" per-class accuracy as an option?

E.g., instead of binary, we could use another parameter called method.

method='standard' computes the regular accuracy

method='binary' computes per-class accuracy for the positive class

method='average' computes average per class accuracy similar to https://github.com/rasbt/mlxtend/blob/master/mlxtend/evaluate/scoring.py#L113

What do you think?

Yes, that should have been documented. Thanks for pointing that out.

As for the suggestion, I think if you are looking to introduce modularity with respect to the individual metrics in "scoring.py", this would be a good first step.
However, it would be interesting to know if you indeed want to eventually deprecate the current methodology of finding metrics and move towards standalone functions, like this?

I think it would be good to support both, like scikit-learn does. The 'string' arguments in scoring are useful as a short-cut. However, it doesn't allow options like pos_label. Inside scoring, the idea would then be to use the per_class_accuracy function to implement the current 'average per-class accuracy'

So, implementation-wise, I think the API would be less confusing if we have just a single accuracy function that can compute the three types of metrics, based on the choice of "method". This however, means no separate per_class_accuracy method, as was originally intended.

I agree. I think it's best if we call it just classification_accuracy? And then inside we have the "per class" option

i have to correct this, "accuracy_score" may be better because we also have lift_score already, and it is generally more consistent with sklearn then.

rasbt · 2019-11-13T16:32:11Z

Sorry, it was a busy week and just got a chance to look at it. I have a suggestion (mention in the comment above). Let me know what you think. Also, it's weird that the unit test for Windows fails. Not sure if this is necessarily related to this PR or a general issue. Will investigate.

Features: * new pos_label argument(got to change name, as there could be ambiguity with positive_label in the original scoring function). Is None by default, in which case, average accuracy is returned * uses existing _accuracy function for computing per class accuracy

This reverts commit 4ee5d6a.

rasbt · 2019-11-13T22:55:45Z

Just did a rebase of your changes on top of the master branch to resolve the CI issues. You would need to do a

git pull origin per_class_accuracy

on your end before making any additional changes.

…lxtend into per_class_accuracy

deepandas11 · 2019-11-14T00:34:33Z

Thank you so much for resolving the CI issue.
No worries about the delay.

Function capable of finding different kinds of accuracy metrics in three different 'method' modes.

deepandas11 · 2019-11-14T14:56:34Z

So, I have made the changes as discussed.
Another feature we can now possibly add to the accuracy scores is the normalize feature like in sklearn. (But maybe in a separate PR. I can open an issue if you feel we should add it)

rasbt

Thanks for making the changes. It looks good to me overall. Just have a few minor comments, and I think considering adding normalize already might be a good idea.

I can take care of refactoring scoring.py then to use the new accuracy_score there internally, so no worries about that.
I will also make sure the documentation builds correctly then.

mlxtend/evaluate/accuracy.py

deepandas11 · 2019-11-15T00:17:44Z

Have made the changes. Although, I think you have already seen that I made a few changes to scoring.py in aa095d7 to accommodate the new accuracy function.

rasbt · 2019-11-15T05:12:17Z

Thanks a lot! I am currently a bit swamped but will try to take a look at it on the weekend!

rasbt

Looks pretty good! Here are a few minor issues I found.

mlxtend/evaluate/accuracy.py

rasbt · 2019-11-22T19:02:41Z

Thanks for making those changes. Looks good to me now. I added the documentation, let me know if that looks good to you.

deepandas11 · 2019-11-24T19:49:38Z

Looks great.

rasbt · 2019-11-24T19:55:33Z

Cool thanks! I will merge then!

deepandas11 changed the title ~~Created new per class accuracy function.~~ [WIP] Created new per class accuracy function. Nov 8, 2019

rasbt reviewed Nov 8, 2019

View reviewed changes

Deepan Das added 4 commits November 8, 2019 22:19

Revert "Created new per class accuracy function."

10e1183

This reverts commit 4ee5d6a.

Adding separate per_class_accuracy function

4319203

Unit Tests for per class accuracy

617c8cd

Enabling module import

4982353

deepandas11 changed the title ~~[WIP] Created new per class accuracy function.~~ Created new per class accuracy function. Nov 12, 2019

rasbt reviewed Nov 13, 2019

View reviewed changes

Deepan Das added 5 commits November 13, 2019 16:53

Revert "Created new per class accuracy function."

bb024db

This reverts commit 4ee5d6a.

Adding separate per_class_accuracy function

f30fdb3

Unit Tests for per class accuracy

75fb5e3

Enabling module import

d8abfcd

rasbt force-pushed the per_class_accuracy branch from 4982353 to d8abfcd Compare November 13, 2019 22:54

Merge branch 'per_class_accuracy' of https://github.com/deepandas11/m…

71cbeba

…lxtend into per_class_accuracy

Deepan Das added 7 commits November 14, 2019 02:38

Renamed per_class_accuracy to accuracy

df7ccb2

Generalization of per_class_accuracy to accuracy

8be486b

Function capable of finding different kinds of accuracy metrics in three different 'method' modes.

Remove private _accuracy function to make way for new accuracy function

aa09fd7

Renaming tests for old per class accuracy to accuracy

cecf485

Enabling module import

3f80402

Added tests for new accuracy function

11234e0

modified test assertion

6215117

rasbt reviewed Nov 14, 2019

View reviewed changes

mlxtend/evaluate/accuracy.py Outdated Show resolved Hide resolved

mlxtend/evaluate/accuracy.py Show resolved Hide resolved

mlxtend/evaluate/accuracy.py Outdated Show resolved Hide resolved

Deepan Das added 3 commits November 15, 2019 00:46

refactoring and adding normalization

f4e0fdd

adding non-normalized tests

8ddcb72

fixing test issues

2133c7f

rasbt approved these changes Nov 17, 2019

View reviewed changes

mlxtend/evaluate/accuracy.py Outdated Show resolved Hide resolved

mlxtend/evaluate/accuracy.py Outdated Show resolved Hide resolved

Deepan Das and others added 2 commits November 18, 2019 00:08

Resolved issues

de72221

accuracy_score docs

d814e0d

rasbt merged commit 9cb5f6c into rasbt:master Nov 24, 2019

rasbt mentioned this pull request Dec 1, 2019

per-class accuracy #615

Closed

deepandas11 deleted the per_class_accuracy branch December 1, 2019 18:03

rasbt mentioned this pull request Jan 29, 2020

v0.17.1 #660

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Created new per class accuracy function. #624

Created new per class accuracy function. #624

deepandas11 commented Nov 8, 2019 •

edited by rasbt

Loading

pep8speaks commented Nov 8, 2019 •

edited

Loading

coveralls commented Nov 8, 2019 •

edited

Loading

rasbt Nov 8, 2019

deepandas11 Nov 8, 2019 •

edited

Loading

deepandas11 commented Nov 13, 2019 •

edited

Loading

rasbt Nov 13, 2019

deepandas11 Nov 14, 2019

rasbt Nov 14, 2019

deepandas11 Nov 14, 2019

rasbt Nov 14, 2019

rasbt Nov 14, 2019

rasbt commented Nov 13, 2019

rasbt commented Nov 13, 2019

deepandas11 commented Nov 14, 2019

deepandas11 commented Nov 14, 2019

rasbt left a comment

deepandas11 commented Nov 15, 2019 •

edited

Loading

rasbt commented Nov 15, 2019

rasbt left a comment

rasbt commented Nov 22, 2019

deepandas11 commented Nov 24, 2019

rasbt commented Nov 24, 2019

Created new per class accuracy function. #624

Created new per class accuracy function. #624

Conversation

deepandas11 commented Nov 8, 2019 • edited by rasbt Loading

Description

Related issues or pull requests

Pull Request Checklist

pep8speaks commented Nov 8, 2019 • edited Loading

Comment last updated at 2019-11-22 19:01:42 UTC

coveralls commented Nov 8, 2019 • edited Loading

rasbt Nov 8, 2019

Choose a reason for hiding this comment

deepandas11 Nov 8, 2019 • edited Loading

Choose a reason for hiding this comment

deepandas11 commented Nov 13, 2019 • edited Loading

rasbt Nov 13, 2019

Choose a reason for hiding this comment

deepandas11 Nov 14, 2019

Choose a reason for hiding this comment

rasbt Nov 14, 2019

Choose a reason for hiding this comment

deepandas11 Nov 14, 2019

Choose a reason for hiding this comment

rasbt Nov 14, 2019

Choose a reason for hiding this comment

rasbt Nov 14, 2019

Choose a reason for hiding this comment

rasbt commented Nov 13, 2019

rasbt commented Nov 13, 2019

deepandas11 commented Nov 14, 2019

deepandas11 commented Nov 14, 2019

rasbt left a comment

Choose a reason for hiding this comment

deepandas11 commented Nov 15, 2019 • edited Loading

rasbt commented Nov 15, 2019

rasbt left a comment

Choose a reason for hiding this comment

rasbt commented Nov 22, 2019

deepandas11 commented Nov 24, 2019

rasbt commented Nov 24, 2019

deepandas11 commented Nov 8, 2019 •

edited by rasbt

Loading

pep8speaks commented Nov 8, 2019 •

edited

Loading

coveralls commented Nov 8, 2019 •

edited

Loading

deepandas11 Nov 8, 2019 •

edited

Loading

deepandas11 commented Nov 13, 2019 •

edited

Loading

deepandas11 commented Nov 15, 2019 •

edited

Loading