Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[snorkel] Remove warning (or change to info) #949

Closed
dvsrepo opened this issue Jan 13, 2022 · 3 comments · Fixed by #952 or #1071
Closed

[snorkel] Remove warning (or change to info) #949

dvsrepo opened this issue Jan 13, 2022 · 3 comments · Fixed by #952 or #1071
Assignees
@dcfidalgo
Projects
Release

Comments

@dvsrepo
Copy link
Member

dvsrepo commented Jan 13, 2022

Having always this red message when scoring the model is not desirable:

WARNING:rubrix.labeling.text_classification.label_models:Metrics are only calculated over non-abstained predictions!
{'accuracy': 0.5953488372093023}

I'd either remove it or change to info level.

Also is it not possible to provide more detailed metrics (precision, recall, fscore)?
@frascuchon frascuchon added this to To do in Release via automation Jan 13, 2022
@dcfidalgo
Copy link
Contributor

I will remove it, it's an artifact of how snorkel does it. I will move it to the docstring, should be enough.

Regarding the more detailed metrics, definitely possible, I will open a separate issue for that. It

@dvsrepo
Copy link
Member Author

dvsrepo commented Jan 13, 2022

💯

This was referenced Jan 13, 2022
Merged
Closed
Release automation moved this from To do to Done Jan 13, 2022
@frascuchon frascuchon moved this from Done to Release Ready in Release Jan 31, 2022
@frascuchon
Copy link
Member

frascuchon commented Jan 31, 2022 •
edited

Documentation does not show the include NOTE reference in docstring

https://rubrix.readthedocs.io/en/master/reference/python/python_labeling.html#rubrix.labeling.text_classification.label_models.Snorkel.score

Screenshot 2022-02-01 at 11 05 30

@frascuchon frascuchon reopened this Jan 31, 2022
Release automation moved this from Release Ready to In progress Jan 31, 2022
@dcfidalgo dcfidalgo mentioned this issue Feb 1, 2022
Merged
Release automation moved this from In progress to Done Feb 1, 2022
frascuchon pushed a commit that referenced this issue Feb 1, 2022
docs(#949): change note to admonition (#1071)
88e3678
@frascuchon frascuchon moved this from Done to Dev deployment in Release Feb 2, 2022
@frascuchon frascuchon moved this from Dev deployment to Release Ready in Release Feb 2, 2022
frascuchon pushed a commit that referenced this issue Feb 2, 2022
@frascuchon
docs(#949): change note to admonition (#1071)
067fad7 
(cherry picked from commit 88e3678)
@frascuchon frascuchon moved this from Release Ready to Closed in Release Feb 2, 2022
dvsrepo added a commit that referenced this issue Feb 10, 2022
@dvsrepo
Merge branch 'master' of https://github.com/recognai/rubrix
c8bbcee 
* 'master' of https://github.com/recognai/rubrix: (33 commits)
  fix(#1045): fix overall precision (#1087)
  fix(#1081): prevent add records of different task (#1085)
  fix(#1045): calculate overall precision from overall correct/incorrect in rules (#1086)
  fix(#924): parse new error format in UI (#1082)
  fix(#1054): Optimize Long records (#1080)
  docs(#949): change note to admonition (#1071)
  fix(#1053): metadata modal position   (#1068)
  fix(#1067): fix rule definition link when no labels are defined (#1069)
  fix(#1065): 'B' tag for beginning tokens (#1066)
  feat(#1054):  optimize long records view (#1064)
  feat(#924): parse validation error, including submitted information (#1056)
  fix(#1058): sort by % data in rules list (#1062)
  fix(#1050): generalizes entity span validation (#1055)
  fix: missing Optional import
  fix(cleanlab): set cleanlab n_jobs=1 as default (#1059)
  feat(#982): Show filters in labelling rules view (#1038)
  feat(#932): label models now modify the prediction_agent when calling LabelModel.predict (#1049)
  fix(#821): Token classifier QA 2 (#1057)
  ci: fix path filter condition
  refactor(#924): normalize API error responses (#1031)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dcfidalgo dcfidalgo

Labels
None yet
Projects
No open projects
Release
Approved Release QA
Milestone
No milestone
3 participants
@dvsrepo @frascuchon @dcfidalgo