Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create an experiment to show the effects of learning p_non_comp to precision and recall #3

Open
meneguzzi opened this issue Feb 21, 2017 · 0 comments
Assignees

Comments

@meneguzzi
Copy link
Contributor

meneguzzi commented Feb 21, 2017

Answering @scranefield question on how to test stuff, we need to decide:

Felipe which is the best experiment to run to evaluate the effects of learning p_non_comp on norm learning?

My hypothesis is that if we get p_non_comp wrong, it depends on the value of p_sanc, I can think of six cases:

  1. p_sanc high, p_non_comp correct: ideal case, we should get high high-precision, high-recall
  2. p_sanc low, p_non_comp correct: we may overestimate some norms, so recall should be OK
  3. p_sanc high, p_non_comp low: we may get low recall (because we won't see much non-compliant behaviour)
  4. p_sanc low, p_non_comp low: we may get low recall (because we won't see much non-compliant behaviour), and low precision because the ones we do get may not get sanctioned
  5. p_sanc high, p_non_comp high: we should get low precision because we will mistakenly think a lot of the generated behaviour is non-compliant, even when we don't see sanctioning
  6. p_sanc low, p_non_comp high: we may get really high low recall (because we will see non-compliant behaviour), but low precision because we will mistakenly think a lot of the correct behaviour is non-compliant.
@meneguzzi meneguzzi self-assigned this Feb 21, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant