-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add testing for feat functions #44
Conversation
swanhl
commented
Nov 29, 2021
- Test on FEAT output correctness using a mocked up small dataset
- No test cases for perturbation & shapely feature tests, requires model input and results are not deterministic (involves randomness)
This pull request is being automatically deployed with Vercel (learn more). 🔍 Inspect: https://vercel.com/cylynx/verifyml/FWq7RtH8h6YHZS4qhrJ1uNuE3Rvh |
test_obj1.run(test_data) | ||
test_obj2.run(test_data) | ||
|
||
assert test_obj1.result.iloc[0].p_value == 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we change the example, to have a p-value < 1. Otherwise, these two test are not that meaningful. With two different thresholds, one should pass and the other fail.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both still have a p-value of 1
For Perturbation and SHAPFeatureImportance, I recommend creating a simple model as input using one of scikitlearn test datasets https://scikit-learn.org/stable/datasets/toy_dataset.html and assert on the results. |