Skip to content

Conversation

@crawlingcub
Copy link
Contributor

@crawlingcub crawlingcub commented Oct 1, 2021

This is a fix for another set of flaky tests: test_multi_inp_shapley_batch_scalar_tensor_0d, test_multi_inp_shapley_batch_scalar_float and test_multi_inp_shapley_batch_scalar_tensor_1d in test_shapley.py. Note that test_multi_inp_shapley_batch_scalar_int was not flaky to begin with.

I observed that increasing n_samples up to 3500 reduces flakiness by a lot. This increases the runtime from about 1.7s to 4.5s though. The tests only failed about 6/7 times out of 1000 runs with only the n_samples change. Hence, I also increased the delta to 1.2 based on the observed failures to bring that to ~0%. Let me know in case you would prefer lower n_samples but higher delta to reduce the runtime.

Thanks!

cc @vivekmig

@vivekmig
Copy link
Contributor

Looks great, thanks @crawlingcub! I think the runtime increase should be fine.

@facebook-github-bot
Copy link
Contributor

@vivekmig has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants