Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Inconsistent Test Outcome from TargetDistributionDataCheck #2608

Merged
merged 4 commits into from Aug 9, 2021

Conversation

ParthivNaresh
Copy link
Contributor

Fixes #2581

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

@codecov
Copy link

codecov bot commented Aug 9, 2021

Codecov Report

Merging #2608 (dcdad9a) into main (efcf5d1) will increase coverage by 0.1%.
The diff coverage is 100.0%.

Impacted file tree graph

@@           Coverage Diff           @@
##            main   #2608     +/-   ##
=======================================
+ Coverage   99.9%   99.9%   +0.1%     
=======================================
  Files        295     295             
  Lines      26847   26848      +1     
=======================================
+ Hits       26801   26802      +1     
  Misses        46      46             
Impacted Files Coverage Δ
...alml/data_checks/target_distribution_data_check.py 100.0% <100.0%> (ø)
...hecks_tests/test_target_distribution_data_check.py 100.0% <100.0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update efcf5d1...dcdad9a. Read the comment docs.

@@ -97,7 +97,7 @@ def validate(self, X, y):

if log_detected:
details = {
"shapiro-statistic/pvalue": f"{round(shapiro_test_og.statistic, 3)}/{round(shapiro_test_og.pvalue, 3)}"
"shapiro-statistic/pvalue": f"{round(shapiro_test_og.statistic, 2)}/{round(shapiro_test_og.pvalue, 3)}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the previous implementation, 4-5 out of 100 runs would result in slightly different values due to how rounding would occur after a certain number of decimal places. This would throw off the statistic to the thousandths place by 1, resulting in a failure. I'm rounding all values here to keep the results the same in the data check and the test block.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet. If the value of this ratio impacted whether or not we did the log transform, then I would have suggested we instead do the rounding in the test as opposed to the implementation. Since this is just to provide extra details to users, I think it's ok to round however we want!

else:
y = np.random.normal(4, 1, 100)
y = [9.2, 7.66, 4.93, 3.29, 4.06, -1.28, 4.95, 6.77, 9.07, 7.67]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both these test blocks would previously throw errors 7-8 times out of a 100 runs. Even after changing the values to be more gaussian in distribution with larger sample sizes, they still failed 1-2 times out of the 100,000 simulations I ran. Randomness leads to a somewhat lognormal distribution every once in a while. I decided to replace these with constant smaller samples of a lognormal distribution so we don't have to worry about inconsistency again.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed!

Copy link
Collaborator

@chukarsten chukarsten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@freddyaboulton freddyaboulton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @ParthivNaresh !!

@@ -30,14 +30,14 @@ def validate(self, X, y):
Example:
>>> from scipy.stats import lognorm
>>> X = None
>>> y = lognorm.rvs(0.1, size=1000)
>>> target_checdk = TargetDistributionDataCheck()
>>> y = [0.946, 0.972, 1.154, 0.954, 0.969, 1.222, 1.038, 0.999, 0.973, 0.897]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 agree that it's best to not rely on random vaues for this

@@ -97,7 +97,7 @@ def validate(self, X, y):

if log_detected:
details = {
"shapiro-statistic/pvalue": f"{round(shapiro_test_og.statistic, 3)}/{round(shapiro_test_og.pvalue, 3)}"
"shapiro-statistic/pvalue": f"{round(shapiro_test_og.statistic, 2)}/{round(shapiro_test_og.pvalue, 3)}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet. If the value of this ratio impacted whether or not we did the log transform, then I would have suggested we instead do the rounding in the test as opposed to the implementation. Since this is just to provide extra details to users, I think it's ok to round however we want!

else:
y = np.random.normal(4, 1, 100)
y = [9.2, 7.66, 4.93, 3.29, 4.06, -1.28, 4.95, 6.77, 9.07, 7.67]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed!

@ParthivNaresh ParthivNaresh merged commit 3a00d47 into main Aug 9, 2021
@chukarsten chukarsten mentioned this pull request Aug 12, 2021
@freddyaboulton freddyaboulton deleted the Fix-Inconsistent-Test-Outcome-Lognormal branch May 13, 2022 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flaky test in nightlies: target distribution data check
3 participants