Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test to ensure binary pipeline maintains threshold after pickling #3027

Merged
merged 6 commits into from
Nov 9, 2021

Conversation

bchen1116
Copy link
Contributor

Testing thresholds

@bchen1116 bchen1116 self-assigned this Nov 9, 2021
@codecov
Copy link

codecov bot commented Nov 9, 2021

Codecov Report

Merging #3027 (108fdfb) into main (3348464) will increase coverage by 0.1%.
The diff coverage is 100.0%.

Impacted file tree graph

@@           Coverage Diff           @@
##            main   #3027     +/-   ##
=======================================
+ Coverage   99.8%   99.8%   +0.1%     
=======================================
  Files        312     312             
  Lines      30162   30183     +21     
=======================================
+ Hits       30073   30094     +21     
  Misses        89      89             
Impacted Files Coverage Δ
evalml/tests/pipeline_tests/test_pipelines.py 99.8% <100.0%> (+0.1%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3348464...108fdfb. Read the comment docs.

@@ -2877,3 +2878,31 @@ def test_component_graph_pipeline_initialized():
]
== "median"
)


def test_pickled_pipeline_preserves_threshold(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add some of this logic to test_serialization_protocol or test_load_pickled_pipeline_with_custom_objective instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@freddyaboulton both of those tests seem to use cloudpickle, but I wanted to use pickle. Would it be better to edit those tests to include my pickle test, or leave the test as is?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, I see! I wonder if it's possible to merge the tests. We could have a different test fixture than pickled_pipeline_path that uses pickle instead of cloudpickle and move some of the logic from this test in to test_load_pickled_pipeline_with_custom_objective.

Up to you though. If it's better to keep the tests separate, I'm down for it but maybe let's move this test next to test_load_pickled_pipeline_with_custom_objective so that the pickling tests are at least grouped.

Copy link
Contributor

@freddyaboulton freddyaboulton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me @bchen1116 ! Left a non-blocking suggestion for how to organize these tests.

@@ -2877,3 +2878,31 @@ def test_component_graph_pipeline_initialized():
]
== "median"
)


def test_pickled_pipeline_preserves_threshold(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, I see! I wonder if it's possible to merge the tests. We could have a different test fixture than pickled_pipeline_path that uses pickle instead of cloudpickle and move some of the logic from this test in to test_load_pickled_pipeline_with_custom_objective.

Up to you though. If it's better to keep the tests separate, I'm down for it but maybe let's move this test next to test_load_pickled_pipeline_with_custom_objective so that the pickling tests are at least grouped.

@bchen1116 bchen1116 merged commit 42dec60 into main Nov 9, 2021
@freddyaboulton freddyaboulton deleted the bc_pickle_threshold branch May 13, 2022 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants