Refactor tests to avoid using importorskip

[freddyaboulton]

Pull Request Description

Fixes #2922

---

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

[codecov]

Codecov Report

Merging #3126 (9fd8f5b) into main (30b2a6a) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #3126     +/-   ##
=======================================
+ Coverage   99.8%   99.8%   +0.1%     
=======================================
  Files        315     315             
  Lines      30684   30711     +27     
=======================================
+ Hits       30593   30620     +27     
  Misses        91      91             

Impacted Files	Coverage Δ
.../automl_tests/test_automl_search_classification.py	100.0% <100.0%> (ø)
...ests/automl_tests/test_automl_search_regression.py	100.0% <100.0%> (ø)
evalml/tests/automl_tests/test_automl_utils.py	100.0% <100.0%> (ø)
...lml/tests/automl_tests/test_iterative_algorithm.py	100.0% <100.0%> (ø)
...l/tests/automl_tests/test_pipeline_search_plots.py	100.0% <100.0%> (ø)
...alml/tests/component_tests/test_arima_regressor.py	100.0% <100.0%> (ø)
.../tests/component_tests/test_catboost_classifier.py	100.0% <100.0%> (ø)
...l/tests/component_tests/test_catboost_regressor.py	100.0% <100.0%> (ø)
...alml/tests/component_tests/test_lgbm_classifier.py	100.0% <100.0%> (ø)
...valml/tests/component_tests/test_lgbm_regressor.py	100.0% <100.0%> (ø)
... and 21 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30b2a6a...9fd8f5b. Read the comment docs.

[freddyaboulton]

Will prevent us from using importorskip in the future:

[angela97lin]

Really cool idea, thank you for adding this as well as fixing all of the current references to importorskip! 😁

[angela97lin]

Really awesome work @freddyaboulton! LGTM, I just had some nit-picky comments about:

1. The message for our noncore_dependency marker
2. Having to import in each test we want to use the module. I wonder if there's a way to create some fixture for the file to get around that, but I'm not sure?

Thanks for doing this and adding a lint script to prevent us from falling back to this pattern! 🙏

[angela97lin]

Nice! Is the reason why we mark these tests because they're slow to run? Do we want to mention instead that they contain dependencies that are not part of the core requirements instead?

[freddyaboulton]

This is what happens when you copy-paste from the pytest docs 😂 Will update the message.

[angela97lin]

Really cool idea, thank you for adding this as well as fixing all of the current references to importorskip! 😁

[angela97lin]

It's interesting that we don't test anything with the outputs 😅

[angela97lin]

Does this mean that if we want to use a non-core dependency in our tests, we now have to import it in every test that we want to use it in? 😮

I wonder if this will cause the tests to run longer, since we have to import the same modules over and over. Probably not a noticeable difference given our tests but it makes me wonder if there's a way to create a fixture or something the way that we do for our estimators, and then call that instead of importing: catboost = import_or_raise("catboost", error_msg=cb_error_msg)

[freddyaboulton]

Thanks for the suggestion @angela97lin ! At first I thought creating the fixtures happened before the test was skipped but it turns out that it's the opposite. Which I guess makes sense. I'm creating fixtures for the commonly imported non-core dependencies in our tests. Don't think it'll run any faster, test_graphs has multiple importorskip("plotly.graph_objects"), but I think this will be great for reducing code duplication.