CI job to check that all test dirs are included in package #1054
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1054 +/- ##
=======================================
Coverage 99.91% 99.91%
=======================================
Files 187 188 +1
Lines 10278 10286 +8
=======================================
+ Hits 10269 10277 +8
Misses 9 9
Continue to review full report at Codecov.
|
freddyaboulton
requested review from
dsherry,
jeremyliweishih,
angela97lin and
eccabay
There was a problem hiding this comment.
Looks good! After this you may need to go in the repo settings and make sure this test is required moving forward. You can find it here under required status checks.
|
Thanks for the heads-up @jeremyliweishih ! I'll be sure to modify the settings. |
.circleci/config.yml
Outdated
| - install_dependencies_dev | ||
| - run: | | ||
| source test_python/bin/activate | ||
| make check-all-test-dirs-included |
There was a problem hiding this comment.
Nice, circleci changes look good
.check_all_test_dirs_included.py
Outdated
| test_dir = "evalml/tests" | ||
| all_test_dirs_with_init_files = [module for module in all_modules if "evalml.tests" in module] | ||
| all_test_dirs = [dirname for dirname, _, files in os.walk(test_dir) if "__pycache__" not in dirname and "test" in os.path.split(dirname)[1]] | ||
| assert len(all_test_dirs) == len(all_test_dirs_with_init_files), "Not every test directory in evalml/tests has an __init__.py file!" |
There was a problem hiding this comment.
@freddyaboulton is there any reason you can't use pytest?
I also threw in pathlib.Path because why not make the pathing platform-agnostic 😆
from setuptools import find_packages
import os
from pathlib import Path
import pytest
def test_all_test_dirs_included():
all_modules = find_packages()
test_dir = Path("evalml", "tests")
... # etcThere was a problem hiding this comment.
Ah, and if you do this, I think there's no need for any changes to the CI! No need to add a new job. Because this unit test will run alongside all the others.
Makefile
Outdated
| @@ -43,6 +43,10 @@ installdeps-test: | |||
| installdeps-dev: | |||
| pip install -r dev-requirements.txt -q | |||
|
|
|||
| .PHONY: check-all-test-dirs-included | |||
| check-all-test-dirs-included: | |||
| python .check_all_test_dirs_included.py | |||
There was a problem hiding this comment.
I suggest we move this to evalml/tests/check_all_test_dirs_included.py
| test_dir = os.path.dirname(__file__) | ||
| all_test_dirs_with_init_files = [module for module in all_modules if "evalml.tests" in module] | ||
| all_test_dirs = [dirname for dirname, _, files in os.walk(test_dir) if "__pycache__" not in dirname and "test" in os.path.split(dirname)[1]] | ||
| assert len(all_test_dirs) == len(all_test_dirs_with_init_files) |
|
@angela97lin That's a good question! I was hoping you wouldn't be able to import a file in a module that didn't have an init but that's not the case (#1057). I think since we haven't had issues with that problem before and since that's outside the scope of this issue I think it's ok as it is now? |
|
@dsherry Thanks for the feedback. You're right, much simpler as a unit test! |
freddyaboulton
deleted the
1031-check-all-test-files-included-in-package
branch
Pull Request Description
Closes #1031 . The new job should is called
check_all_test_dirs_included. I tested it locally and by creating two dummy PRs and seeing if the CI job passes/fails as it should.#1055 should fail the new check because the new test directory does not have an init file.
#1056 should pass the new check.
After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of
docs/source/release_notes.rstto include this pull request by adding :pr:123.