Remove conda reqs and requirements-all.txt; simplify CI installs via setup.py extras#916
Merged
Merged
Conversation
Removes unmaintained/unenforced dependency-tracking artifacts and simplifies CI to use setup.py's existing extras directly. - Delete conda/ directory entirely (conda-reqs*.txt, README.md). These files were only referenced by tests/test_pkg_imports.py::test_conda_reqs and not used by any CI workflow, docs build, or pre-commit hook. - Delete requirements-all.txt. setup.py already defines EXTRAS['all'] (equivalent content) and EXTRAS['test'] (= 'all' + 'dev' extras combined, equivalent to requirements-all.txt + requirements-dev.txt). `pip install -e ".[test]"` is a drop-in replacement. - Remove 'include requirements-all.txt' from MANIFEST.in. - Update .github/workflows/python-package.yml (build + lint jobs) and azure-pipelines.yml to install via `pip install -e ".[test]"` instead of separate requirements-all.txt/requirements-dev.txt steps. Also fix a stray doc/requirements-all.txt cache-key typo in the docs job (that job only ever used docs/requirements.txt). - Update .devcontainer/Dockerfile to install msticpy[test] directly from GitHub instead of copying/installing requirements-all.txt and requirements-dev.txt separately. - Remove the check_reqs_all local pre-commit hook, which auto-rewrote requirements-all.txt (stripping hand-maintained security-vuln comments) on every commit touching a .py file. - Document tools/create_reqs_all.py as a manual, on-demand generator for anyone who wants a frozen requirements-all.txt locally (already supports --print/--diff/--req-all-path; no code changes needed beyond the docstring). - tests/test_pkg_imports.py: remove test_conda_reqs (the conda vs. setup.py 'package version consistency check'); keep and simplify test_missing_pkgs_req. - Fix a real bug in tools/toollib/import_analyzer.py::_check_std_lib: stdlib detection only checked sys.prefix, which is wrong for venvs (especially uv-managed ones where the physical stdlib lives under sys.base_prefix, outside the venv prefix). This caused ~60 stdlib modules to be misclassified as missing packages in test_missing_pkgs_req on this environment (verified: same failure pre-existed on main). - Fix tools/Module-Analysis.ipynb example cell referencing the now- removed requirements-all.txt. Verified: full pytest suite (MSTICPYCONFIG=tests/msticpyconfig-test.yaml) passes cleanly - 1463 passed, 68 skipped, 0 failures (previously 1 failure in test_missing_pkgs_req, now fixed). ruff/pylint/pydocstyle pass on all changed Python files. YAML syntax validated for python-package.yml, azure-pipelines.yml, and .pre-commit-config.yaml. Confirmed `pip install -e ".[test]"` resolves cleanly (255 packages) as a drop-in replacement for the old two-file install. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
CI was failing on this branch (and on `main` itself - confirmed via
recent CI history, so this is a pre-existing repo-wide break, not
something introduced by this PR):
1. build (all Python versions): azure-mgmt-resource 26.0.0 (released
2026-06-24) moved ResourceManagementClient from azure.mgmt.resource
to azure.mgmt.resource.resources, breaking
msticpy.context.azure.azure_data - and everything that imports it
(Sentinel context tests, config editors, etc). Capped
azure-mgmt-resource to <26.0.0 in setup.py's "azure" extra (this
branch no longer has conda-reqs.txt/requirements-all.txt to update,
since those were removed - setup.py is now the sole source of
truth).
2. lint (mypy): mypy 2.3.0 infers its syntax target `python_version`
from setup.cfg's `python_requires = >=3.10` when not told
otherwise, rather than the actual interpreter running it. This
causes a false positive on numpy 2.5.1's stub file, which uses a
PEP 695 `type` statement (Python 3.12+ only):
numpy/__init__.pyi:737: error: Type statement is only supported
in Python 3.12 and greater [syntax]
Fixed by passing `--python-version ${{ matrix.python-version }}`
explicitly to mypy in the lint job, so it always checks against the
actual running Python version.
Once unblocked, mypy surfaced 6 additional real (pre-existing, but
previously unreached because mypy aborted early) type errors,
fixed:
- 4x try/except `Version` re-import fallback pattern
(packaging.version.Version -> distutils.version.LooseVersion)
was tagged `# type: ignore[assignment]` but mypy now reports it
as `no-redef`; added `no-redef` to the ignore codes in
msticpy/init/nbinit.py, msticpy/init/azure_ml_tools.py,
msticpy/init/pivot_core/pivot_magic_core.py, and
msticpy/analysis/polling_detection.py.
- msticpy/analysis/eventcluster.py: `color = [0, 0, 0, 1]` assigned
a list[int] to a variable typed as tuple[float, float, float,
float] (an RGBA color tuple) - changed to `(0.0, 0.0, 0.0, 1.0)`.
- msticpy/vis/foliummap.py: `pygeohash = None` in the ImportError
fallback branch needed `# type: ignore[assignment]` since
`pygeohash` is otherwise typed as a module.
Verified: full pytest suite (MSTICPYCONFIG=tests/msticpyconfig-test.yaml)
passes - 1463 passed, 68 skipped, 0 failures. `mypy --python-version
3.12/3.13`, ruff, pydocstyle, and pylint (10.00/10) all clean on the
changed files.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Merges PR #915 (dependency bumps + azure-mgmt-resource/mypy CI fixes, now on main) into this branch. Resolved delete/modify conflicts on conda/conda-reqs*.txt and requirements-all.txt by keeping their removal (this branch's purpose).
Bumps msticpy/_version.py from 3.0.1 to 3.0.2 (patch release: security dependency bumps from #915, CI fixes, and this PR's conda/requirements-all cleanup - no breaking API changes). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes unmaintained/unenforced dependency-tracking artifacts and simplifies CI to use
setup.py's existing extras directly, addressing 5 related cleanups:conda/directory entirely (conda-reqs*.txt,README.md). These files were only referenced bytests/test_pkg_imports.py::test_conda_reqs— not used by any CI workflow, docs build, or pre-commit hook.requirements-all.txt.setup.pyalready definesEXTRAS["all"](equivalent content) andEXTRAS["test"](=all+devextras combined — equivalent torequirements-all.txt+requirements-dev.txt).pip install -e ".[test]"is a drop-in replacement, used to update:.github/workflows/python-package.yml(build + lint jobs) — also fixes a straydoc/requirements-all.txtcache-key typo in the docs job (unrelated to requirements-all.txt at all)azure-pipelines.yml.devcontainer/DockerfileMANIFEST.in(dropped theinclude requirements-all.txtline)tests/test_pkg_imports.py::test_missing_pkgs_req, removedtest_conda_reqs(the conda vs. setup.py "package version consistency check").tools/toollib/import_analyzer.py::_check_std_lib: stdlib detection only checkedsys.prefix, which is wrong for venvs (especiallyuv-managed ones where the physical stdlib lives undersys.base_prefix, outside the venv prefix). This caused ~60 stdlib modules to be misclassified as missing packages — confirmed the same failure pre-existed onmain.check_reqs_allpre-commit hook (it auto-rewroterequirements-all.txt— including stripping hand-maintained security-vuln comments — on every commit touching a.pyfile).tools/create_reqs_all.pyis kept and documented as a manual, on-demand generator for anyone who wants a frozenrequirements-all.txtlocally.Testing
MSTICPYCONFIG=tests/msticpyconfig-test.yaml): 1463 passed, 68 skipped, 0 failures (previously 1 failure intest_missing_pkgs_req, now fixed).ruff check,ruff format --check,pydocstyle,pylintall pass on changed Python files.python-package.yml,azure-pipelines.yml,.pre-commit-config.yaml.pip install -e ".[test]"resolves cleanly (255 packages) as a drop-in replacement for the old two-file install.