DEPS: Sync environment.yml with CI dep files

[mroeschke]

• closes CI: dev requirements version inconsistencies #45178 (Replace xxxx with the Github issue number)
• closes CI: Incompatible versions in resolved dependencies #43804 (Replace xxxx with the Github issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.

Also add a GHA job to check that the generated requirements-dev.txt file, synced from environment.yml, can be installed.

[mroeschke]

@twoertwein any idea why adding some dependencies to environment.yml would uncover these typing errors?
https://github.com/pandas-dev/pandas/runs/6803850160?check_suite_focus=true

[twoertwein]

The reportMissingImports from pyright indicate that the PyQt package is no longer installed. But many of the mypy errors look not related to PyQt: is it using a newer version of numpy (locally I'm using numpy 1.22.4)?

[twoertwein]

This could cause installing an older numpy version which could(?) explain most of the errors (but not the pyqt stuff).

[twoertwein]

As there is one dedicated CI run for numpy-dev, it would make sense to use the latest numpy compatible with numba (even for typing). Reverting #45244 would probably fix most of the numpy errors.

[mroeschke]

This file should only affect our specific PY 3.10 build which just runs the unit tests though. The typing checks should have an environment that is set up by environment.yml

[twoertwein]

Sorry, I meant that since we anyways run the unit tests with NumPy-dev in a separate workflow, prioritizing the latest numba version over the latest (released) NumPy version (in environment.yml) could be fine. Either way, it would be good to limit the numba version or the numpy version in environment.yml.

[simonjayhawkins]

As there is one dedicated CI run for numpy-dev, it would make sense to use the latest numpy compatible with numba (even for typing). Reverting #45244 would probably fix most of the numpy errors.

The problem I have with this is that new contribritors when setting up an environment will get the latest numpy and have mypy errors by default.

We should make the contributor experience pain free so (imo) we should use environment.yaml for the typing validation to match the local dev env .

Otherwise, this just makes it difficult for people to contribute to the typing issues.

Now, numba is included in environment.yaml so I'm not sure why when I set up a clean dev locally I get numpy 1.23.1 and on ci we get 1.22.4 (maybe there is some caching on ci?)

My comments here are from looking into this a couple of weeks ago. So this comment here now maybe out of date. Will look again soon.

[twoertwein]

Now, numba is included in environment.yaml so I'm not sure why when I set up a clean dev locally I get numpy 1.23.1 and on ci we get 1.22.4 (maybe there is some caching on ci?)

I must admit that I don't use the official way to setup a pandas-dev env, but it would be great to ensure that the officially documented pandas-env does not cause mypy errors.

Maybe numba has different numpy-constraints on conda-forge (or conda installs incompatible versions)? When I ask poetry to install numba = ">=0.53.1" (as in environment.yml) and numpy = ">=1.23.0", it is unable to find a solution (at least not on Linux with python 3.10).

[simonjayhawkins]

I must admit that I don't use the official way to setup a pandas-dev env, but it would be great to ensure that the officially documented pandas-env does not cause mypy errors.

yes I need to double check that's still true.

[twoertwein]

If pyqt has no type annotations (not sure whether that is the case), there is not much value in raising errors: could add # pyright: reportMissingImports = false at the top of pandas/io/clipboard/__init__.py

[twoertwein]

If pyqt has no type annotations (not sure whether that is the case), there is not much value in raising errors: could add # pyright: reportMissingImports = false at the top of pandas/io/clipboard/__init__.py

I think pandas/io/clipboard/__init__.py is copied from Pyperclip: and already excluded from the "main" pyright run but not from the pyright reportGeneralTypeIssues run.

Would be good to add the excludes from

pandas/pyproject.toml

Line 160 in 297c59a

exclude = ["pandas/tests", "pandas/io/clipboard", "pandas/util/version"]

here

pandas/pyright_reportGeneralTypeIssues.json

Line 13 in 297c59a

"pandas/tests",

something like

        # exclude tests
        "pandas/tests",
        # exclude vendored files
        "pandas/io/clipboard",
        "pandas/util/version",
        # and all files that currently don't pass
        ...

Happy to open a PR or you can include these changes in your PR.

[mroeschke]

@twoertwein would appreciate another look at the tying changes

[twoertwein]

@twoertwein would appreciate another look at the tying changes

The ignore comments look good to me (just a bit annoying that we need to undo those changes when numba supports numpy 1.22.4).

It sounds reasonable to me to use the latest numba-compatible numpy and numpy-dev for testing but technically, there could be some weird behavior with the newest numpy that is already fixed in numpy-dev.

[jreback]

lgtm. (prob need to manually backport).

[jreback]

@meeseeksdev backport 1.4.x

[simonjayhawkins]

There are typing failures on main from this commit.

[twoertwein]

There are typing failures on main from this commit.

It seems that the CI installed numpy 1.22.4 and numba 0.55.2. For some reason, they did not happily co-exist before (even though neither of them had a new release recently).