Fix errors and add mypy to CI #2190

jhale · 2022-05-20T07:30:20Z

Some of the MetaClass stuff seems to be tricky to type, although my Python typing knowledge is admittedly limited.

jhale · 2022-05-20T07:32:20Z

Would close #1988 once merged.

…nto jhale/mypy-fixes-and-ci

jhale · 2022-05-20T08:12:33Z

The type checkers do not like constructs like this where we use a raise of TypeError to detect a C++ or Python object.

https://github.com/FEniCS/dolfinx/blob/main/python/dolfinx/fem/bcs.py#L59

dolfinx/fem/bcs.py:62: error: "Iterable[Union[Any, FunctionSpace]]" has no attribute "_cpp_object"

I would propose changing to e.g. isinstance?

jhale · 2022-05-20T15:44:45Z

I've made some progress:

Number of type errors in DOLFINx Python API is now down to 32 errors in 3 files. Most are related to our use of MetaClasses or Single Dispatch, which are obviously tricky areas with respect to typing. I would need some help fixing these.
mypy enabled non-blocking in CI.
Install py.typed files into DOLFINx Python so that user's IDEs actually pickup on type hints.
There are still many easy mistakes to fix in our demos, or, our DOLFINx API typing. This would need discussion in terms of a) do we want to open up what can be passed to the API? or b) Happy with the API typing, and fix the demos.

DOLFINx Python errors:

dolfinx/fem/assemble.py:170: error: Dispatch type "ndarray[Any, Any]" must be subtype of fallback function first argument "FormMetaClass"
dolfinx/fem/assemble.py:230: error: Too many arguments for "assemble_matrix"
dolfinx/fem/assemble.py:230: error: Argument 1 to "assemble_matrix" has incompatible type "MatrixCSRMetaClass"; expected "FormMetaClass"
dolfinx/fem/assemble.py:230: error: Argument 2 to "assemble_matrix" has incompatible type "FormMetaClass"; expected "Optional[List[DirichletBCMetaClass]]"
dolfinx/fem/assemble.py:230: error: Argument 3 to "assemble_matrix" has incompatible type "List[DirichletBCMetaClass]"; expected "float"
dolfinx/fem/assemble.py:235: error: Dispatch type "MatrixCSRMetaClass" must be subtype of fallback function first argument "FormMetaClass"
dolfinx/fem/forms.py:168: error: Missing return statement
dolfinx/fem/forms.py:185: error: Item "Iterable[FormMetaClass]" of "Union[FormMetaClass, Iterable[FormMetaClass]]" has no attribute "function_spaces"
dolfinx/fem/petsc.py:181: error: Too many arguments for "assemble_vector"
dolfinx/fem/petsc.py:203: error: Too many arguments for "assemble_vector"
dolfinx/fem/petsc.py:220: error: Too many arguments for "assemble_vector_nest"
dolfinx/fem/petsc.py:234: error: Too many arguments for "assemble_vector"
dolfinx/fem/petsc.py:256: error: Too many arguments for "assemble_vector_block"
dolfinx/fem/petsc.py:256: error: Argument 2 to "assemble_vector_block" has incompatible type "List[FormMetaClass]"; expected "List[List[FormMetaClass]]"
dolfinx/fem/petsc.py:256: error: Argument 3 to "assemble_vector_block" has incompatible type "List[List[FormMetaClass]]"; expected "List[DirichletBCMetaClass]"
dolfinx/fem/petsc.py:256: error: Argument 5 to "assemble_vector_block" has incompatible type "Optional[Any]"; expected "float"
dolfinx/fem/petsc.py:291: error: Argument 1 to "bcs_by_block" has incompatible type "Iterable[Optional[FunctionSpace]]"; expected "Iterable[FunctionSpace]"
dolfinx/fem/petsc.py:302: error: Argument 1 to "bcs_by_block" has incompatible type "Iterable[Optional[FunctionSpace]]"; expected "Iterable[FunctionSpace]"
dolfinx/fem/petsc.py:325: error: Too many arguments for "assemble_matrix"
dolfinx/fem/petsc.py:325: error: Argument 2 to "assemble_matrix" has incompatible type "FormMetaClass"; expected "List[DirichletBCMetaClass]"
dolfinx/fem/petsc.py:325: error: Argument 3 to "assemble_matrix" has incompatible type "List[DirichletBCMetaClass]"; expected "float"
dolfinx/fem/petsc.py:354: error: Argument 2 to "assemble_matrix_nest" has incompatible type "List[List[FormMetaClass]]"; expected "List[DirichletBCMetaClass]"
dolfinx/fem/petsc.py:371: error: Too many arguments for "assemble_matrix"
dolfinx/fem/petsc.py:371: error: Argument 2 to "assemble_matrix" has incompatible type "FormMetaClass"; expected "List[DirichletBCMetaClass]"
dolfinx/fem/petsc.py:371: error: Argument 3 to "assemble_matrix" has incompatible type "List[DirichletBCMetaClass]"; expected "float"
dolfinx/fem/petsc.py:391: error: Too many arguments for "assemble_matrix_block"
dolfinx/fem/petsc.py:391: error: Argument 2 to "assemble_matrix_block" has incompatible type "List[List[FormMetaClass]]"; expected "List[DirichletBCMetaClass]"
dolfinx/fem/petsc.py:391: error: Argument 3 to "assemble_matrix_block" has incompatible type "List[DirichletBCMetaClass]"; expected "float"
dolfinx/fem/petsc.py:461: error: Argument 1 to "bcs_by_block" has incompatible type "Iterable[Optional[FunctionSpace]]"; expected "Iterable[FunctionSpace]"
dolfinx/fem/petsc.py:468: error: Argument 3 to "apply_lifting" has incompatible type "Iterable[Iterable[DirichletBCMetaClass]]"; expected "List[List[DirichletBCMetaClass]]"
dolfinx/fem/petsc.py:564: error: "assemble_matrix" gets multiple values for keyword argument "bcs"
dolfinx/fem/petsc.py:697: error: Argument 3 to "assemble_matrix" has incompatible type "List[DirichletBCMetaClass]"; expected "float"
Found 32 errors in 3 files (checked 21 source files)

Demo errors:

dolfinx/fem/assemble.py:170: error: Dispatch type "ndarray[Any, Any]" must be subtype of fallback function first argument "FormMetaClass"
dolfinx/fem/assemble.py:230: error: Too many arguments for "assemble_matrix"
dolfinx/fem/assemble.py:230: error: Argument 1 to "assemble_matrix" has incompatible type "MatrixCSRMetaClass"; expected "FormMetaClass"
dolfinx/fem/assemble.py:230: error: Argument 2 to "assemble_matrix" has incompatible type "FormMetaClass"; expected "Optional[List[DirichletBCMetaClass]]"
dolfinx/fem/assemble.py:230: error: Argument 3 to "assemble_matrix" has incompatible type "List[DirichletBCMetaClass]"; expected "float"
dolfinx/fem/assemble.py:235: error: Dispatch type "MatrixCSRMetaClass" must be subtype of fallback function first argument "FormMetaClass"
dolfinx/fem/forms.py:168: error: Missing return statement
dolfinx/fem/forms.py:185: error: Item "Iterable[FormMetaClass]" of "Union[FormMetaClass, Iterable[FormMetaClass]]" has no attribute "function_spaces"
dolfinx/fem/petsc.py:181: error: Too many arguments for "assemble_vector"
dolfinx/fem/petsc.py:203: error: Too many arguments for "assemble_vector"
dolfinx/fem/petsc.py:220: error: Too many arguments for "assemble_vector_nest"
dolfinx/fem/petsc.py:234: error: Too many arguments for "assemble_vector"
dolfinx/fem/petsc.py:256: error: Too many arguments for "assemble_vector_block"
dolfinx/fem/petsc.py:256: error: Argument 2 to "assemble_vector_block" has incompatible type "List[FormMetaClass]"; expected "List[List[FormMetaClass]]"
dolfinx/fem/petsc.py:256: error: Argument 3 to "assemble_vector_block" has incompatible type "List[List[FormMetaClass]]"; expected "List[DirichletBCMetaClass]"
dolfinx/fem/petsc.py:256: error: Argument 5 to "assemble_vector_block" has incompatible type "Optional[Any]"; expected "float"
dolfinx/fem/petsc.py:291: error: Argument 1 to "bcs_by_block" has incompatible type "Iterable[Optional[FunctionSpace]]"; expected "Iterable[FunctionSpace]"
dolfinx/fem/petsc.py:302: error: Argument 1 to "bcs_by_block" has incompatible type "Iterable[Optional[FunctionSpace]]"; expected "Iterable[FunctionSpace]"
dolfinx/fem/petsc.py:325: error: Too many arguments for "assemble_matrix"
dolfinx/fem/petsc.py:325: error: Argument 2 to "assemble_matrix" has incompatible type "FormMetaClass"; expected "List[DirichletBCMetaClass]"
dolfinx/fem/petsc.py:325: error: Argument 3 to "assemble_matrix" has incompatible type "List[DirichletBCMetaClass]"; expected "float"
dolfinx/fem/petsc.py:354: error: Argument 2 to "assemble_matrix_nest" has incompatible type "List[List[FormMetaClass]]"; expected "List[DirichletBCMetaClass]"
dolfinx/fem/petsc.py:371: error: Too many arguments for "assemble_matrix"
dolfinx/fem/petsc.py:371: error: Argument 2 to "assemble_matrix" has incompatible type "FormMetaClass"; expected "List[DirichletBCMetaClass]"
dolfinx/fem/petsc.py:371: error: Argument 3 to "assemble_matrix" has incompatible type "List[DirichletBCMetaClass]"; expected "float"
dolfinx/fem/petsc.py:391: error: Too many arguments for "assemble_matrix_block"
dolfinx/fem/petsc.py:391: error: Argument 2 to "assemble_matrix_block" has incompatible type "List[List[FormMetaClass]]"; expected "List[DirichletBCMetaClass]"
dolfinx/fem/petsc.py:391: error: Argument 3 to "assemble_matrix_block" has incompatible type "List[DirichletBCMetaClass]"; expected "float"
dolfinx/fem/petsc.py:461: error: Argument 1 to "bcs_by_block" has incompatible type "Iterable[Optional[FunctionSpace]]"; expected "Iterable[FunctionSpace]"
dolfinx/fem/petsc.py:468: error: Argument 3 to "apply_lifting" has incompatible type "Iterable[Iterable[DirichletBCMetaClass]]"; expected "List[List[DirichletBCMetaClass]]"
dolfinx/fem/petsc.py:564: error: "assemble_matrix" gets multiple values for keyword argument "bcs"
dolfinx/fem/petsc.py:697: error: Argument 3 to "assemble_matrix" has incompatible type "List[DirichletBCMetaClass]"; expected "float"
demo/demo_types.py:40: error: Argument "points" to "create_rectangle" has incompatible type "Tuple[Tuple[float, float], Tuple[float, float]]"; expected "List[ndarray[Any, Any]]"
demo/demo_types.py:40: error: Argument "n" to "create_rectangle" has incompatible type "Tuple[int, int]"; expected "List[Any]"
demo/demo_types.py:59: error: Argument "marker" to "locate_entities_boundary" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_types.py:61: error: Argument "entities" to "locate_dofs_topological" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_stokes.py:140: error: Argument 3 to "locate_entities_boundary" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_stokes.py:141: error: Argument 1 to "dirichletbc" has incompatible type "ndarray[Any, Any]"; expected "Union[Function, Constant]"
demo/demo_stokes.py:141: error: Argument 2 to "dirichletbc" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_stokes.py:141: error: Argument 3 to "locate_dofs_topological" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_stokes.py:146: error: Argument 3 to "locate_entities_boundary" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_stokes.py:147: error: Argument 2 to "dirichletbc" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_stokes.py:147: error: Argument 3 to "locate_dofs_topological" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_stokes.py:206: error: Argument 1 to "bcs_by_block" has incompatible type "Iterable[Optional[FunctionSpace]]"; expected "Iterable[FunctionSpace]"
demo/demo_stokes.py:207: error: Argument 2 to "set_bc_nest" has incompatible type "Iterable[Iterable[DirichletBCMetaClass]]"; expected "List[List[DirichletBCMetaClass]]"
demo/demo_stokes.py:422: error: Argument 3 to "locate_entities_boundary" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_stokes.py:423: error: Argument 3 to "locate_dofs_topological" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_stokes.py:424: error: Argument 2 to "dirichletbc" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_stokes.py:430: error: Argument 3 to "locate_entities_boundary" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_stokes.py:431: error: Argument 3 to "locate_dofs_topological" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_stokes.py:432: error: Argument 2 to "dirichletbc" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_stokes.py:439: error: Argument 2 to "locate_dofs_geometrical" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_stokes.py:440: error: Argument 2 to "dirichletbc" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_static-condensation.py:65: error: Argument 3 to "locate_entities_boundary" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_static-condensation.py:66: error: Argument 4 to "meshtags" has incompatible type "int"; expected "ndarray[Any, Any]"
demo/demo_static-condensation.py:75: error: Argument 3 to "locate_entities_boundary" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_static-condensation.py:76: error: Argument 3 to "locate_dofs_topological" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_static-condensation.py:77: error: Argument 2 to "dirichletbc" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_poisson.py:88: error: Argument "points" to "create_rectangle" has incompatible type "Tuple[Tuple[float, float], Tuple[float, float]]"; expected "List[ndarray[Any, Any]]"
demo/demo_poisson.py:88: error: Argument "n" to "create_rectangle" has incompatible type "Tuple[int, int]"; expected "List[Any]"
demo/demo_poisson.py:106: error: Argument "marker" to "locate_entities_boundary" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_poisson.py:113: error: Argument "entities" to "locate_dofs_topological" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_poisson.py:119: error: Argument "dofs" to "dirichletbc" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_poisson.py:158: error: Missing positional argument "dim" in call to "create_vtk_mesh"
demo/demo_lagrange_variants.py:117: error: Argument "points" to "create_rectangle" has incompatible type "Tuple[Tuple[float, float], Tuple[float, float]]"; expected "List[ndarray[Any, Any]]"
demo/demo_lagrange_variants.py:117: error: Argument "n" to "create_rectangle" has incompatible type "Tuple[int, int]"; expected "List[Any]"
demo/demo_lagrange_variants.py:122: error: Argument "marker" to "locate_entities_boundary" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_lagrange_variants.py:125: error: Argument "entities" to "locate_dofs_topological" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_lagrange_variants.py:127: error: Argument "dofs" to "dirichletbc" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_lagrange_variants.py:180: error: Argument 2 to "eval" of "Function" has incompatible type "List[int]"; expected "ndarray[Any, Any]"
demo/demo_interpolation-io.py:36: error: Argument 2 to "create_rectangle" has incompatible type "Tuple[Tuple[float, float], Tuple[float, float]]"; expected "List[ndarray[Any, Any]]"
demo/demo_interpolation-io.py:36: error: Argument 3 to "create_rectangle" has incompatible type "Tuple[int, int]"; expected "List[Any]"
demo/demo_interpolation-io.py:44: error: Argument 3 to "locate_entities" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_interpolation-io.py:45: error: Argument 3 to "locate_entities" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_interpolation-io.py:55: error: Argument 2 to "VectorFunctionSpace" has incompatible type "Tuple[str, int]"; expected "ElementMetaData"
demo/demo_interpolation-io.py:73: error: Missing positional argument "dim" in call to "create_vtk_mesh"
demo/demo_gmsh.py:123: error: Argument 4 to "meshtags_from_entities" has incompatible type "signedinteger[_32Bit]"; expected "ndarray[Any, Any]"
demo/demo_gmsh.py:183: error: Argument 4 to "meshtags_from_entities" has incompatible type "signedinteger[_32Bit]"; expected "ndarray[Any, Any]"
demo/demo_gmsh.py:260: error: Argument 4 to "meshtags_from_entities" has incompatible type "signedinteger[_32Bit]"; expected "ndarray[Any, Any]"
demo/demo_elasticity.py:119: error: Argument 2 to "VectorFunctionSpace" has incompatible type "Tuple[str, int]"; expected "ElementMetaData"
demo/demo_elasticity.py:130: error: Argument "marker" to "locate_entities_boundary" has incompatible type "Callable[[Any], Any]"; expected "FunctionType"
demo/demo_elasticity.py:132: error: Argument 1 to "dirichletbc" has incompatible type "ndarray[Any, Any]"; expected "Union[Function, Constant]"
demo/demo_elasticity.py:133: error: Argument 2 to "dirichletbc" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_elasticity.py:133: error: Argument "entities" to "locate_dofs_topological" has incompatible type "ndarray[Any, Any]"; expected "List[int]"
demo/demo_cahn-hilliard.py:304: error: Missing positional argument "dim" in call to "create_vtk_mesh"
Found 85 errors in 12 files (checked 14 source files)

…od found.

It seems like there are projects to generate typing data from pybind11 automagically but I think it looks too complex for a few methods.

…s-and-ci

…nto jhale/mypy-fixes-and-ci

francesco-ballarin · 2022-05-28T07:50:52Z

Can @jhale point me to which part of the diff requires python >= 3.8? I know that mypy wheels have dropped support for older python versions (at least on linux) so that must be part of the reason, but I am wondering if there are other changes in the diff that require python >= 3.8.

I am happy to keep around a weekly CI on python 3.7 for this project by shipping a minor patch on the python version check in setup.py. Admittedly, I only run a very basic demo there, but that's representative of the basic usage of the library (e.g. for teaching).

jhale added 5 commits May 20, 2022 09:17

Remove adjoint, unused and untested.

9fbe4ed

Remove this, cannot work with missing RTLD_NOW in modern ctypes.

17f5d63

Work in progress for mypy pass (MetaClass stuff is tricky).

f7f9d33

Add mypy to ubuntu image

0a849cd

Again

bd8a508

jhale marked this pull request as draft May 20, 2022 07:32

jhale added 4 commits May 20, 2022 09:32

Merge branch 'main' into jhale/mypy-fixes-and-ci

d695e3e

Move later

c417f26

Merge branch 'jhale/mypy-fixes-and-ci' of github.com:FEniCS/dolfinx i…

2aa08c1

…nto jhale/mypy-fixes-and-ci

Bit more typing, remove adjoint import

3494584

jhale requested a review from garth-wells May 20, 2022 08:12

jhale requested a review from mscroggs May 20, 2022 08:13

jhale added 5 commits May 20, 2022 10:23

Fix but more typing errors back

068c3e8

Adding if makes type check pass

764faae

35 errors left in python/dolfinx

5d2c439

Remaining issues are due to use of single dispatch

129fda7

Add py.typed files so that users actually see type annotations

edff60b

jhale added 10 commits May 20, 2022 19:19

Generalised arguments types for mesh constructors

a3b5a86

Fix singledispatch definition to raise NotImplementedError if no meth…

1d50a2e

…od found.

Fixes all typing issues in assemble.

31d55be

9 left in demos, 22 in main library (mainly in petsc.py)

6a20000

flake8 fixes

5a02e4b

Merge branch 'main' into jhale/mypy-fixes-and-ci

cdebd1a

Demos pass, petsc.py and two strange ones to go

2b1e356

Liberal mypy passes, all tests pass

ff21edf

Fix fallback vector

5f49d1a

Fixes

04dd71e

jhale added 27 commits May 24, 2022 11:13

Type pybind11 .

5b62896

It seems like there are projects to generate typing data from pybind11 automagically but I think it looks too complex for a few methods.

Remove mypy install from CI

8bcf985

Also wrap cell_perm_gmsh

917e3cf

Make dim optional, grab tdim?

239d93d

Reallow value to be passed

d078159

Allow lists to eval

99bc314

Remove comment

73f4d96

Better API?

ade54ba

Prefer this design.

7962032

Better singledispatch la

89c891b

Bug fixes

16ec231

Fix

e0e30ec

Merge branch 'main' into jhale/mypy-fixes-and-ci

458bf41

Ignore return error

947051b

flake8

9296b15

Remove explicit casts

d4244c6

Remove cryptic typings, ignore errors

b352006

Revert some files to main

83cb9eb

Remove explicit dim

5477334

Small tweaks.

4dadcba

Revert

e431eb0

flake8

6bd4d06

Merge branch 'main' of github.com:FEniCS/dolfinx into jhale/mypy-fixe…

cca3b9f

…s-and-ci

Merge branch 'main' into jhale/mypy-fixes-and-ci

8ebad9d

Fix, should discuss this again

ecc87b3

Merge remote-tracking branch 'origin/main' into jhale/mypy-fixes-and-ci

a2d49c5

Merge branch 'jhale/mypy-fixes-and-ci' of github.com:FEniCS/dolfinx i…

2163afb

…nto jhale/mypy-fixes-and-ci

jhale merged commit 6c631eb into main May 25, 2022

jhale deleted the jhale/mypy-fixes-and-ci branch May 25, 2022 14:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix errors and add mypy to CI #2190

Fix errors and add mypy to CI #2190

jhale commented May 20, 2022

jhale commented May 20, 2022

jhale commented May 20, 2022 •

edited

jhale commented May 20, 2022 •

edited

francesco-ballarin commented May 28, 2022

Fix errors and add mypy to CI #2190

Fix errors and add mypy to CI #2190

Conversation

jhale commented May 20, 2022

jhale commented May 20, 2022

jhale commented May 20, 2022 • edited

jhale commented May 20, 2022 • edited

francesco-ballarin commented May 28, 2022

jhale commented May 20, 2022 •

edited

jhale commented May 20, 2022 •

edited