Bumped mypy and tensorflow-cpu to newer versions

[sisco0]

The changes present in the pyproject.tml Poetry file were required to overcome the issues when running ./dev/test.sh in a clean new Poetry environment.

A wider definition of these changes is presented in #961.

[sisco0]

This seems to take infinite time to build, so this action should be canceled @tanertopal @danieljanes .

[sisco0]

Rolled back to mypy 0.910. How could we get into 0.930? The previous Github Action run needs to be canceled.

[danieljanes]

@sisco0 what issues did you see with mypy 0.930? Always good to bump dependencies! We usually pin dev-dependencies to a fixed version (==, not sure why this isn't the case for those types-* deps, we should pin them as well).

[sisco0]

The issues could be seen as poetry install ends up in an infinite loop, this could be reviewed by checking out the commit 0d6a9ef locally and running poetry install -vvv. I am unsure of how could we get mypy 0.930 into our current development dependencies of Flower. By the way, there is still a need of cancelling running Github Actions for this.

[danieljanes]

Thanks for the update @sisco0 , I was able to reproduce the described behaviour. As a first step, we've updated pip/setuptools/poetry to the latest possible versions in #970 Let's see what else we need to make to update to 0.930

[danieljanes]

@sisco0 we dropped quite a few optional dependencies in #869 - tensorflow-cpu amongst them. Could you try to update to mypy 0.930 again? I could do it in a separate PR, but I don't want to steal your contribution :) Ideally try it without the Python version range.

[sisco0]

I tried newly with python version specifier set to ">=3.6.2,<3.10" and we got no luck with numpy. The output for poetry install -vvv is shown below:

   1: fact: numpy (1.21.5) requires Python >=3.7,<3.11
   1: derived: not numpy (==1.21.5)
   1: fact: numpy (1.21.4) requires Python >=3.7,<3.11
   1: derived: not numpy (==1.21.4)
   1: fact: numpy (1.21.3) requires Python >=3.7,<3.11
   1: derived: not numpy (==1.21.3)
   1: fact: numpy (1.21.2) requires Python >=3.7,<3.11
   1: derived: not numpy (==1.21.2)
   1: fact: numpy (1.21.1) requires Python >=3.7
   1: derived: not numpy (==1.21.1)
   1: fact: numpy (1.21.0) requires Python >=3.7
   1: derived: not numpy (==1.21.0)
   1: fact: numpy (1.20.3) requires Python >=3.7
   1: derived: not numpy (==1.20.3)
   1: fact: numpy (1.20.2) requires Python >=3.7
   1: derived: not numpy (==1.20.2)

If we go on with python = ">=3.7.0,<3.10" it works as expected but the tests fail newly with mypy at ./dev/test.sh (see log below) with Python version 3.8.10. Please, do not hesitate to make all the required changes in my branch or create a new Pull Request as I need help on how to solve this issue. I am happy to push forward our duties.

I attach the failing log for ./dev/test.sh below:

Click here to see log

╰─ ./dev/test.sh                                                                                                                                                                                                                                                           ─╯
=== test.sh ===
- clang-format:  done
Skipped 1 files
- isort:         done
All done! ✨ 🍰 ✨
70 files would be left unchanged.
- black:         done
- init_py_check: done
- docformatter:  done
src/py/flwr/dataset/utils/common.py:38: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:48: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:74: error: Call to untyped function "unique" in typed context
src/py/flwr/dataset/utils/common.py:85: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:95: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:101: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:103: error: Call to untyped function "split" in typed context
src/py/flwr/dataset/utils/common.py:109: error: Call to untyped function "concatenate" in typed context
src/py/flwr/dataset/utils/common.py:114: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:121: error: Call to untyped function "concatenate" in typed context
src/py/flwr/dataset/utils/common.py:183: error: Call to untyped function "unique" in typed context
src/py/flwr/dataset/utils/common.py:198: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:201: error: Redundant cast to "ndarray[Any, Any]"
src/py/flwr/dataset/utils/common.py:201: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:204: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:207: error: Redundant cast to "ndarray[Any, Any]"
src/py/flwr/dataset/utils/common.py:207: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:211: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:212: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:244: error: Call to untyped function "append" in typed context
src/py/flwr/dataset/utils/common.py:246: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:256: error: Function "numpy.array" is not valid as a type
src/py/flwr/dataset/utils/common.py:256: note: Perhaps you need "Callable[...]" or a callback protocol?
src/py/flwr/dataset/utils/common.py:257: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:277: error: np.array? has no attribute "size"
src/py/flwr/dataset/utils/common.py:285: error: Value of type np.array? is not indexable
src/py/flwr/dataset/utils/common.py:285: error: Unsupported target for indexed assignment (np.array?)
src/py/flwr/dataset/utils/common.py:286: error: Unsupported target for indexed assignment (np.array?)
src/py/flwr/dataset/utils/common.py:294: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:295: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:326: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:327: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:331: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:334: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "ndarray[Any, Any]"
src/py/flwr/dataset/utils/common.py:345: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:345: error: Call to untyped function "concatenate" in typed context
src/py/flwr/dataset/utils/common.py:346: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:351: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:370: error: Call to untyped function "histogram" in typed context
src/py/flwr/dataset/utils/common.py:378: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:378: error: Incompatible default for argument "dirichlet_dist" (default has type "None", argument has type "ndarray[Any, Any]")
src/py/flwr/dataset/utils/common.py:380: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:382: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common.py:421: error: Call to untyped function "unique" in typed context
src/py/flwr/dataset/utils/common.py:436: error: Incompatible types in assignment (expression has type "Tuple[ndarray[Any, Any], List[int]]", variable has type "ndarray[Any, Any]")
src/py/flwr/dataset/utils/common.py:449: error: Missing type parameters for generic type "ndarray"
src/py/flwr/common/typing.py:23: error: Missing type parameters for generic type "ndarray"
src/py/flwr/dataset/utils/common_test.py:66: error: Call to untyped function "concatenate" in typed context
src/py/flwr/dataset/utils/common_test.py:126: error: Call to untyped function "assert_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:127: error: Call to untyped function "assert_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:128: error: Call to untyped function "assert_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:129: error: Call to untyped function "assert_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:171: error: Call to untyped function "assert_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:172: error: Call to untyped function "assert_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:179: error: Call to untyped function "expand_dims" in typed context
src/py/flwr/dataset/utils/common_test.py:253: error: Call to untyped function "concatenate" in typed context
src/py/flwr/dataset/utils/common_test.py:262: error: Call to untyped function "assert_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:262: error: Value of type "object" is not indexable
src/py/flwr/dataset/utils/common_test.py:319: error: Call to untyped function "assert_array_almost_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:344: error: Argument 2 to "sample_without_replacement" has incompatible type "List[object]"; expected "List[List[ndarray[Any, Any]]]"
src/py/flwr/dataset/utils/common_test.py:366: error: Argument 2 to "sample_without_replacement" has incompatible type "List[object]"; expected "List[List[ndarray[Any, Any]]]"
src/py/flwr/dataset/utils/common_test.py:395: error: Need type annotation for "expected_x"
src/py/flwr/dataset/utils/common_test.py:400: error: Argument 2 to "sample_without_replacement" has incompatible type "List[object]"; expected "List[List[ndarray[Any, Any]]]"
src/py/flwr/dataset/utils/common_test.py:457: error: Call to untyped function "max" in typed context
src/py/flwr/dataset/utils/common_test.py:467: error: Need type annotation for "uniform"
src/py/flwr/dataset/utils/common_test.py:467: error: Module has no attribute "float"; maybe "float_", "cfloat", or "float96"?
src/py/flwr/dataset/utils/common_test.py:479: error: Call to untyped function "assert_array_almost_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:494: error: Call to untyped function "assert_array_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:497: error: Call to untyped function "assert_array_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:500: error: Call to untyped function "assert_array_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:517: error: Call to untyped function "concatenate" in typed context
src/py/flwr/dataset/utils/common_test.py:518: error: Call to untyped function "concatenate" in typed context
src/py/flwr/dataset/utils/common_test.py:534: error: Call to untyped function "concatenate" in typed context
src/py/flwr/dataset/utils/common_test.py:535: error: Call to untyped function "concatenate" in typed context
src/py/flwr/dataset/utils/common_test.py:538: error: Call to untyped function "assert_array_equal" in typed context
src/py/flwr/dataset/utils/common_test.py:557: error: Call to untyped function "concatenate" in typed context
src/py/flwr/dataset/utils/common_test.py:558: error: Call to untyped function "concatenate" in typed context
src/py/flwr/common/parameter.py:37: error: Missing type parameters for generic type "ndarray"
src/py/flwr/common/parameter.py:40: error: Call to untyped function "save" in typed context
src/py/flwr/common/parameter.py:44: error: Missing type parameters for generic type "ndarray"
src/py/flwr/common/parameter.py:47: error: Call to untyped function "load" in typed context
src/py/flwr/common/parameter.py:48: error: Missing type parameters for generic type "ndarray"
src/py/flwr/common/parameter_test.py:36: error: Call to untyped function "assert_equal" in typed context
src/py/flwr/common/parameter_test.py:40: error: Call to untyped function "assert_equal" in typed context
src/py/flwr/server/strategy/aggregate_test.py:41: error: Call to untyped function "assert_equal" in typed context
src/py/flwr/client/numpy_client.py:97: error: Missing type parameters for generic type "ndarray"
src/py/flwr/client/numpy_client.py:125: error: Missing type parameters for generic type "ndarray"
src/py/flwr/client/numpy_client.py:126: error: Missing type parameters for generic type "ndarray"
src/py/flwr/client/numpy_client.py:156: error: Missing type parameters for generic type "ndarray"
src/py/flwr/client/numpy_client.py:213: error: Missing type parameters for generic type "ndarray"
src/py/flwr/client/numpy_client.py:220: error: Missing type parameters for generic type "ndarray"
src/py/flwr/client/numpy_client.py:224: error: Missing type parameters for generic type "ndarray"
src/py/flwr/client/numpy_client.py:240: error: Missing type parameters for generic type "ndarray"
src/py/flwr/client/keras_client.py:152: error: Missing type parameters for generic type "ndarray"
src/py/flwr/client/keras_client.py:156: error: Missing type parameters for generic type "ndarray"
src/py/flwr/server/strategy/fedavg_android.py:278: error: Missing type parameters for generic type "ndarray"
src/py/flwr/server/strategy/fedavg_android.py:280: error: Redundant cast to "bytes"
src/py/flwr/server/strategy/fedavg_android.py:283: error: Missing type parameters for generic type "ndarray"
src/py/flwr/server/strategy/fedavg_android.py:285: error: Call to untyped function "frombuffer" in typed context
src/py/flwr/server/strategy/fedavg_android.py:286: error: Missing type parameters for generic type "ndarray"
src/py/flwr/server/strategy/qfedavg.py:176: error: Call to untyped function "append" in typed context
src/py/flwr/server/strategy/qfedavg.py:206: error: Argument 1 to "norm_grad" has incompatible type "List[float]"; expected "List[List[ndarray[Any, Any]]]"
src/py/flwr/server/strategy/fast_and_slow.py:392: error: Missing type parameters for generic type "ndarray"
src/py/flwr/server/strategy/fast_and_slow.py:397: error: Call to untyped function "max" in typed context
src/py/flwr/server/strategy/fast_and_slow.py:398: error: Missing type parameters for generic type "ndarray"
src/py/flwr/server/strategy/fast_and_slow.py:404: error: Missing type parameters for generic type "ndarray"
src/py/flwr/server/strategy/fedyogi.py:143: error: Need type annotation for "x"
src/py/flwr/server/strategy/fedyogi.py:144: error: Need type annotation for "y"
src/py/flwr/server/strategy/fedyogi.py:145: error: List comprehension has incompatible type List[float]; expected List[ndarray[Any, Any]]
src/py/flwr/server/strategy/fedyogi.py:150: error: Need type annotation for "x"
src/py/flwr/server/strategy/fedyogi.py:151: error: Need type annotation for "y"
src/py/flwr/server/strategy/fedadam.py:143: error: Need type annotation for "x"
src/py/flwr/server/strategy/fedadam.py:144: error: Need type annotation for "y"
src/py/flwr/server/strategy/fedadam.py:145: error: List comprehension has incompatible type List[float]; expected List[ndarray[Any, Any]]
src/py/flwr/server/strategy/fedadam.py:150: error: Need type annotation for "x"
src/py/flwr/server/strategy/fedadam.py:151: error: Need type annotation for "y"
src/py/flwr/server/strategy/fedadagrad.py:139: error: Need type annotation for "x"
src/py/flwr/server/strategy/fedadagrad.py:140: error: Need type annotation for "y"
src/py/flwr/server/strategy/fedadagrad.py:141: error: List comprehension has incompatible type List[float]; expected List[ndarray[Any, Any]]
src/py/flwr/server/strategy/fedadagrad.py:146: error: Need type annotation for "x"
src/py/flwr/server/strategy/fedadagrad.py:147: error: Need type annotation for "y"
Found 120 errors in 14 files (checked 169 source files)

[sisco0]

It is mypy in conjunction with numpy version that is causing some errors when running the ./dev/test.sh tests. Maybe some new typing behavior is included in numpy that was not in previous versions. Is this only seen by my side and should I provide further details on my configuration?

[pedropgusmao]

Hi @sisco0 , setting mypy==0.931 and python = "^3.6.2" worked for me. i.e. the dev/test.sh works and I end up with numpy version '1.19.5. Could try this version of mypy, please?

[sisco0]

The solution has been tested locally with success results. Thank you for that @pedropgusmao, as the 0.931 version seems to fix the dependency chain issue that was had before with the 0.930 one ⭐.

[pedropgusmao]

Thanks, @sisco0 !
@danieljanes or @tanertopal , could you please merge this PR? I'm not code-owner ...