Enable type checking of the tests folder

[ordabayevy]

Most of the errors are due to the missing explicit imports. What should we do with them @fritzo ?

mypy --install-types --non-interactive pyro scripts tests
tests/test_generic.py:7: error: Module "pyro.generic" has no attribute "handlers"  [attr-defined]
tests/test_generic.py:7: error: Module "pyro.generic" has no attribute "infer"  [attr-defined]
tests/test_generic.py:7: error: Module "pyro.generic" has no attribute "ops"  [attr-defined]
tests/test_generic.py:7: error: Module "pyro.generic" has no attribute "pyro"  [attr-defined]
tests/test_generic.py:7: error: Module "pyro.generic" has no attribute "pyro_backend"  [attr-defined]
tests/test_generic.py:13: error: Unexpected keyword argument "category" for "__call__" of "_FilterwarningsMarkDecorator"  [call-arg]
/home/yordabay/anaconda3/envs/pyro/lib/python3.8/site-packages/_pytest/mark/structures.py:493: note: "__call__" of "_FilterwarningsMarkDecorator" defined here
tests/ops/test_integrator.py:19: error: First argument to namedtuple() should be "ModelArgs", not "model_args"  [name-match]
tests/ops/test_integrator.py:22: error: First argument to namedtuple() should be "Example", not "test_case"  [name-match]
tests/contrib/test_minipyro.py:11: error: Module "pyro.generic" has no attribute "distributions"  [attr-defined]
tests/contrib/test_minipyro.py:12: error: Module "pyro.generic" has no attribute "infer"  [attr-defined]
tests/contrib/test_minipyro.py:12: error: Module "pyro.generic" has no attribute "ops"  [attr-defined]
tests/contrib/test_minipyro.py:12: error: Module "pyro.generic" has no attribute "optim"  [attr-defined]
tests/contrib/test_minipyro.py:12: error: Module "pyro.generic" has no attribute "pyro"  [attr-defined]
tests/contrib/test_minipyro.py:12: error: Module "pyro.generic" has no attribute "pyro_backend"  [attr-defined]
tests/test_primitives.py:62: error: "Callable[[VarArg(Any), KwArg(Any)], Any]" has no attribute "batch_shape"  [attr-defined]
tests/test_primitives.py:63: error: "Callable[[VarArg(Any), KwArg(Any)], Any]" has no attribute "event_shape"  [attr-defined]
tests/poutine/test_poutines.py:858: error: No overload variant of "trace" matches argument types "Callable[[], Any]", "str"  [call-overload]
tests/poutine/test_poutines.py:858: note: Possible overload variants:
tests/poutine/test_poutines.py:858: note:     def trace(fn: None = ..., graph_type: Optional[Literal['flat', 'dense']] = ..., param_only: Optional[bool] = ...) -> TraceMessenger
tests/poutine/test_poutines.py:858: note:     def [_P`-1, _T] trace(fn: Callable[_P, _T], graph_type: Optional[Literal['flat', 'dense']] = ..., param_only: Optional[bool] = ...) -> TraceHandler[_P, _T]
tests/poutine/test_poutines.py:969: error: Dict entry 0 has incompatible type "str": "float"; expected "str": "Tensor"  [dict-item]
tests/optim/test_optim.py:93: error: Module has no attribute "LambdaLR"  [attr-defined]
tests/optim/test_optim.py:100: error: Module has no attribute "StepLR"  [attr-defined]
tests/optim/test_optim.py:108: error: Module has no attribute "ExponentialLR"  [attr-defined]
tests/optim/test_optim.py:111: error: Module has no attribute "ReduceLROnPlateau"  [attr-defined]
tests/optim/test_optim.py:169: error: Module has no attribute "Adam"  [attr-defined]
tests/optim/test_optim.py:169: error: Module has no attribute "RMSprop"  [attr-defined]
tests/optim/test_optim.py:169: error: Module has no attribute "SGD"  [attr-defined]
tests/optim/test_optim.py:176: error: Module has no attribute "Adam"  [attr-defined]
tests/optim/test_optim.py:176: error: Module has no attribute "SGD"  [attr-defined]
tests/optim/test_optim.py:335: error: Module has no attribute "Adam"  [attr-defined]
tests/optim/test_optim.py:338: error: Module has no attribute "RMSprop"  [attr-defined]
tests/optim/test_optim.py:339: error: Module has no attribute "SGD"  [attr-defined]
tests/optim/test_optim.py:341: error: Module has no attribute "LambdaLR"  [attr-defined]
tests/optim/test_optim.py:349: error: Module has no attribute "StepLR"  [attr-defined]
tests/optim/test_optim.py:358: error: Module has no attribute "ExponentialLR"  [attr-defined]
tests/optim/test_optim.py:362: error: Module has no attribute "ReduceLROnPlateau"  [attr-defined]
tests/optim/test_multi.py:21: error: Module has no attribute "Adam"  [attr-defined]
tests/optim/test_multi.py:22: error: Argument 1 to "TorchMultiOptimizer" has incompatible type "Type[Adam]"; expected "Optimizer"  [arg-type]
tests/optim/test_multi.py:26: error: Module has no attribute "Adam"  [attr-defined]
tests/optim/test_multi.py:31: error: Module has no attribute "Adam"  [attr-defined]
tests/nn/test_module.py:17: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/nn/test_module.py:501: error: Too few arguments for "PyroParam"  [call-arg]
tests/nn/test_module.py:505: error: Too few arguments for "PyroParam"  [call-arg]
tests/nn/test_module.py:509: error: Missing positional argument "init_value" in call to "PyroParam"  [call-arg]
tests/infer/test_svgd.py:11: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/infer/test_jit.py:30: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/infer/test_jit.py:592: error: Argument 3 to "CondIndepStackFrame" has incompatible type "Tensor"; expected "int"  [arg-type]
tests/infer/test_jit.py:597: error: Argument 3 to "CondIndepStackFrame" has incompatible type "Tensor"; expected "int"  [arg-type]
tests/infer/test_gradient.py:28: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/distributions/test_sine_skewed.py:12: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/distributions/test_reshape.py:7: error: Module "pyro.distributions.torch" has no attribute "Bernoulli"  [attr-defined]
tests/distributions/test_nanmasked.py:13: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/distributions/test_mask.py:9: error: Module "pyro.distributions.torch" has no attribute "Bernoulli"  [attr-defined]
tests/distributions/test_lkj.py:17: error: Module "pyro.distributions.torch" has no attribute "LKJCholesky"  [attr-defined]
tests/infer/mcmc/test_hmc.py:65: error: First argument to namedtuple() should be "T", not "TestExample"  [name-match]
tests/distributions/test_rejector.py:18: error: Argument 1 to "map" has incompatible type "Type[Size]"; expected "Callable[[object], Size]"  [arg-type]
tests/contrib/test_zuko.py:11: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/contrib/test_zuko.py:23: error: Incompatible types in assignment (expression has type "Type[Normal]", variable has type "Type[MultivariateNormal]")  [assignment]
tests/contrib/test_zuko.py:35: error: "MultivariateNormal" has no attribute "rsample_and_log_prob"  [attr-defined]
tests/contrib/tracking/test_em.py:16: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/infer/mcmc/test_nuts.py:32: error: First argument to namedtuple() should be "T", not "TestExample"  [name-match]
tests/contrib/mue/test_models.py:12: error: Module "pyro.optim" has no attribute "MultiStepLR"  [attr-defined]
tests/contrib/easyguide/test_easyguide.py:16: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/contrib/easyguide/test_easyguide.py:71: error: Metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases  [misc]
tests/infer/test_valid_models.py:30: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/infer/test_autoguide.py:53: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/infer/test_autoguide.py:63: error: Cannot assign to a type  [misc]
tests/infer/test_autoguide.py:63: error: Incompatible types in assignment (expression has type "ParameterSet", variable has type "Type[AutoGaussianFunsor]")  [assignment]
tests/infer/test_autoguide.py:448: error: The type "Type[AutoGaussianFunsor]" is not generic and not indexable  [misc]
tests/infer/test_autoguide.py:959: error: The type "Type[AutoGaussianFunsor]" is not generic and not indexable  [misc]
tests/infer/reparam/test_strategies.py:13: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/contrib/gp/test_models.py:28: error: First argument to namedtuple() should be "T", not "TestGPModel"  [name-match]
tests/contrib/gp/test_likelihoods.py:14: error: First argument to namedtuple() should be "T", not "TestGPLikelihood"  [name-match]
tests/contrib/gp/test_kernels.py:31: error: First argument to namedtuple() should be "T", not "TestGPKernel"  [name-match]
tests/contrib/gp/test_conditional.py:14: error: First argument to namedtuple() should be "T", not "TestConditional"  [name-match]
tests/contrib/forecast/test_forecaster.py:13: error: Module "pyro.optim" has no attribute "Adam"  [attr-defined]
tests/contrib/forecast/test_forecaster.py:16: error: Metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases  [misc]
tests/contrib/forecast/test_forecaster.py:30: error: Metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases  [misc]
tests/contrib/forecast/test_forecaster.py:44: error: Metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases  [misc]
tests/contrib/forecast/test_forecaster.py:59: error: Metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases  [misc]
tests/contrib/forecast/test_forecaster.py:81: error: Metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases  [misc]
tests/contrib/forecast/test_forecaster.py:244: error: Metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases  [misc]
tests/contrib/forecast/test_forecaster.py:269: error: Metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases  [misc]
tests/contrib/forecast/test_evaluate.py:16: error: Metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases  [misc]
tests/perf/test_benchmark.py:25: error: First argument to namedtuple() should be "Model", not "TestModel"  [name-match]
tests/perf/test_benchmark.py:194: error: Incompatible types in assignment (expression has type "Set[Any]", variable has type "List[filter[Any]]")  [assignment]
tests/contrib/funsor/test_vectorized_markov.py:21: error: Name "pyro" already defined (by an import)  [no-redef]
tests/contrib/funsor/test_valid_models_enum.py:28: error: Name "pyro" already defined (by an import)  [no-redef]
tests/contrib/funsor/test_tmc.py:22: error: Name "pyro" already defined (by an import)  [no-redef]
tests/contrib/funsor/test_named_handlers.py:19: error: Name "pyro" already defined (by an import)  [no-redef]
tests/contrib/funsor/test_infer_discrete.py:22: error: Name "pyro" already defined (by an import)  [no-redef]
tests/contrib/funsor/test_enum_funsor.py:25: error: Name "pyro" already defined (by an import)  [no-redef]
tests/contrib/funsor/test_valid_models_sequential_plate.py:19: error: Name "pyro" already defined (by an import)  [no-redef]
tests/contrib/funsor/test_valid_models_plate.py:20: error: Name "pyro" already defined (by an import)  [no-redef]
Found 92 errors in 41 files (checked 551 source files)

[fritzo]

@ordabayevy Thanks for enabling this! I'm embarassed we released before testing 😊

What should we do

Let's see, there are a few kinds of errors:

• Let's ignore pyro.generic and minipyro for now; they are very dynamic
• Explicit + implicit imports seem like a good idea, as we agreed on in Explicitly import distributions from torch #3333 . That should at least fix the optim errors.
• The PyroParam errors look real, I guess you've looked into them.
• The primitives errors look like a type isn't being propagated, similar to the problem in Fix typing of poutine.trace(model).get_trace() #3334
• In test_poutines, it looks like we may need a wider type for poutine.trace.
• The metaclass conflict errors look scary, but I'd have to take a closer look.

Do you want to take a first pass, or divvy these errors up, or do you want me to take the first pass?

[ordabayevy]

You can have a first pass if you want @fritzo . I'll have a stab at pyro.nn.module meanwhile.

[fritzo]

@ordabayevy How about we follow the same typing workflow as the rest of Pyro?

1. Enable mypy on the tests directory, but
2. add mypy ignores to setup.cfg for each failing file or directory, and then
3. gradually add types as tracked by Using Python type hints #2550

I've added ignores in this PR, and for each ignore have added a check box in #2250

[fritzo]

(approving my own modifications to @ordabayevy's PR)

[ordabayevy]

Sounds good to me @fritzo !