Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Python type hints #708

Merged
merged 28 commits into from
Jun 5, 2023
Merged

Use Python type hints #708

merged 28 commits into from
Jun 5, 2023

Conversation

juhoinkinen
Copy link
Member

@juhoinkinen juhoinkinen commented May 24, 2023

The first round of type hints were generated by running monkeytype apply --pep_563 on all modules.

Then type hints have been manually corrected, and simplified by (at least with)

  • using only float for int | float (allowed by PEP484)
  • using dict[X, Any] for many dictionaries, because otherwise more narrow union types become long and verbose (the type of the key/value cannot be set as union inside the dict, e.g. dict[str, float | str] does not work, but this should be written as dict[X, str] | dict[X, int]; )
  • using regular float for numpy.float64 and int for numpy.int32 etc.

I have been using this mypy configuration (in mypy.ini), which allows to just run mypy in root of Annif repo and it makes mypy ignore a bunch of errors:

[mypy]
files = annif/
ignore_missing_imports = True
disable_error_code = assignment, var-annotated, attr-defined, union-attr, return, has-type, dict-item, call-arg, misc, index, override

With this config mypy gives "only" 20 errors.

Show errors

annif/suggestion.py:163: error: Argument 1 to "from_iterable" of "chain" has incompatible type "Iterable[SuggestionBatch]"; expected "Iterable[Iterable[<nothing>]]"  [arg-type]
annif/suggestion.py:163: error: Argument 1 to "from_iterable" of "chain" has incompatible type "Iterable[SuggestionBatch]"; expected "Iterable[Iterable[Any]]"  [arg-type]
annif/eval.py:267: error: Argument 4 to "output_result_per_subject" of "EvaluationBatch" has incompatible type "Optional[str]"; expected "str"  [arg-type]
annif/backend/backend.py:118: error: Argument 1 to "int" has incompatible type "Optional[Any]"; expected "Union[str, bytes, bytearray, memoryview, array[Any], mmap, _CData, PickleBuffer, SupportsInt, SupportsIndex, SupportsTrunc]"  [arg-type]
annif/lexical/mllm.py:113: error: Value of type variable "_SCT" of "zeros" cannot be "bool"  [type-var]
annif/backend/stwfsa.py:110: error: "object" not callable  [operator]
annif/backend/hyperopt.py:107: error: Argument "callbacks" to "optimize" of "Study" has incompatible type "List[Callable[[Study, Trial], None]]"; expected "Optional[List[Callable[[Study, FrozenTrial], None]]]"  [arg-type]
annif/backend/http.py:36: error: Incompatible return value type (got "Union[bool, str, None]", expected "Optional[bool]")  [return-value]
annif/backend/http.py:43: error: Argument 1 to "parse" has incompatible type "Union[bool, str]"; expected "Union[bytes, str, IO[str], IO[Any]]"  [arg-type]
annif/backend/ensemble.py:157: error: Incompatible return value type (got "Union[bool, datetime, None]", expected "Optional[datetime]")  [return-value]
annif/corpus/document.py:67: error: Argument 2 to "from_string" of "SubjectSet" has incompatible type "Optional[SubjectIndex]"; expected "SubjectIndex"  [arg-type]
annif/corpus/document.py:67: error: Argument 3 to "from_string" of "SubjectSet" has incompatible type "Optional[str]"; expected "str"  [arg-type]
annif/vocab.py:93: error: Incompatible return value type (got "None", expected "SubjectFileSKOS")  [return-value]
annif/vocab.py:102: error: Incompatible return value type (got "None", expected "SubjectFileSKOS")  [return-value]
annif/vocab.py:111: error: Incompatible return value type (got "Optional[List[str]]", expected "List[str]")  [return-value]
annif/transform/__init__.py:47: error: Argument 1 to "TransformChain" has incompatible type "List[ABCMeta]"; expected "List[Type[BaseTransform]]"  [arg-type]
annif/project.py:198: error: Incompatible return value type (got "Optional[AnnifVocabulary]", expected "AnnifVocabulary")  [return-value]
annif/project.py:204: error: Incompatible return value type (got "Optional[str]", expected "str")  [return-value]
annif/project.py:221: error: Incompatible return value type (got "Union[bool, datetime, None]", expected "Optional[bool]")  [return-value]
annif/project.py:225: error: Incompatible return value type (got "Union[bool, datetime, None]", expected "Optional[datetime]")  [return-value]
Found 20 errors in 12 files (checked 55 source files)

When not ignoring any errors, mypy throws 182 of them.

Show all errors
annif/analyzer/analyzer.py:25: error: Skipping analyzing "nltk.tokenize": module is installed, but missing library stubs or py.typed marker  [import]
annif/analyzer/analyzer.py:25: error: Skipping analyzing "nltk": module is installed, but missing library stubs or py.typed marker  [import]
annif/openapi/validation.py:7: error: Skipping analyzing "connexion": module is installed, but missing library stubs or py.typed marker  [import]
annif/openapi/validation.py:8: error: Skipping analyzing "connexion.exceptions": module is installed, but missing library stubs or py.typed marker  [import]
annif/openapi/validation.py:9: error: Skipping analyzing "connexion.utils": module is installed, but missing library stubs or py.typed marker  [import]
annif/lexical/tokenset.py:44: error: Need type annotation for "_index"  [var-annotated]
annif/lexical/tokenset.py:58: error: Need type annotation for "subj_tsets" (hint: "subj_tsets: Dict[<type>, <type>] = ...")  [var-annotated]
annif/exception.py:45: error: Incompatible types in assignment (expression has type "str", base class "AnnifException" defined the type as "None")  [assignment]
annif/exception.py:51: error: Incompatible types in assignment (expression has type "str", base class "AnnifException" defined the type as "None")  [assignment]
annif/exception.py:58: error: Incompatible types in assignment (expression has type "str", base class "AnnifException" defined the type as "None")  [assignment]
annif/exception.py:64: error: Incompatible types in assignment (expression has type "str", base class "AnnifException" defined the type as "None")  [assignment]
annif/suggestion.py:9: error: Skipping analyzing "scipy.sparse": module is installed, but missing library stubs or py.typed marker  [import]
annif/suggestion.py:163: error: Argument 1 to "from_iterable" of "chain" has incompatible type "Iterable[SuggestionBatch]"; expected "Iterable[Iterable[<nothing>]]"  [arg-type]
annif/suggestion.py:163: error: Argument 1 to "from_iterable" of "chain" has incompatible type "Iterable[SuggestionBatch]"; expected "Iterable[Iterable[Any]]"  [arg-type]
annif/parallel.py:87: error: Incompatible types in assignment (expression has type "None", variable has type "int")  [assignment]
annif/lexical/util.py:9: error: Skipping analyzing "scipy.sparse": module is installed, but missing library stubs or py.typed marker  [import]
annif/lexical/util.py:24: error: "Node" has no attribute "language"  [attr-defined]
annif/eval.py:8: error: Skipping analyzing "scipy.sparse": module is installed, but missing library stubs or py.typed marker  [import]
annif/eval.py:8: error: Skipping analyzing "scipy": module is installed, but missing library stubs or py.typed marker  [import]
annif/eval.py:9: error: Skipping analyzing "sklearn.metrics": module is installed, but missing library stubs or py.typed marker  [import]
annif/eval.py:19: error: Skipping analyzing "scipy.sparse._arrays": module is installed, but missing library stubs or py.typed marker  [import]
annif/eval.py:84: error: Need type annotation for "_suggestion_arrays" (hint: "_suggestion_arrays: List[<type>] = ...")  [var-annotated]
annif/eval.py:85: error: Need type annotation for "_gold_subject_arrays" (hint: "_gold_subject_arrays: List[<type>] = ...")  [var-annotated]
annif/eval.py:105: error: "SubjectSet" has no attribute "__iter__" (not iterable)  [attr-defined]
annif/eval.py:233: error: "SubjectIndex" has no attribute "__iter__" (not iterable)  [attr-defined]
annif/eval.py:234: error: "SubjectIndex" has no attribute "__iter__" (not iterable)  [attr-defined]
annif/eval.py:267: error: Argument 4 to "output_result_per_subject" of "EvaluationBatch" has incompatible type "Optional[str]"; expected "str"  [arg-type]
annif/__init__.py:36: error: Skipping analyzing "connexion": module is installed, but missing library stubs or py.typed marker  [import]
annif/__init__.py:36: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
annif/util.py:19: error: Need type annotation for "logged" (hint: "logged: Set[<type>] = ...")  [var-annotated]
annif/backend/backend.py:118: error: Argument 1 to "int" has incompatible type "Optional[Any]"; expected "Union[str, bytes, bytearray, memoryview, array[Any], mmap, _CData, PickleBuffer, SupportsInt, SupportsIndex, SupportsTrunc]"  [arg-type]
annif/lexical/mllm.py:10: error: Skipping analyzing "joblib": module is installed, but missing library stubs or py.typed marker  [import]
annif/lexical/mllm.py:13: error: Skipping analyzing "sklearn.ensemble": module is installed, but missing library stubs or py.typed marker  [import]
annif/lexical/mllm.py:14: error: Skipping analyzing "sklearn.feature_extraction.text": module is installed, but missing library stubs or py.typed marker  [import]
annif/lexical/mllm.py:15: error: Skipping analyzing "sklearn.tree": module is installed, but missing library stubs or py.typed marker  [import]
annif/lexical/mllm.py:41: error: List or tuple literal expected as the second argument to "namedtuple()"  [misc]
annif/lexical/mllm.py:47: error: List or tuple literal expected as the second argument to "namedtuple()"  [misc]
annif/lexical/mllm.py:51: error: Second argument of IntEnum() must be string, tuple, list or dict literal for mypy to determine Enum members  [misc]
annif/lexical/mllm.py:65: error: Unexpected keyword argument "doc_length" for "Candidate"  [call-arg]
annif/lexical/mllm.py:65: error: Unexpected keyword argument "subject_id" for "Candidate"  [call-arg]
annif/lexical/mllm.py:65: error: Unexpected keyword argument "freq" for "Candidate"  [call-arg]
annif/lexical/mllm.py:65: error: Unexpected keyword argument "is_pref" for "Candidate"  [call-arg]
annif/lexical/mllm.py:65: error: Unexpected keyword argument "n_tokens" for "Candidate"  [call-arg]
annif/lexical/mllm.py:65: error: Unexpected keyword argument "ambiguity" for "Candidate"  [call-arg]
annif/lexical/mllm.py:65: error: Unexpected keyword argument "first_occ" for "Candidate"  [call-arg]
annif/lexical/mllm.py:65: error: Unexpected keyword argument "last_occ" for "Candidate"  [call-arg]
annif/lexical/mllm.py:65: error: Unexpected keyword argument "spread" for "Candidate"  [call-arg]
annif/lexical/mllm.py:112: error: "Candidate" has no attribute "subject_id"  [attr-defined]
annif/lexical/mllm.py:113: error: Value of type variable "_SCT" of "zeros" cannot be "bool"  [type-var]
annif/lexical/mllm.py:113: error: "ModelData" has no attribute "related"  [attr-defined]
annif/lexical/mllm.py:115: error: "ModelData" has no attribute "broader"  [attr-defined]
annif/lexical/mllm.py:116: error: "ModelData" has no attribute "narrower"  [attr-defined]
annif/lexical/mllm.py:117: error: "ModelData" has no attribute "related"  [attr-defined]
annif/lexical/mllm.py:118: error: "ModelData" has no attribute "collection"  [attr-defined]
annif/lexical/mllm.py:120: error: "Candidate" has no attribute "subject_id"  [attr-defined]
annif/lexical/mllm.py:121: error: "Type[Feature]" has no attribute "freq"  [attr-defined]
annif/lexical/mllm.py:121: error: "Candidate" has no attribute "freq"  [attr-defined]
annif/lexical/mllm.py:122: error: "Type[Feature]" has no attribute "doc_freq"  [attr-defined]
annif/lexical/mllm.py:122: error: "ModelData" has no attribute "doc_freq"  [attr-defined]
annif/lexical/mllm.py:123: error: "Type[Feature]" has no attribute "subj_freq"  [attr-defined]
annif/lexical/mllm.py:123: error: "ModelData" has no attribute "subj_freq"  [attr-defined]
annif/lexical/mllm.py:124: error: "Type[Feature]" has no attribute "tfidf"  [attr-defined]
annif/lexical/mllm.py:124: error: "Candidate" has no attribute "freq"  [attr-defined]
annif/lexical/mllm.py:124: error: "ModelData" has no attribute "idf"  [attr-defined]
annif/lexical/mllm.py:125: error: "Type[Feature]" has no attribute "is_pref"  [attr-defined]
annif/lexical/mllm.py:125: error: "Candidate" has no attribute "is_pref"  [attr-defined]
annif/lexical/mllm.py:126: error: "Type[Feature]" has no attribute "n_tokens"  [attr-defined]
annif/lexical/mllm.py:126: error: "Candidate" has no attribute "n_tokens"  [attr-defined]
annif/lexical/mllm.py:127: error: "Type[Feature]" has no attribute "ambiguity"  [attr-defined]
annif/lexical/mllm.py:127: error: "Candidate" has no attribute "ambiguity"  [attr-defined]
annif/lexical/mllm.py:128: error: "Type[Feature]" has no attribute "first_occ"  [attr-defined]
annif/lexical/mllm.py:128: error: "Candidate" has no attribute "first_occ"  [attr-defined]
annif/lexical/mllm.py:129: error: "Type[Feature]" has no attribute "last_occ"  [attr-defined]
annif/lexical/mllm.py:129: error: "Candidate" has no attribute "last_occ"  [attr-defined]
annif/lexical/mllm.py:130: error: "Type[Feature]" has no attribute "spread"  [attr-defined]
annif/lexical/mllm.py:130: error: "Candidate" has no attribute "spread"  [attr-defined]
annif/lexical/mllm.py:131: error: "Type[Feature]" has no attribute "doc_length"  [attr-defined]
annif/lexical/mllm.py:131: error: "Candidate" has no attribute "doc_length"  [attr-defined]
annif/lexical/mllm.py:132: error: "Type[Feature]" has no attribute "broader"  [attr-defined]
annif/lexical/mllm.py:133: error: "Type[Feature]" has no attribute "narrower"  [attr-defined]
annif/lexical/mllm.py:134: error: "Type[Feature]" has no attribute "related"  [attr-defined]
annif/lexical/mllm.py:135: error: "Type[Feature]" has no attribute "collection"  [attr-defined]
annif/lexical/mllm.py:160: error: Unexpected keyword argument "broader" for "ModelData"  [call-arg]
annif/lexical/mllm.py:160: error: Unexpected keyword argument "narrower" for "ModelData"  [call-arg]
annif/lexical/mllm.py:160: error: Unexpected keyword argument "related" for "ModelData"  [call-arg]
annif/lexical/mllm.py:160: error: Unexpected keyword argument "collection" for "ModelData"  [call-arg]
annif/lexical/mllm.py:160: error: Unexpected keyword argument "doc_freq" for "ModelData"  [call-arg]
annif/lexical/mllm.py:160: error: Unexpected keyword argument "subj_freq" for "ModelData"  [call-arg]
annif/lexical/mllm.py:160: error: Unexpected keyword argument "idf" for "ModelData"  [call-arg]
annif/lexical/mllm.py:247: error: Need type annotation for "_doc_freq"  [var-annotated]
annif/lexical/mllm.py:249: error: Need type annotation for "_subj_freq"  [var-annotated]
annif/lexical/mllm.py:351: error: "Candidate" has no attribute "subject_id"  [attr-defined]
annif/corpus/skos.py:26: error: Skipping analyzing "joblib": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/yake.py:11: error: Skipping analyzing "joblib": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/yake.py:12: error: Skipping analyzing "yake": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/yake.py:30: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/yake.py:39: error: Dict entry 1 has incompatible type "str": "float"; expected "str": "int"  [dict-item]
annif/backend/yake.py:40: error: Dict entry 2 has incompatible type "str": "str"; expected "str": "int"  [dict-item]
annif/backend/yake.py:43: error: Dict entry 5 has incompatible type "str": "None"; expected "str": "int"  [dict-item]
annif/backend/yake.py:44: error: Dict entry 6 has incompatible type "str": "List[str]"; expected "str": "int"  [dict-item]
annif/backend/yake.py:169: error: Item "None" of "Optional[Any]" has no attribute "get"  [union-attr]
annif/backend/stwfsa.py:6: error: Skipping analyzing "stwfsapy.predictor": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/stwfsa.py:32: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/stwfsa.py:50: error: Dict entry 0 has incompatible type "str": "str"; expected "str": "int"  [dict-item]
annif/backend/stwfsa.py:51: error: Dict entry 1 has incompatible type "str": "str"; expected "str": "int"  [dict-item]
annif/backend/stwfsa.py:52: error: Dict entry 2 has incompatible type "str": "str"; expected "str": "int"  [dict-item]
annif/backend/stwfsa.py:110: error: "object" not callable  [operator]
annif/backend/stwfsa.py:130: error: Item "None" of "Optional[Any]" has no attribute "suggest_proba"  [union-attr]
annif/backend/mixins.py:8: error: Skipping analyzing "joblib": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/mixins.py:9: error: Skipping analyzing "sklearn.feature_extraction.text": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/mixins.py:17: error: Skipping analyzing "scipy.sparse._csr": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/mixins.py:40: error: "ChunkingBackend" has no attribute "debug"  [attr-defined]
annif/backend/mixins.py:43: error: "ChunkingBackend" has no attribute "project"  [attr-defined]
annif/backend/mixins.py:44: error: "ChunkingBackend" has no attribute "debug"  [attr-defined]
annif/backend/mixins.py:49: error: "ChunkingBackend" has no attribute "debug"  [attr-defined]
annif/backend/mixins.py:64: error: "TfidfVectorizerMixin" has no attribute "datadir"  [attr-defined]
annif/backend/mixins.py:66: error: "TfidfVectorizerMixin" has no attribute "debug"  [attr-defined]
annif/backend/mixins.py:71: error: "TfidfVectorizerMixin" has no attribute "backend_id"  [attr-defined]
annif/backend/mixins.py:77: error: "TfidfVectorizerMixin" has no attribute "info"  [attr-defined]
annif/backend/mixins.py:81: error: "TfidfVectorizerMixin" has no attribute "datadir"  [attr-defined]
annif/backend/hyperopt.py:45: error: "Trial" has no attribute "value"  [attr-defined]
annif/backend/hyperopt.py:107: error: Argument "callbacks" to "optimize" of "Study" has incompatible type "List[Callable[[Study, Trial], None]]"; expected "Optional[List[Callable[[Study, FrozenTrial], None]]]"  [arg-type]
annif/backend/http.py:22: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/http.py:35: error: Signature of "is_trained" incompatible with supertype "AnnifBackend"  [override]
annif/backend/http.py:36: error: Incompatible return value type (got "Union[bool, str, None]", expected "Optional[bool]")  [return-value]
annif/backend/http.py:43: error: Argument 1 to "parse" has incompatible type "Union[bool, str]"; expected "Union[bytes, str, IO[str], IO[Any]]"  [arg-type]
annif/backend/dummy.py:15: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/corpus/subject.py:74: error: Incompatible types in assignment (expression has type "None", variable has type "Dict[str, Optional[str]]")  [assignment]
annif/corpus/subject.py:91: error: Item "None" of "Optional[List[str]]" has no attribute "__iter__" (not iterable)  [union-attr]
annif/corpus/subject.py:119: error: Need type annotation for "_subjects" (hint: "_subjects: List[<type>] = ...")  [var-annotated]
annif/corpus/subject.py:120: error: Need type annotation for "_uri_idx" (hint: "_uri_idx: Dict[<type>, <type>] = ...")  [var-annotated]
annif/corpus/subject.py:121: error: Need type annotation for "_label_idx" (hint: "_label_idx: Dict[<type>, <type>] = ...")  [var-annotated]
annif/corpus/subject.py:143: error: Incompatible types in assignment (expression has type "List[Any]", variable has type "None")  [assignment]
annif/corpus/subject.py:197: error: "None" has no attribute "__iter__" (not iterable)  [attr-defined]
annif/corpus/subject.py:202: error: "SubjectIndex" has no attribute "__iter__" (not iterable)  [attr-defined]
annif/backend/tfidf.py:9: error: Skipping analyzing "gensim.similarities": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/tfidf.py:9: error: Skipping analyzing "gensim": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/tfidf.py:10: error: Skipping analyzing "gensim.matutils": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/tfidf.py:21: error: Skipping analyzing "scipy.sparse._csr": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/tfidf.py:34: error: Need type annotation for "_buffer" (hint: "_buffer: List[<type>] = ...")  [var-annotated]
annif/backend/tfidf.py:67: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/tfidf.py:110: error: Item "None" of "Optional[Any]" has no attribute "vocabulary_"  [union-attr]
annif/backend/tfidf.py:136: error: Item "None" of "Optional[Any]" has no attribute "transform"  [union-attr]
annif/backend/tfidf.py:137: error: Value of type "Optional[Any]" is not indexable  [index]
annif/backend/svc.py:7: error: Skipping analyzing "joblib": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/svc.py:9: error: Skipping analyzing "scipy.special": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/svc.py:9: error: Skipping analyzing "scipy": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/svc.py:10: error: Skipping analyzing "sklearn.svm": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/svc.py:19: error: Skipping analyzing "scipy.sparse._csr": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/svc.py:27: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/svc.py:105: error: Item "None" of "Optional[Any]" has no attribute "classes_"  [union-attr]
annif/backend/svc.py:115: error: Item "None" of "Optional[Any]" has no attribute "transform"  [union-attr]
annif/backend/svc.py:116: error: Item "None" of "Optional[Any]" has no attribute "decision_function"  [union-attr]
annif/backend/omikuji.py:8: error: Skipping analyzing "omikuji": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/omikuji.py:21: error: Skipping analyzing "scipy.sparse._csr": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/omikuji.py:29: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/omikuji.py:80: error: Item "None" of "Optional[Any]" has no attribute "vocabulary_"  [union-attr]
annif/backend/omikuji.py:140: error: Item "None" of "Optional[Any]" has no attribute "transform"  [union-attr]
annif/backend/omikuji.py:143: note: (Skipping most remaining errors due to unresolved imports or missing stubs; fix these first)
annif/backend/mllm.py:7: error: Skipping analyzing "joblib": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/fasttext.py:8: error: Skipping analyzing "fasttext": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/fasttext.py:17: error: Skipping analyzing "fasttext.FastText": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/pav.py:10: error: Skipping analyzing "joblib": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/pav.py:12: error: Skipping analyzing "scipy.sparse": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/pav.py:13: error: Skipping analyzing "sklearn.isotonic": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/nn_ensemble.py:10: error: Skipping analyzing "joblib": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/nn_ensemble.py:11: error: Skipping analyzing "lmdb": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/nn_ensemble.py:13: error: Skipping analyzing "tensorflow.keras.backend": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/nn_ensemble.py:13: error: Skipping analyzing "tensorflow.keras": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/nn_ensemble.py:13: error: Skipping analyzing "tensorflow": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/nn_ensemble.py:14: error: Skipping analyzing "scipy.sparse": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/nn_ensemble.py:15: error: Skipping analyzing "tensorflow.keras.layers": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/nn_ensemble.py:16: error: Skipping analyzing "tensorflow.keras.models": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/nn_ensemble.py:17: error: Skipping analyzing "tensorflow.keras.utils": module is installed, but missing library stubs or py.typed marker  [import]
annif/backend/nn_ensemble.py:28: error: Skipping analyzing "tensorflow.python.framework.ops": module is installed, but missing library stubs or py.typed marker  [import]
annif/analyzer/voikko.py:6: error: Skipping analyzing "voikko.libvoikko": module is installed, but missing library stubs or py.typed marker  [import]
annif/analyzer/voikko.py:6: error: Skipping analyzing "voikko": module is installed, but missing library stubs or py.typed marker  [import]
annif/analyzer/snowball.py:14: error: Skipping analyzing "nltk.stem.snowball": module is installed, but missing library stubs or py.typed marker  [import]
annif/analyzer/snowball.py:14: error: Skipping analyzing "nltk.stem": module is installed, but missing library stubs or py.typed marker  [import]
annif/analyzer/snowball.py:14: error: Skipping analyzing "nltk": module is installed, but missing library stubs or py.typed marker  [import]
annif/rest.py:8: error: Skipping analyzing "connexion": module is installed, but missing library stubs or py.typed marker  [import]
annif/rest.py:18: error: Skipping analyzing "connexion.lifecycle": module is installed, but missing library stubs or py.typed marker  [import]
annif/cli_util.py:11: error: Skipping analyzing "click_log": module is installed, but missing library stubs or py.typed marker  [import]
annif/cli.py:13: error: Skipping analyzing "click_log": module is installed, but missing library stubs or py.typed marker  [import]
Found 182 errors in 32 files (checked 55 source files)
Closes #690.

@juhoinkinen juhoinkinen added this to the 1.0 milestone May 24, 2023
annif/project.py Fixed Show fixed Hide fixed
@codecov
Copy link

codecov bot commented May 24, 2023

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (4f6994b) 99.66% compared to head (252c75f) 99.67%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #708   +/-   ##
=======================================
  Coverage   99.66%   99.67%           
=======================================
  Files          89       89           
  Lines        6295     6380   +85     
=======================================
+ Hits         6274     6359   +85     
  Misses         21       21           
Impacted Files Coverage Δ
annif/__init__.py 100.00% <100.00%> (ø)
annif/analyzer/__init__.py 93.93% <100.00%> (+0.39%) ⬆️
annif/analyzer/analyzer.py 100.00% <100.00%> (ø)
annif/analyzer/simple.py 100.00% <100.00%> (ø)
annif/analyzer/simplemma.py 100.00% <100.00%> (ø)
annif/analyzer/snowball.py 100.00% <100.00%> (ø)
annif/analyzer/spacy.py 100.00% <100.00%> (ø)
annif/analyzer/voikko.py 100.00% <100.00%> (ø)
annif/backend/__init__.py 100.00% <100.00%> (ø)
annif/backend/backend.py 100.00% <100.00%> (ø)
... and 39 more

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@juhoinkinen juhoinkinen force-pushed the issue690-use-python-type-hints branch from 1ca7f9b to 7c3c5dc Compare May 24, 2023 14:03
@juhoinkinen juhoinkinen force-pushed the issue690-use-python-type-hints branch from a193ade to 2711ba4 Compare May 25, 2023 08:08
@juhoinkinen juhoinkinen force-pushed the issue690-use-python-type-hints branch 2 times, most recently from 12d710b to ab9e815 Compare May 25, 2023 09:31
@juhoinkinen juhoinkinen force-pushed the issue690-use-python-type-hints branch from ab9e815 to bb9951f Compare May 25, 2023 10:01
annif/backend/ensemble.py Fixed Show fixed Hide fixed
annif/backend/mllm.py Fixed Show fixed Hide fixed
@juhoinkinen
Copy link
Member Author

I annotated manually one module (annif.backend.hyperopt.py) that monkeytype was not able to process. These annif package modules are still lacking type hints altogether:

  • annif/backend/mixins.py
  • annif/cli.py
  • annif/corpus/document.py
  • annif/corpus/init.py
  • annif/default_config.py
  • annif/lexical/init.py
  • annif/openapi/init.py
  • annif/parallel.py
  • annif/util.py
  • annif/views.py

Not sure if they all need type hints.

Currently running mypy --ignore-missing-imports produces 154 errors...

Errors
annif/lexical/tokenset.py:44: error: Need type annotation for "_index"  [var-annotated]
annif/lexical/tokenset.py:58: error: Need type annotation for "subj_tsets" (hint: "subj_tsets: Dict[<type>, <type>] = ...")  [var-annotated]
annif/exception.py:47: error: Incompatible types in assignment (expression has type "str", base class "AnnifException" defined the type as "None")  [assignment]
annif/exception.py:53: error: Incompatible types in assignment (expression has type "str", base class "AnnifException" defined the type as "None")  [assignment]
annif/exception.py:60: error: Incompatible types in assignment (expression has type "str", base class "AnnifException" defined the type as "None")  [assignment]
annif/exception.py:66: error: Incompatible types in assignment (expression has type "str", base class "AnnifException" defined the type as "None")  [assignment]
annif/suggestion.py:157: error: Argument 1 to "SuggestionResults" has incompatible type "Generator[SuggestionBatch, None, None]"; expected "List[SuggestionBatch]"  [arg-type]
annif/suggestion.py:161: error: Argument 1 to "from_iterable" of "chain" has incompatible type "List[SuggestionBatch]"; expected "Iterable[Iterable[<nothing>]]"  [arg-type]
annif/suggestion.py:161: error: Argument 1 to "from_iterable" of "chain" has incompatible type "List[SuggestionBatch]"; expected "Iterable[Iterable[Any]]"  [arg-type]
annif/lexical/util.py:24: error: "Node" has no attribute "language"  [attr-defined]
annif/eval.py:85: error: Need type annotation for "_suggestion_arrays" (hint: "_suggestion_arrays: List[<type>] = ...")  [var-annotated]
annif/eval.py:86: error: Need type annotation for "_gold_subject_arrays" (hint: "_gold_subject_arrays: List[<type>] = ...")  [var-annotated]
annif/eval.py:97: error: Argument 1 to "from_sequence" of "SuggestionBatch" has incompatible type "Union[List[List[SubjectSuggestion]], List[Iterator[Any]]]"; expected "List[List[SubjectSuggestion]]"  [arg-type]
annif/eval.py:106: error: "SubjectSet" has no attribute "__iter__" (not iterable)  [attr-defined]
annif/eval.py:178: error: Incompatible types in assignment (expression has type "dict_keys[str, Callable[[], Any]]", variable has type "Sequence[str]")  [assignment]
annif/eval.py:234: error: "SubjectIndex" has no attribute "__iter__" (not iterable)  [attr-defined]
annif/eval.py:235: error: "SubjectIndex" has no attribute "__iter__" (not iterable)  [attr-defined]
annif/eval.py:268: error: Argument 4 to "output_result_per_subject" of "EvaluationBatch" has incompatible type "Optional[str]"; expected "str"  [arg-type]
annif/backend/backend.py:52: error: Name "datetime.datetime" is not defined  [name-defined]
annif/backend/backend.py:113: error: Argument 1 to "int" has incompatible type "Optional[Any]"; expected "Union[str, bytes, bytearray, memoryview, array[Any], mmap, _CData, PickleBuffer, SupportsInt, SupportsIndex, SupportsTrunc]"  [arg-type]
annif/lexical/mllm.py:39: error: List or tuple literal expected as the second argument to "namedtuple()"  [misc]
annif/lexical/mllm.py:45: error: List or tuple literal expected as the second argument to "namedtuple()"  [misc]
annif/lexical/mllm.py:49: error: Second argument of IntEnum() must be string, tuple, list or dict literal for mypy to determine Enum members  [misc]
annif/lexical/mllm.py:63: error: Unexpected keyword argument "doc_length" for "Candidate"  [call-arg]
annif/lexical/mllm.py:63: error: Unexpected keyword argument "subject_id" for "Candidate"  [call-arg]
annif/lexical/mllm.py:63: error: Unexpected keyword argument "freq" for "Candidate"  [call-arg]
annif/lexical/mllm.py:63: error: Unexpected keyword argument "is_pref" for "Candidate"  [call-arg]
annif/lexical/mllm.py:63: error: Unexpected keyword argument "n_tokens" for "Candidate"  [call-arg]
annif/lexical/mllm.py:63: error: Unexpected keyword argument "ambiguity" for "Candidate"  [call-arg]
annif/lexical/mllm.py:63: error: Unexpected keyword argument "first_occ" for "Candidate"  [call-arg]
annif/lexical/mllm.py:63: error: Unexpected keyword argument "last_occ" for "Candidate"  [call-arg]
annif/lexical/mllm.py:63: error: Unexpected keyword argument "spread" for "Candidate"  [call-arg]
annif/lexical/mllm.py:110: error: "Candidate" has no attribute "subject_id"  [attr-defined]
annif/lexical/mllm.py:111: error: Value of type variable "_SCT" of "zeros" cannot be "bool"  [type-var]
annif/lexical/mllm.py:111: error: "ModelData" has no attribute "related"  [attr-defined]
annif/lexical/mllm.py:113: error: "ModelData" has no attribute "broader"  [attr-defined]
annif/lexical/mllm.py:114: error: "ModelData" has no attribute "narrower"  [attr-defined]
annif/lexical/mllm.py:115: error: "ModelData" has no attribute "related"  [attr-defined]
annif/lexical/mllm.py:116: error: "ModelData" has no attribute "collection"  [attr-defined]
annif/lexical/mllm.py:118: error: "Candidate" has no attribute "subject_id"  [attr-defined]
annif/lexical/mllm.py:119: error: "Type[Feature]" has no attribute "freq"  [attr-defined]
annif/lexical/mllm.py:119: error: "Candidate" has no attribute "freq"  [attr-defined]
annif/lexical/mllm.py:120: error: "Type[Feature]" has no attribute "doc_freq"  [attr-defined]
annif/lexical/mllm.py:120: error: "ModelData" has no attribute "doc_freq"  [attr-defined]
annif/lexical/mllm.py:121: error: "Type[Feature]" has no attribute "subj_freq"  [attr-defined]
annif/lexical/mllm.py:121: error: "ModelData" has no attribute "subj_freq"  [attr-defined]
annif/lexical/mllm.py:122: error: "Type[Feature]" has no attribute "tfidf"  [attr-defined]
annif/lexical/mllm.py:122: error: "Candidate" has no attribute "freq"  [attr-defined]
annif/lexical/mllm.py:122: error: "ModelData" has no attribute "idf"  [attr-defined]
annif/lexical/mllm.py:123: error: "Type[Feature]" has no attribute "is_pref"  [attr-defined]
annif/lexical/mllm.py:123: error: "Candidate" has no attribute "is_pref"  [attr-defined]
annif/lexical/mllm.py:124: error: "Type[Feature]" has no attribute "n_tokens"  [attr-defined]
annif/lexical/mllm.py:124: error: "Candidate" has no attribute "n_tokens"  [attr-defined]
annif/lexical/mllm.py:125: error: "Type[Feature]" has no attribute "ambiguity"  [attr-defined]
annif/lexical/mllm.py:125: error: "Candidate" has no attribute "ambiguity"  [attr-defined]
annif/lexical/mllm.py:126: error: "Type[Feature]" has no attribute "first_occ"  [attr-defined]
annif/lexical/mllm.py:126: error: "Candidate" has no attribute "first_occ"  [attr-defined]
annif/lexical/mllm.py:127: error: "Type[Feature]" has no attribute "last_occ"  [attr-defined]
annif/lexical/mllm.py:127: error: "Candidate" has no attribute "last_occ"  [attr-defined]
annif/lexical/mllm.py:128: error: "Type[Feature]" has no attribute "spread"  [attr-defined]
annif/lexical/mllm.py:128: error: "Candidate" has no attribute "spread"  [attr-defined]
annif/lexical/mllm.py:129: error: "Type[Feature]" has no attribute "doc_length"  [attr-defined]
annif/lexical/mllm.py:129: error: "Candidate" has no attribute "doc_length"  [attr-defined]
annif/lexical/mllm.py:130: error: "Type[Feature]" has no attribute "broader"  [attr-defined]
annif/lexical/mllm.py:131: error: "Type[Feature]" has no attribute "narrower"  [attr-defined]
annif/lexical/mllm.py:132: error: "Type[Feature]" has no attribute "related"  [attr-defined]
annif/lexical/mllm.py:133: error: "Type[Feature]" has no attribute "collection"  [attr-defined]
annif/lexical/mllm.py:158: error: Unexpected keyword argument "broader" for "ModelData"  [call-arg]
annif/lexical/mllm.py:158: error: Unexpected keyword argument "narrower" for "ModelData"  [call-arg]
annif/lexical/mllm.py:158: error: Unexpected keyword argument "related" for "ModelData"  [call-arg]
annif/lexical/mllm.py:158: error: Unexpected keyword argument "collection" for "ModelData"  [call-arg]
annif/lexical/mllm.py:158: error: Unexpected keyword argument "doc_freq" for "ModelData"  [call-arg]
annif/lexical/mllm.py:158: error: Unexpected keyword argument "subj_freq" for "ModelData"  [call-arg]
annif/lexical/mllm.py:158: error: Unexpected keyword argument "idf" for "ModelData"  [call-arg]
annif/lexical/mllm.py:245: error: Need type annotation for "_doc_freq"  [var-annotated]
annif/lexical/mllm.py:247: error: Need type annotation for "_subj_freq"  [var-annotated]
annif/lexical/mllm.py:349: error: "Candidate" has no attribute "subject_id"  [attr-defined]
annif/corpus/skos.py:93: error: Invalid index type "str" for "Union[defaultdict[str, List[str]], defaultdict[None, List[str]]]"; expected type "None"  [index]
annif/corpus/skos.py:94: error: Invalid index type "str" for "Union[defaultdict[str, List[str]], defaultdict[None, List[str]]]"; expected type "None"  [index]
annif/corpus/skos.py:95: error: Invalid index type "None" for "Union[defaultdict[str, List[str]], defaultdict[None, List[str]]]"; expected type "str"  [index]
annif/corpus/skos.py:96: error: Invalid index type "None" for "Union[defaultdict[str, List[str]], defaultdict[None, List[str]]]"; expected type "str"  [index]
annif/backend/yake.py:31: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/yake.py:40: error: Dict entry 1 has incompatible type "str": "float"; expected "str": "int"  [dict-item]
annif/backend/yake.py:41: error: Dict entry 2 has incompatible type "str": "str"; expected "str": "int"  [dict-item]
annif/backend/yake.py:44: error: Dict entry 5 has incompatible type "str": "None"; expected "str": "int"  [dict-item]
annif/backend/yake.py:45: error: Dict entry 6 has incompatible type "str": "List[str]"; expected "str": "int"  [dict-item]
annif/backend/yake.py:170: error: Item "None" of "Optional[Any]" has no attribute "get"  [union-attr]
annif/backend/yake.py:173: error: Argument 2 to "max" has incompatible type "int"; expected "floating[_64Bit]"  [arg-type]
annif/backend/yake.py:177: error: "list" expects 1 type argument, but 2 given  [type-arg]
annif/backend/yake.py:178: error: "list" expects 1 type argument, but 2 given  [type-arg]
annif/backend/stwfsa.py:32: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/stwfsa.py:50: error: Dict entry 0 has incompatible type "str": "str"; expected "str": "int"  [dict-item]
annif/backend/stwfsa.py:51: error: Dict entry 1 has incompatible type "str": "str"; expected "str": "int"  [dict-item]
annif/backend/stwfsa.py:52: error: Dict entry 2 has incompatible type "str": "str"; expected "str": "int"  [dict-item]
annif/backend/stwfsa.py:110: error: "object" not callable  [operator]
annif/backend/stwfsa.py:130: error: Item "None" of "Optional[Any]" has no attribute "suggest_proba"  [union-attr]
annif/backend/hyperopt.py:45: error: "Trial" has no attribute "value"  [attr-defined]
annif/backend/hyperopt.py:107: error: Argument "callbacks" to "optimize" of "Study" has incompatible type "List[Callable[[Study, Trial], None]]"; expected "Optional[List[Callable[[Study, FrozenTrial], None]]]"  [arg-type]
annif/backend/http.py:8: error: Library stubs not installed for "dateutil.parser"  [import]
annif/backend/http.py:8: note: Hint: "python3 -m pip install types-python-dateutil"
annif/backend/http.py:8: note: (or run "mypy --install-types" to install all missing stub packages)
annif/backend/http.py:8: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
annif/backend/http.py:8: error: Library stubs not installed for "dateutil"  [import]
annif/backend/http.py:9: error: Library stubs not installed for "requests"  [import]
annif/backend/http.py:10: error: Library stubs not installed for "requests.exceptions"  [import]
annif/backend/http.py:10: note: Hint: "python3 -m pip install types-requests"
annif/backend/http.py:22: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/http.py:36: error: Incompatible return value type (got "Union[bool, str, None]", expected "bool")  [return-value]
annif/backend/dummy.py:15: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/corpus/subject.py:44: error: The return type of a generator function should be "Generator" or one of its supertypes  [misc]
annif/corpus/subject.py:52: error: Argument 1 to "serialize_subjects_to_skos" has incompatible type "None"; expected "Iterator[Any]"  [arg-type]
annif/corpus/subject.py:72: error: Incompatible types in assignment (expression has type "None", variable has type "Dict[str, Optional[str]]")  [assignment]
annif/corpus/subject.py:89: error: Item "None" of "Optional[List[str]]" has no attribute "__iter__" (not iterable)  [union-attr]
annif/corpus/subject.py:94: error: The return type of a generator function should be "Generator" or one of its supertypes  [misc]
annif/corpus/subject.py:103: error: Argument 1 to "serialize_subjects_to_skos" has incompatible type "None"; expected "Iterator[Any]"  [arg-type]
annif/corpus/subject.py:117: error: Need type annotation for "_subjects" (hint: "_subjects: List[<type>] = ...")  [var-annotated]
annif/corpus/subject.py:118: error: Need type annotation for "_uri_idx" (hint: "_uri_idx: Dict[<type>, <type>] = ...")  [var-annotated]
annif/corpus/subject.py:119: error: Need type annotation for "_label_idx" (hint: "_label_idx: Dict[<type>, <type>] = ...")  [var-annotated]
annif/corpus/subject.py:134: error: Incompatible return value type (got "None", expected "List[str]")  [return-value]
annif/corpus/subject.py:141: error: Incompatible types in assignment (expression has type "List[Any]", variable has type "None")  [assignment]
annif/corpus/subject.py:195: error: "None" has no attribute "__iter__" (not iterable)  [attr-defined]
annif/corpus/subject.py:200: error: "SubjectIndex" has no attribute "__iter__" (not iterable)  [attr-defined]
annif/corpus/subject.py:241: error: Argument 1 of "__eq__" is incompatible with supertype "object"; supertype defines the argument type as "object"  [override]
annif/corpus/subject.py:241: note: This violates the Liskov substitution principle
annif/corpus/subject.py:241: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
annif/corpus/subject.py:241: note: It is recommended for "__eq__" to work with arbitrary objects, for example:
annif/corpus/subject.py:241: note:     def __eq__(self, other: object) -> bool:
annif/corpus/subject.py:241: note:         if not isinstance(other, SubjectSet):
annif/corpus/subject.py:241: note:             return NotImplemented
annif/corpus/subject.py:241: note:         return <logic to compare two SubjectSet instances>
annif/backend/tfidf.py:32: error: Need type annotation for "_buffer" (hint: "_buffer: List[<type>] = ...")  [var-annotated]
annif/backend/tfidf.py:65: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/tfidf.py:108: error: "None" has no attribute "vocabulary_"  [attr-defined]
annif/backend/tfidf.py:134: error: "None" has no attribute "transform"  [attr-defined]
annif/backend/tfidf.py:135: error: Value of type "Optional[Any]" is not indexable  [index]
annif/backend/svc.py:27: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/svc.py:105: error: Item "None" of "Optional[Any]" has no attribute "classes_"  [union-attr]
annif/backend/svc.py:115: error: "None" has no attribute "transform"  [attr-defined]
annif/backend/svc.py:116: error: Item "None" of "Optional[Any]" has no attribute "decision_function"  [union-attr]
annif/backend/omikuji.py:29: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/omikuji.py:80: error: "None" has no attribute "vocabulary_"  [attr-defined]
annif/backend/omikuji.py:140: error: "None" has no attribute "transform"  [attr-defined]
annif/backend/omikuji.py:143: error: Need type annotation for "batch_results" (hint: "batch_results: List[<type>] = ...")  [var-annotated]
annif/backend/omikuji.py:150: error: Item "None" of "Optional[Any]" has no attribute "predict"  [union-attr]
annif/backend/mllm.py:32: error: "AnnifBackend" has no attribute "_load_train_data"  [attr-defined]
annif/backend/mllm.py:38: error: "AnnifBackend" has no attribute "_generate_candidates"  [attr-defined]
annif/backend/mllm.py:42: error: Name "np.float" is not defined  [name-defined]
annif/backend/mllm.py:49: error: "AnnifBackend" has no attribute "_model"  [attr-defined]
annif/backend/mllm.py:55: error: "AnnifBackend" has no attribute "_model"  [attr-defined]
annif/backend/mllm.py:57: error: "AnnifBackend" has no attribute "_model"  [attr-defined]
annif/backend/mllm.py:60: error: "AnnifBackend" has no attribute "_prediction_to_result"  [attr-defined]
annif/backend/mllm.py:78: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/mllm.py:89: error: Dict entry 2 has incompatible type "str": "float"; expected "str": "int"  [dict-item]
annif/backend/mllm.py:93: error: Signature of "get_hp_optimizer" incompatible with supertype "AnnifHyperoptBackend"  [override]
annif/backend/mllm.py:93: note:      Superclass:
annif/backend/mllm.py:93: note:          def get_hp_optimizer(self, corpus: DocumentCorpus) -> Any
annif/backend/mllm.py:93: note:      Subclass:
annif/backend/mllm.py:93: note:          def get_hp_optimizer(self, corpus: DocumentCorpus, metric: str) -> MLLMOptimizer
annif/backend/mllm.py:156: error: Item "None" of "Optional[MLLMModel]" has no attribute "generate_candidates"  [union-attr]
annif/backend/mllm.py:170: error: Item "None" of "Optional[MLLMModel]" has no attribute "predict"  [union-attr]
annif/backend/fasttext.py:26: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/fasttext.py:48: error: Dict entry 1 has incompatible type "str": "float"; expected "str": "int"  [dict-item]
annif/backend/fasttext.py:50: error: Dict entry 3 has incompatible type "str": "str"; expected "str": "int"  [dict-item]
annif/backend/fasttext.py:156: error: Item "None" of "Optional[Any]" has no attribute "predict"  [union-attr]
annif/backend/fasttext.py:170: error: Need type annotation for "label_scores"  [var-annotated]
annif/backend/ensemble.py:135: error: Incompatible return value type (got "Union[floating[_64Bit], float]", expected "float")  [return-value]
annif/backend/ensemble.py:145: error: Incompatible types in assignment (expression has type "str", base class "AnnifBackend" defined the type as "None")  [assignment]
annif/backend/ensemble.py:155: error: No return value expected  [return-value]
annif/backend/ensemble.py:157: error: Signature of "get_hp_optimizer" incompatible with supertype "AnnifHyperoptBackend"  [override]
annif/backend/ensemble.py:157: note:      Superclass:
annif/backend/ensemble.py:157: note:          def get_hp_optimizer(self, corpus: DocumentCorpus) -> Any
annif/backend/ensemble.py:157: note:      Subclass:
annif/backend/ensemble.py:157: note: (Skipping most remaining errors due to unresolved imports or missing stubs; fix these first)
Found 154 errors in 20 files (checked 55 source files)

@juhoinkinen
Copy link
Member Author

Now I noticed that the hints of the standard collection types do not need to be from the ones from the typing module, but I could use the built-in types, when the from __future__ import annotations is used (PEP585).

Also, it could be considered to try using type hints only with Python 3.10. That would allow simpler syntax for Union[] and dropping using Optional[] (PEP604).

@osma
Copy link
Member

osma commented May 26, 2023

Also, it could be considered to try using type hints only with Python 3.10.

Sounds promising, but would the code still run with Python 3.8 and 3.9 if it included type hints using the new syntax?

@juhoinkinen
Copy link
Member Author

Also, it could be considered to try using type hints only with Python 3.10.

Sounds promising, but would the code still run with Python 3.8 and 3.9 if it included type hints using the new syntax?

Changing Union[x, y] to x | y in few places did not make the unit tests fail on Python 3.8, but did not even affect mypy runs...

@osma
Copy link
Member

osma commented May 26, 2023

I think we should use the simplest, most elegant, most modern approach that works with Python 3.8 (the annotations don't have to work, but the code needs to run). So starting directly with PEP 585 and PEP 604 would make sense if that can be done.

@juhoinkinen
Copy link
Member Author

I think we should use the simplest, most elegant, most modern approach that works with Python 3.8 (the annotations don't have to work, but the code needs to run). So starting directly with PEP 585 and PEP 604 would make sense if that can be done.

Using the from __future__ import annotations postpones the evaluation of annotations, so the annotations "can contain whatever", so PEP 585 and 604 can be used even with Python 3.8.

And it seems mypy also supports the PEP 585 and 604 features/syntax also on Python 3.8 from mypy version 0.800 onwards (when using the mentioned import).

- Use standard collection types instead of types from Typing (PEP 585)
- Write union types as X | Y (PEP 604)
- Write optional values as X | None (PEP 604)
return self.vocab.subjects

def _get_info(self, key):
def _get_info(self, key: str) -> bool | datetime | None:

Check notice

Code scanning / CodeQL

Explicit returns mixed with implicit (fall through) returns Note

Mixing implicit and explicit returns may indicate an error as implicit returns always return None.
def __init__(
self,
backend_id: str,
config_params: dict[str, Any] | SectionProxy,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This union could be replaced by a MutableMapping, but the current dict[str, Any] | SectionProxy union seemed more clear.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should convert objects from configparser module into plain dicts before passing them into the backend? I think that would be cleaner

@sonarcloud
Copy link

sonarcloud bot commented Jun 2, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@juhoinkinen
Copy link
Member Author

juhoinkinen commented Jun 2, 2023

I think this could be merged now, and with a squash merge. This just adds type hints to function/method signatures and is already touching quite many lines. However, runtime behaviour should not be affected.

There are many mypy errors (see PR description/first comment), but resolving many of them would need modifying actual code (which could then affect runtime, e.g. "X" has no attribute "__iter__" (not iterable), Incompatible return value type (got "Union[bool, str, None]", expected "Optional[bool]") [return-value]), or at least annotating variables inside functions. Another PR could be more appropriate for this.

BTW, mypyc is the tool that

compiles Python modules to C extensions. It uses standard Python type hints to generate fast code.

@juhoinkinen juhoinkinen marked this pull request as ready for review June 2, 2023 12:14
Copy link
Member

@osma osma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, let's leave the next steps (e.g. fixing mypy warnings, possible use of mypyc) to subsequent PRs. I agree that a squash merge makes sense.

@juhoinkinen juhoinkinen merged commit 579701c into main Jun 5, 2023
13 checks passed
@juhoinkinen juhoinkinen deleted the issue690-use-python-type-hints branch June 5, 2023 08:47
@juhoinkinen juhoinkinen changed the title Issue690 use python type hints Use Python type hints Aug 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use Python type hints
2 participants