Mypy checking for build.py by Volker-Weissmann · Pull Request #11740 · mesonbuild/meson

Volker-Weissmann · 2023-04-26T20:56:35Z

run_mypy.py currently does not check mesonbuild/build.py, much to my distaste. Currently, mypy would show a lot of warnings in build.py if it did.

I fixed all but one mypy warning in build.py.
Some of these mypy warnings hinted at real bugs, I fixed those.
The 3 test cases I added were inspired by the mypy warnings and my experiences fixing them.

There is still one mypy warning left:

mesonbuild/build.py:2125:40: error: Argument 2 to "join" has incompatible type "Union[str, bool]"; expected "str"  [arg-type]

That one hints at a real bug that I can use to create a crash. See #11748.

What do you think? Should I do ninjabackend.py next? I suspect that #1633 is hiding in the mypy warnings of ninjabackend.py

And no, I'm not gaming any LOC metric here. :-)

eli-schwartz

Only took a quick look, here are my initial thoughts.

eli-schwartz · 2023-04-26T21:41:24Z

        self.sources_map[f] = o

+    def is_linkable_output(self, output: str) -> bool:
+        return False


This is annoying and happens because for the sake of actually implementing it we base this off of BuildTarget, but for proper semantics / type safety we really would be better off if it was a CustomTarget.

Should I do something about that before you merge, or are you just informing me on the state of meson.

eli-schwartz · 2023-04-26T21:48:17Z

-            str, build.BuildTarget, build.CustomTarget, build.CustomTargetIndex,
-            build.ExtractedObjects, build.GeneratedList, ExternalProgram,
-            mesonlib.File]]
+            str, mesonlib.File, build.CustomTarget, build.CustomTargetIndex, build.GeneratedList]]


What happens if you do pass a BuildTarget or extracted object or some such in, here? Why is it more correct to change the allowance?

Passing ExternalProgram or ExtractedObjects results in an unhandled exception before this PR.

As to why I also removed build.BuildTarget:
The argument will be passed to msgfmt or itstool eventually.

These tools accept *.mo, *.xml and *.desktop files. (They don't check the filename, but just care about the content.) I don't see a way how a BuildTarget could also be a valid and useful *.mo or *.xml or *.desktop file.

If you pass an ELF file to msgfmt, it will print warnings, and output an empty file.

eli-schwartz · 2023-04-26T21:50:41Z

            assert len(t.install_tag) == num_out
+            for x in outdirs_unchecked:
+                assert not x is True
+            outdirs = T.cast(T.List[T.Union[str, T.Literal[False]]], outdirs_unchecked)


Suggested change

outdirs = T.cast(T.List[T.Union[str, T.Literal[False]]], outdirs_unchecked)

outdirs = T.cast('T.List[T.Union[str, T.Literal[False]]]', outdirs_unchecked)

T.cast always needs to quote the type annotation being used -- otherwise it gets evaluated at runtime. It is not affected by from __future__ import annotations.

Yes, that's what made the tests fail.

eli-schwartz · 2023-04-26T21:52:34Z

        pickle.dump(self.create_test_serialisation(tests), datafile)

-    def construct_target_rel_paths(self, t: T.Union[build.Target, build.CustomTargetIndex], workdir: T.Optional[str]) -> T.List[str]:
+    def construct_target_rel_paths(self, t: T.Union[build.Target, build.CustomTargetIndex], workdir: T.Optional[str]) -> T.Sequence[str]:


Sequences of str are bad. Like, really bad. T.List[str] will accept ['foo', 'bar'] but T.Sequence[str] will accept that and also 'foo', because a string is a sequence of single-character strings.

What exactly is wrong with it as it is right now?

T.Sequence[str] will accept that and also 'foo'

I did not know that.

What exactly is wrong with it as it is right now?

I think we are facing a dilemma:

You cannot implicitly convert T.List[str] to T.List[T.Union[str, other]] or mypy will complain

You can implicitly convert T.Sequence[str] to T.Sequence[T.Union[str, other]], but as you said, sequences of str's are bad

You cannot implicitly convert ImmutableListProtocol[str] to ImmutableListProtocol[T.Union[str, other]]

You could copy the list, but that means a performance hit

You could insert a T.cast call, but that is ugly and disables many mypy checks.

I think I will be going with the last one.

eli-schwartz · 2023-04-26T21:59:42Z

        # TODO: if it's possible to annotate get_option or validate_option_value
        # in the future we might be able to remove the cast here
-        return T.cast('T.Union[str, int, bool, WrapMode]', self.options[key].value)
+        return T.cast(T.Union[str, int, bool, 'WrapMode'], self.options[key].value)


Don't remove these quotes.

(Actually, it is not strictly necessary to remove any quotes. Usually it doesn't particularly hurt, it just takes a bit to look at each change.)

eli-schwartz · 2023-04-26T22:00:02Z

        # provided by typed_kwargs
        if isinstance(opts, dict):
-            return T.cast('T.Dict[OptionKey, str]', opts)
+            return T.cast(T.Dict[OptionKey, str], opts)


eli-schwartz · 2023-04-26T22:03:31Z

            sources: T.List['SourceOutputs'],
            structured_sources: T.Optional[StructuredSources],
-            objects: T.List[ObjectTypes],
+            objects: T.Sequence[T.Union[str, File, 'ExtractedObjects']],


I would think we need a new type alias for this. Also, list > sequence.

Feed a custom_target output to cc.preprocess, then feed that output to a generator.

…tgt.extract_objects`

`get_transitive_link_deps_mapping` has an `@lru_cache`. Therefore, we should never modify its output, only copies of its output.

…tstool_join`

Volker-Weissmann · 2023-04-30T19:27:47Z

I fixed everything you told me to and made the test pass.

tristan957 · 2023-04-30T22:27:42Z

@dcbaker does your PR make mypy happy for build.py?

dcbaker · 2023-05-03T21:38:32Z

@tristan957 no, but it does fix a lot of mypy issues that aren't apparent because it's not 100% clear what's coming in from the interpreter. There's a lot of stuff in there that's pretty broken.

I'm also solving the Sequence thing, but just enforcing that all string paths are converted to files in the Interpreter, so that the backend never has to deal with them. As an added bonus that makes sequences safe to use because we don't have strings to deal with

bruchar1

With annotations from __future__, you shouldn't need to quote types. I know many types are quoted for historical reasons, but if we touch them, I think it's better to correct them.

bruchar1 · 2023-05-04T12:04:40Z

                raise InvalidArguments(f'Bad object of type {type(s).__name__!r} in target {self.name!r}.')

-    def process_sourcelist(self, sources: T.List['SourceOutputs']) -> None:
+    def process_sourcelist(self, sources: ImmutableListProtocol['SourceOutputs']) -> None:


Suggested change

def process_sourcelist(self, sources: ImmutableListProtocol['SourceOutputs']) -> None:

def process_sourcelist(self, sources: ImmutableListProtocol[SourceOutputs]) -> None:

bruchar1 · 2023-05-04T12:05:22Z


    def extract_all_objects_recurse(self) -> T.List[T.Union[str, 'ExtractedObjects']]:
-        objs = [self.extract_all_objects()]
+        objs: T.List[T.Union[str, 'ExtractedObjects']] = [self.extract_all_objects()]


Suggested change

objs: T.List[T.Union[str, 'ExtractedObjects']] = [self.extract_all_objects()]

objs: T.List[T.Union[str, ExtractedObjects]] = [self.extract_all_objects()]

bruchar1 · 2023-05-04T12:06:51Z


    def extract_all_objects_recurse(self) -> T.List[T.Union[str, 'ExtractedObjects']]:
-        return self.get_outputs()
+        return T.cast('T.List[T.Union[str, ExtractedObjects]]', self.get_outputs())


Suggested change

return T.cast('T.List[T.Union[str, ExtractedObjects]]', self.get_outputs())

return T.cast(T.List[T.Union[str, ExtractedObjects]], self.get_outputs())

bruchar1 · 2023-05-04T12:07:03Z

        return CustomTargetIndex(self, self.outputs[index])

-    def __setitem__(self, index, value):
+    def __setitem__(self, index: int, value: 'CustomTargetIndex') -> None:


Suggested change

def __setitem__(self, index: int, value: 'CustomTargetIndex') -> None:

def __setitem__(self, index: int, value: CustomTargetIndex) -> None:

bruchar1 · 2023-05-04T12:07:45Z

        # If None, self.link_args will be used
        self.raw_link_args: T.Optional[T.List[str]] = None
-        self.sources: T.List[T.Union['FileOrString', 'CustomTarget', 'StructuredSources']] = []
+        self.sources: T.List[T.Union['File', 'CustomTarget', 'StructuredSources']] = []


Suggested change

self.sources: T.List[T.Union['File', 'CustomTarget', 'StructuredSources']] = []

self.sources: T.List[T.Union[File, CustomTarget, StructuredSources]] = []

bruchar1 · 2023-05-04T12:08:06Z

        return self.is_found

-    def get_sources(self) -> T.List[T.Union['FileOrString', 'CustomTarget', 'StructuredSources']]:
+    def get_sources(self) -> ImmutableListProtocol[T.Union['File', 'CustomTarget', 'StructuredSources']]:


Suggested change

def get_sources(self) -> ImmutableListProtocol[T.Union['File', 'CustomTarget', 'StructuredSources']]:

def get_sources(self) -> ImmutableListProtocol[T.Union[File, CustomTarget, StructuredSources]]:

bruchar1 · 2023-05-04T12:08:38Z

+    environment: 'Environment'
+    subdir: 'str'


Suggested change

environment: 'Environment'

subdir: 'str'

environment: Environment

subdir: str

bruchar1 · 2023-05-04T12:09:05Z

        extra_args = kwargs['args']
        targets: T.List['Target'] = []
-        gmotargets: T.List['build.CustomTarget'] = []
+        gmotargets: T.List['build.Target'] = []


Suggested change

gmotargets: T.List['build.Target'] = []

gmotargets: T.List[build.Target] = []

bruchar1 · 2023-05-04T12:09:26Z


 def typeslistify(item: 'T.Union[_T, T.Sequence[_T]]',
-                 types: 'T.Union[T.Type[_T], T.Tuple[T.Type[_T]]]') -> T.List[_T]:
+                 types: 'T.Union[T.Type, T.Tuple[T.Type, ...]]') -> T.List[_T]:


Suggested change

types: 'T.Union[T.Type, T.Tuple[T.Type, ...]]') -> T.List[_T]:

types: T.Union[T.Type, T.Tuple[T.Type, ...]]) -> T.List[_T]:

Volker-Weissmann requested a review from jpakkane as a code owner April 26, 2023 20:56

Volker-Weissmann force-pushed the typehints_1 branch from 25e95dd to f76b790 Compare April 26, 2023 21:04

eli-schwartz reviewed Apr 26, 2023

View reviewed changes

eli-schwartz added the typing label Apr 27, 2023

Volker-Weissmann mentioned this pull request Apr 28, 2023

install and install_dir is broken #11748

Open

Volker-Weissmann marked this pull request as draft April 28, 2023 15:01

Volker-Weissmann force-pushed the typehints_1 branch 3 times, most recently from 8022b0b to 439ffe8 Compare April 30, 2023 13:11

Volker-Weissmann added 2 commits April 30, 2023 15:34

Add a testcase

fcd3948

Feed a custom_target output to cc.preprocess, then feed that output to a generator.

Fix error message when an unknown generated file is passed to `build_…

a35a9ef

…tgt.extract_objects`

Volker-Weissmann force-pushed the typehints_1 branch 5 times, most recently from 191d324 to 8d65d71 Compare April 30, 2023 16:26

Volker-Weissmann added 5 commits April 30, 2023 19:32

Fix error message when attempting to link to a preprocessed source file

781c1dd

Fix potential get_transitive_link_deps_mapping caching bug

e918a64

`get_transitive_link_deps_mapping` has an `@lru_cache`. Therefore, we should never modify its output, only copies of its output.

Minor cleanup: Remove get_java_args

b95cd4d

Fix accepted types for the input arg of i18n.merge_file and `i18n.i…

e61a54c

…tstool_join`

Fix all mypy warnings in mesonbuild/build.py

decd8ab

Volker-Weissmann force-pushed the typehints_1 branch from 8d65d71 to decd8ab Compare April 30, 2023 17:33

Volker-Weissmann marked this pull request as ready for review April 30, 2023 19:27

bruchar1 reviewed May 4, 2023

View reviewed changes

jpakkane force-pushed the master branch from 00d6953 to 6447832 Compare August 19, 2025 20:51

bonzini closed this May 8, 2026

	outdirs = T.cast(T.List[T.Union[str, T.Literal[False]]], outdirs_unchecked)
	outdirs = T.cast('T.List[T.Union[str, T.Literal[False]]]', outdirs_unchecked)

	def process_sourcelist(self, sources: ImmutableListProtocol['SourceOutputs']) -> None:
	def process_sourcelist(self, sources: ImmutableListProtocol[SourceOutputs]) -> None:

	objs: T.List[T.Union[str, 'ExtractedObjects']] = [self.extract_all_objects()]
	objs: T.List[T.Union[str, ExtractedObjects]] = [self.extract_all_objects()]

	return T.cast('T.List[T.Union[str, ExtractedObjects]]', self.get_outputs())
	return T.cast(T.List[T.Union[str, ExtractedObjects]], self.get_outputs())

	def __setitem__(self, index: int, value: 'CustomTargetIndex') -> None:
	def __setitem__(self, index: int, value: CustomTargetIndex) -> None:

	self.sources: T.List[T.Union['File', 'CustomTarget', 'StructuredSources']] = []
	self.sources: T.List[T.Union[File, CustomTarget, StructuredSources]] = []

	def get_sources(self) -> ImmutableListProtocol[T.Union['File', 'CustomTarget', 'StructuredSources']]:
	def get_sources(self) -> ImmutableListProtocol[T.Union[File, CustomTarget, StructuredSources]]:

	gmotargets: T.List['build.Target'] = []
	gmotargets: T.List[build.Target] = []

	types: 'T.Union[T.Type, T.Tuple[T.Type, ...]]') -> T.List[_T]:
	types: T.Union[T.Type, T.Tuple[T.Type, ...]]) -> T.List[_T]:

Uh oh!

Conversation

Volker-Weissmann commented Apr 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eli-schwartz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Volker-Weissmann Apr 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Volker-Weissmann Apr 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Volker-Weissmann commented Apr 30, 2023

Uh oh!

tristan957 commented Apr 30, 2023

Uh oh!

dcbaker commented May 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bruchar1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Volker-Weissmann commented Apr 26, 2023 •

edited

Loading

Volker-Weissmann Apr 30, 2023 •

edited

Loading

Volker-Weissmann Apr 28, 2023 •

edited

Loading

dcbaker commented May 3, 2023 •

edited

Loading