Skip to content

Major rewrite of the rewriter and the static introspection tool - #12277

Merged
dcbaker merged 23 commits into
mesonbuild:masterfrom
Volker-Weissmann:introspection_rewrite
May 29, 2025
Merged

Major rewrite of the rewriter and the static introspection tool#12277
dcbaker merged 23 commits into
mesonbuild:masterfrom
Volker-Weissmann:introspection_rewrite

Conversation

@Volker-Weissmann

@Volker-Weissmann Volker-Weissmann commented Sep 20, 2023

Copy link
Copy Markdown
Contributor

The rewriter and the static introspection tool used to be very broken, now it is less broken.

The most important changes are:

  1. We now have class UnknownValue for more explicit handling of situations that are too complex/impossible.
  2. If you write
var = 'foo'
name = var
var = 'bar'
executable(name, 'foo.c')

the tool now knows that the name of the executable is foo and not bar. See dataflow_dag and node_to_runtime_value for details on how we do this.

To test my work I wrote a script that:

  1. git clone's a couple of big projects using meson (e.g. systemd).
  2. Checks if the outputs of meson introspect meson.build --targets and meson introspect build_folder --targets are not contradicting each other.
  3. Checks if the output of meson introspect meson.build --targets is the same for two different versions of meson (the one we want to test and a known good version).
  4. Iterates over all targets that are found using meson introspect meson.build --targets and checks if
meson rewrite target tgt add rewrite_test_source.c
meson rewrite target tgt add rewrite_test_source.c
meson introspect meson.build --targets
meson rewrite target tgt rm rewrite_test_source.c
meson introspect meson.build --targets

does not crash and produces the expected output.

I think this script is very useful (it found a ton of bugs), but I do not know where to put it, so it currently only exists on my machine. It is too slow (1 hour iirc, haven't measured) to run it in the CI pipeline. In the docs you write

All the software on this list is tested for regressions before release

Is this testing (partially) automated? If so, could we merge my script with it?

@bruchar1 @kcgen Afaik you are one of the few people using the rewriter/static introspection tool in production. Could you test your usecases and voice your opinion?

@eli-schwartz You promised me this in a mail:

I would be willing to review the resulting PRs with the intent of
checking that the results "seem to work" and not getting bogged down in
nitty-gritty details. 🙂 I agree that it's relatively obscure
functionality that doesn't always work and the priority should be,
essentially, treating it as brand new code.

Comment thread mesonbuild/ast/interpreter.py Fixed
Comment thread mesonbuild/ast/interpreter.py Fixed
Comment thread mesonbuild/ast/printer.py Fixed
Comment thread mesonbuild/ast/printer.py Fixed
Comment thread mesonbuild/rewriter.py Fixed
Comment thread mesonbuild/rewriter.py Fixed
@Volker-Weissmann
Volker-Weissmann force-pushed the introspection_rewrite branch 2 times, most recently from ed089ca to 25c3cf2 Compare September 20, 2023 17:29
@bruchar1

Copy link
Copy Markdown
Member

Thank you for working on this. I will try to have a deeper look and to test it soon.

For the tests, I think the best thing to do is to ensure that each bug you found is covered by a test.

I wonder if this could be splitted into multiple commits, to ease understanding what each part fixes, especially for trivial fixes that are not directly part of the refactor.

@Volker-Weissmann

Copy link
Copy Markdown
Contributor Author

Thank you for working on this. I will try to have a deeper look and to test it soon.

For the tests, I think the best thing to do is to ensure that each bug you found is covered by a test.

I should have done that everytime it found a bug. I cannot do it now, since I forgot most of the bugs.

I wonder if this could be splitted into multiple commits, to ease understanding what each part fixes, especially for trivial fixes that are not directly part of the refactor.

I will do that for "trivial fixes that are not directly part of the refactor" , but the rest will still be in one big commit since it is hard/impossible to split.

@Volker-Weissmann

Copy link
Copy Markdown
Contributor Author

I splitted the trivial fixes intro seperate commits.

@bruchar1 bruchar1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just did a quick pass on python syntax. I didn't tried the functionality yet.

Comment thread mesonbuild/ast/printer.py Outdated
return SymbolNode(Token('', '', 0, 0, 0, (0, 0), val))

class DataflowDAG:
src_to_tgts: T.DefaultDict[T.Union[BaseNode, UnknownValue], T.Set[T.Union[BaseNode, UnknownValue]]]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a lot of T.Union[BaseNode, UnknownValue]. Maybe should you create a type for that? e.g. NodeOrUnknown = T.Union[BaseNode, UnknownValue].

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm against that. It is less readable and only marginally shorter.

Comment thread mesonbuild/ast/interpreter.py Outdated
Comment thread mesonbuild/ast/interpreter.py Outdated
Comment thread mesonbuild/ast/interpreter.py Outdated
active = set(srcs)
while True:
if reverse:
new: T.Set[T.Union[BaseNode, UnknownValue]] = set()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new set is created on both branches of the if. I think you could initialize it before the if.

Comment thread mesonbuild/mintro.py Outdated
Comment thread mesonbuild/mintro.py Outdated
'conditional',
]
result += [{k: v for k, v in i.items() if k in keys}]
result += [{k: v for k, v in i.__dict__.items() if k in keys}]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This __dict__ seems a code smell... Is it just because mypy do not detect the type of i correctly?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you could rewrite this as:

result += [{k: getattr(i, k) for k in keys}]

Comment thread mesonbuild/mintro.py
intro_types = get_meson_introspection_types()

if 'meson.build' in [os.path.basename(options.builddir), options.builddir]:
# TODO: This if clause is undocumented.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will you do it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is out of scope for this PR. Also, I don't like the command line api here:
I think that meson introspect meson.build --targets should be changed to one of these:

  • meson static-introspect . --targets
  • meson static-introspect --targets
  • meson rewriter --targets

I don't like the fact that meson introspect meson.build --targets and meson introspect builddir --targets are radically different things, but the cli makes it look like they are two just two ways for meson to find the correct directory.

Comment thread mesonbuild/mintro.py Outdated

if 'meson.build' in [os.path.basename(options.builddir), options.builddir]:
# TODO: This if clause is undocumented.
if os.path.basename(options.builddir) == 'meson.build':

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is environment.build_filename constant. Maybe should you use it?

Comment thread mesonbuild/mintro.py Outdated
if 'meson.build' in [os.path.basename(options.builddir), options.builddir]:
# TODO: This if clause is undocumented.
if os.path.basename(options.builddir) == 'meson.build':
sourcedir = '.' if options.builddir == 'meson.build' else options.builddir[:-11]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and use -len(environment.build_filename) here...

Comment thread mesonbuild/mintro.py Outdated
res = [root_dir / i['subdir'] / x for x in res]
res = [x.resolve() for x in res]
return res
def list_targets_from_source(intr: IntrospectionInterpreter) -> T.Any:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of making this Any, it could at least be T.List[T.Dict[str, object]]

Comment thread mesonbuild/mintro.py Outdated
'conditional',
]
result += [{k: v for k, v in i.items() if k in keys}]
result += [{k: v for k, v in i.__dict__.items() if k in keys}]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you could rewrite this as:

result += [{k: getattr(i, k) for k in keys}]

Comment thread mesonbuild/ast/printer.py Outdated
# ignores ParanthesizedNode, the binding power of the inner node is
# relevant.
return precedence_level(node.inner)
raise TypeError

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually you would use RuntimeError, but we have MesonBugError (or maybe it's MesonBugException?). I think we should use that here.

else:
for for_machine in [MachineChoice.BUILD, MachineChoice.HOST]:
self._add_languages(args, required, for_machine)
return UnknownValue()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The explanation for UnknownValue was for cases where we couldn't know what would be returned. I'm not sure why I follow that add_languages returns UnknownValue and not bool. If it returns it will always return a bool (or it will abort, but for the purpose of the rewriter that doesn't really matter, does it?)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

var = add_languages('rust', required: false)
message(var)

prints true on some machines and false on others.
So func_add_languages has to return UnknownValue.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It returns a boolean? I'm just not understanding here, if add_languages(), which returns a bool cannot be statically determined, then what can? There are no functions I know of that always return the same value in every case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You seem confused, let me clear that up:

You know that if you run meson setup builddir, the contents of the resulting builddir-directory depend on

  1. The contents of meson.build
  2. The machine you are using.

If for example both you and I clone the same project that is using meson, and we both run meson setup builddir, those two directories are not (necessarily) identical if we have different machines. This is not a bug, this is intentional. Therefore, if we both run meson introspect builddir, we (might) get different results.

But if we both clone the same project and run meson introspect meson.build we get the same result. If we don't, that is a bug. In other words, the job of ast/introspection.py is to know what happens if we run meson setup on a different, unknown machine. In other words, ast/introspection.py has full knowledge about the contents of meson.build, but no knowledge about the build/host/target machine. Let's say meson.build contains:

srcs = ['1.c']
srcs += files('2.c')
if 3+4 == 7
    srcs += '3.c'
endif
if build_machine.system == 'linux'
    srcs += 'linux-specific.c'
endif
if add_languages('rust', required: false)
    srcs += 'rust.rs'
endif
executable('foo', srcs)

If I run meson introspect meson.build on my machine, the job of meson is to figure out what sources belong to the foo-executable file if you run meson setup on your machine.
It knows that the foo-executable contains the sources 1.c and 2.c. It does not know that it contains '3.c', since I was too lazy to implement that. There is no way for it to know whether the foo-executable contains the sources 'linux-specific.c' or 'rust.rs', since a program running on my machine cannot know whether your machine has a rust compiler installed.

@dcbaker dcbaker Feb 11, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this again, my concern here is that there seems to be a conflation between the type a function returns and whether we can know what value that type is.

Like in the case of add languages above, we know that it returns a bool, we just don't know whether that bool is true or false.

This leads me to wonder whether making UnknownValue generic would be of benefit. Because as you point out, add_languages with required : false will return different values depending on whether rustc is installed. but it won't return a list, or a string, or 7, it will only return true or false. Even with this, there may well be times where we say UnknownValue[TPYE_var] because we can't be more specific, but that still is something.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might improve the typing, but it would only be used for static analysis. At runtime you'd still have UnknownValue().

def function_call(self, node: mparser.FunctionNode) -> T.Optional[InterpreterObject]:
func_name = node.func_name.value
(h_posargs, h_kwargs) = self.reduce_arguments(node.args)
(h_posargs, h_kwargs) = self.reduce_arguments(node.args, include_unknown_args = True)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No spaces around the = operator

@Volker-Weissmann
Volker-Weissmann force-pushed the introspection_rewrite branch 2 times, most recently from ee79d23 to 79661ac Compare October 2, 2023 18:04
@GranMinigun

Copy link
Copy Markdown

Cloned this branch, ran introspect --all on DOSBox Staging's meson.build (that's what e.g. Qt Creator does), it happily crashed:

Unable to evaluate subdir([<mesonbuild.interpreterbase.baseobjects.UnknownValue object at 0x7f7d6651e490>]) in AstInterpreter --> Skipping
Traceback (most recent call last):
  File "/dev/shm/meson/mesonbuild/mesonmain.py", line 194, in run
    return options.run_func(options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dev/shm/meson/mesonbuild/mintro.py", line 541, in run
    return print_results(options, results, indent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dev/shm/meson/mesonbuild/mintro.py", line 501, in print_results
    print(json.dumps(out, indent=indent))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type UnknownValue is not JSON serializable

ERROR: Unhandled python exception

    This is a Meson bug and should be reported!

Python 3.11.5.

@Volker-Weissmann

Copy link
Copy Markdown
Contributor Author

Cloned this branch, ran introspect --all on DOSBox Staging's meson.build (that's what e.g. Qt Creator does), it happily crashed:

Confirmed and on my way to fix it.

@Volker-Weissmann

Copy link
Copy Markdown
Contributor Author

Cloned this branch, ran introspect --all on DOSBox Staging's meson.build (that's what e.g. Qt Creator does), it happily crashed:

Unable to evaluate subdir([<mesonbuild.interpreterbase.baseobjects.UnknownValue object at 0x7f7d6651e490>]) in AstInterpreter --> Skipping
Traceback (most recent call last):
  File "/dev/shm/meson/mesonbuild/mesonmain.py", line 194, in run
    return options.run_func(options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dev/shm/meson/mesonbuild/mintro.py", line 541, in run
    return print_results(options, results, indent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dev/shm/meson/mesonbuild/mintro.py", line 501, in print_results
    print(json.dumps(out, indent=indent))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type UnknownValue is not JSON serializable

ERROR: Unhandled python exception

    This is a Meson bug and should be reported!

Python 3.11.5.

Fixed in b98292e

Comment thread mesonbuild/rewriter.py Outdated
@Volker-Weissmann

Copy link
Copy Markdown
Contributor Author

Any update?

@bonzini

bonzini commented Feb 3, 2025

Copy link
Copy Markdown
Contributor

Hi! Would you be able to rebase the PR?

@Volker-Weissmann

Copy link
Copy Markdown
Contributor Author

Hi! Would you be able to rebase the PR?

I could but only if a maintainer says that it gets merged after rebasing.

@bonzini

bonzini commented Feb 3, 2025

Copy link
Copy Markdown
Contributor

Not a maintainer but I will try to give it a quick review beforehand, at least.

@bonzini bonzini self-assigned this Feb 3, 2025
@bonzini

bonzini commented Feb 5, 2025

Copy link
Copy Markdown
Contributor

It looks sane. It's a huge commit, but that can be fixed after conflicts are resolved.

@Volker-Weissmann

Copy link
Copy Markdown
Contributor Author

We still need someone with merge permissions.

@bonzini

bonzini commented Feb 11, 2025

Copy link
Copy Markdown
Contributor

Well as long as it has conflicts no one will look at it.

To improve type-safety and readability we replace a
dictionary with a new class `IntrospectionDependency`.
The AstInterpreter now stores how deep into if/elif/else we are.
This is currently dead code, but it will be read in future commits.
@Volker-Weissmann
Volker-Weissmann force-pushed the introspection_rewrite branch 2 times, most recently from 83b9549 to 960e8a9 Compare May 8, 2025 15:56
@dcbaker

dcbaker commented May 8, 2025

Copy link
Copy Markdown
Member

@Volker-Weissmann then I'm perfectly happy. I personally like introducing tests with an expected fail status and then fixing them (see the Fortran dependency scanner work), I think there is value in seeing "Look, this doesn't work" and then "See, I fixed it"

@Volker-Weissmann
Volker-Weissmann force-pushed the introspection_rewrite branch from 960e8a9 to e8ba9e1 Compare May 8, 2025 16:05
@Volker-Weissmann

Copy link
Copy Markdown
Contributor Author

Failure seems unrelated

@Volker-Weissmann

Copy link
Copy Markdown
Contributor Author

@bruchar1 @bonzini I believe I addressed all of your concerns

@bruchar1 bruchar1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is something that bothers me about UnknownValue. it seems sometime it represents a Node, and sometime it represents a value. Would it be possible to have an UnknownNode deriving from BaseNode? That way, it would simplify typing, and remove that node vs value confusion.

Comment thread mesonbuild/ast/interpreter.py Outdated
raise TypeError
return ret

def get_cur_value(self, var_name: str, allow_none: bool = False) -> T.Union[BaseNode, UnknownValue]:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems return type can be None as well

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@Volker-Weissmann

Copy link
Copy Markdown
Contributor Author

There is something that bothers me about UnknownValue. it seems sometime it represents a Node, and sometime it represents a value. Would it be possible to have an UnknownNode deriving from BaseNode? That way, it would simplify typing, and remove that node vs value confusion.

While it would be possible to split UnknownValue up into UnknownValue and UnknownNode(BaseNode) I don't believe that to be a good idea. It's difficult for me to put that into words, but let's try:

It would simplify typing a bit, but since Unknown* only exists during static introspection, but not during normal interpretation and because Unknown* is kind of a special case I would like to have it as a seperate type.

I think just having a type called UnknownValue with this help text

This class is only used for the rewriter/static introspection tool and
    indicates that a value cannot be determined statically, either because of
    limitations in our code or because the value differs from machine to
    machine.

should be clear to the reader.

Replace the variable tracking of `AstInterpreter.assignments`
with a slightly better variable tracking called
`AstInterpreter.cur_assignments`.
We now have a class `UnknownValue` for more explicit handling
of situations that are too complex/impossible.
Replace `AstInterpreter.reverse_assignment` with
`AstInterpreter.all_assignment_nodes`.
This does not give us an immediate advantage but
will be useful in future commits.
Some of the evaluate_* functions in AstInterpreter seem very broken
and do not even evaluate all of the AST. I do not know what the
original author thought, so I just fixed it.
Make the AstInterpreter create a directed acyclic graph (called
`dataflow_dag`) that stores the how the data flowes from one node in
the AST to another.

Add `AstInterpreter.node_to_runtime_value` which uses `dataflow_dag`
to find what value a variable at runtime will have.

We don't use dataflow_dag or node_to_runtime_value anywhere yet, but
it will prove useful in future commits.
Without this commit, something like this crashes the static
introspection/rewrite tool:
```
default_options : ['warning_level='
  + run_command(['echo', '3']).stdout().strip()],
```

This commit does not reintroduce mesonbuild#14382.
`resolve_node` is simply a half-broken, worse implementation of
`node_to_runtime_value` that we recently introduced.

In the example below, the static introspection tool/rewriter now
understands that the name of the executable is foo instead of bar:
```
var = 'foo'
name = var
var = 'bar'
executable(name, 'foo.c')
```
`AstInterpreter.node_to_runtime_value` can now resolve function calls.
De-duplicate some code by extracting the common code into the new
`rm_src_or_extra` function.
Change the semantics of IntrospectionBuildTarget.source_nodes
and IntrospectionBuildTarget.extra_files .

The rewriter and the static introspection tool used to be very broken,
now it is *less* broken, hence we add some tests in this commit.

Fixes mesonbuild#11763
Without this commit, the static introspection tool crashes when
introspecting systemd since certain values are `UnknownValue`
which was unexpected.

(I tested sytemd's commit hash fefcb935cd.)
@Volker-Weissmann
Volker-Weissmann force-pushed the introspection_rewrite branch from e8ba9e1 to ce02227 Compare May 11, 2025 13:14
@bonzini

bonzini commented May 24, 2025

Copy link
Copy Markdown
Contributor

@dcbaker with 1.8.1 out of the door let's go

@dcbaker
dcbaker merged commit e7e2584 into mesonbuild:master May 29, 2025
@dcbaker

dcbaker commented May 29, 2025

Copy link
Copy Markdown
Member

merged

@Volker-Weissmann

Copy link
Copy Markdown
Contributor Author

FUCKING FINALLY!

THANKS TO EVERYONE WHO HELPED IN THIS PR.

@eli-schwartz

Copy link
Copy Markdown
Member

Thank you for being able to push this over the finish line, and sorry I wasn't able to be any help myself in the end :D this is 2500 lines of code I mostly do not understand, so I'm grateful someone else ended up being able to review it. Sorry for flaking out. :(

The improvements sound great.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants