GN rules can become out of date with virtually no notice

[matanlurey]

In flutter/engine#50933 @dnfield and I found a BUILD.gn rule where inputs were missing. In practical terms, that means a target was considered cached even when sources have changed. This can effect hermeticness, local development, and potentially CI (depending how much caching is done).

I don't know if its possible to enforce this (looking in https://fuchsia.dev/fuchsia-src/development/build/build_system/best_practices_templates and https://fuchsia.dev/fuchsia-src/development/build/build_system/intro), or if it's worth doing so, but this seems like a general repository health issue.

(/cc @johnmccutchan @zanderso, but happy for this to go through normal triage)

[jonahwilliams]

Generally the way this is solved in other build systems (bazel) is that all listed input files are copied into a randomly generated temporary directory, and the compilation/build step is performed there instead of in the repo CWD. Thus missing files, if actually used, will cause the build to fail to compile.

There is, AFAIK, no other trick besides this.

[chinmaygarde]

The recommended way is to make GN give the tool being invoked the list of inputs. At least in this case, it looks like the tool (I'm assuming gradle since they are all gradle_tasks) is doing some kind of glob and totally ignoring the sources in its invocation. This just seems like a bad template instead of GN issue.

One way to make this better but not fully solve this is to do a glob yourself but that slows down gn gen which is explicitly discouraged and your globs will never be identical to the target tools rules.

[chinmaygarde]

Put another way, you could put really any random file in there and the gradle would still do its own thing.

[zanderso]

I have different advice form @chinmaygarde, which is to use GN's depfile mechanism. This is actually what we do for Dart's GN rules in the tree, as well.

A GN action() target can specify a depfile parameter, which is a file that the script executed by the action is expected to write, which contains all of it's actual dependencies under the source tree. Subsequent invocations of the ninja build will see that file, and know that if any of the files listed in the depfile are dirty, then the action should be re-run.

Ideally, there's a way to add a target to the build.gradle file that will cause it to spit out the list of input files it's really going to touch while building. I suspect this is possible based on this stack overflow thread, but I don't actually have any experience with this:

https://stackoverflow.com/questions/22319130/how-to-get-files-and-directories-name-using-gradle

The run_gradle.py script would invoke that target and use its output to write to the depfile, whose path should be supplied to it as a command line argument.

[chinmaygarde]

Hmm, sources and depfiles are orthogonal. Depfiles (which the tool needs to be able to generate) are used to track dependencies discovered by the compiler or tool. For instance, impellerc uses depfiles to make sure any files #included in our shaders get dependency tracking without us having to manually add each included header to the sources. It is redundent in our rules to specify header includes as shader library sources because the depfiles do this.

But the the first invocation of the tool occurs based on the sources you provide changing. Even if it is triggered by the lack of a depfile, looking for sources in different search paths will not trigger a rebuild unless that path is specified in sources.

[chinmaygarde]

I think in this case, the bigger code smell is using one build tool to invoke another and expecting dependency tracking to just work.

[dnfield]

I had originally hoped to just stop using gradle to build the APK, and instead use GN rules from Chromium to do it.

[johnmccutchan]

Isn't the entire Android ecosystem built around gradle? Moving away from gradle for our own bespoke solution seems like that would likely cause issues for our developers that want to incorporate some existing Android packages into their Flutter application.

Also, if we did that, wouldn't we want to do the same for iOS (and other platforms)?

[zanderso]

I think @dnfield might only be suggesting to use the Chrome GN rules to build the scenario app APK, and the flutter_tool/framework/Flutter SDK would still use gradle to build Flutter apps.

OTOH, I was looking over the Chromium setup the other day, and it is a giant amount of GN templates and python scripts, and it wasn't immediately clear that it would be easy for us to import and wire up.

[johnmccutchan]

Oh, I didn't realize from #144069 (comment) that he was only suggesting we do this for the scenario app. I have less strong opinions about that.

[matanlurey]

I'd prefer to keep using (and build on) Gradle than rolling something ourselves.

If it's mostly a matter with network issues/fetching packages, I think we can commit to fixing that, at least IMO.