Builds with geckoview dependency substitution are very slow (5+ minutes)

[mcomella]

On my machine, builds with geckoview dependency substitution take a long time. For an incremental ./gradlew :browser-state:assembleDebug, it's 5 minutes compared to less than 10 seconds. During the build,

settings.gradle> Read local.properties: /Users/mcomella/dev/mozilla-central/local.proper
ties                                                                                    [substitute-local-geckoview] Found topobjdir /Users/mcomella/dev/mozilla-central/obj-gv-
artifact from topsrcdir /Users/mcomella/dev/mozilla-central                             [substitute-local-geckoview] Will substitute GeckoView (geckoview-{nightly,beta}) with l
ocal GeckoView (geckoview-default) from /Users/mcomella/dev/mozilla-central/obj-gv-artifact/gradle/build/mobile/android/geckoview/maven                                         [substitute-local-geckoview] To update the local GeckoView, run `./mach gradle geckoview
:publishWithGeckoBinariesDebugPublicationToMavenRepository` in /Users/mcomella/dev/mozilla-central

is repeated many times: presumably, this is executed for each module (~113 times). Perhaps we're doing redundant work and can speed up the builds.

Commenting out this line:

android-components/build.gradle

Line 107 in 131dd4b

apply from: "${topsrcdir}/substitute-local-geckoview.gradle"

causes the builds to speed up again. It's unclear if the issue is just reading this file so many times (e.g. it may be recompiled each time) or something specific in the file that's slow.

┆Issue is synchronized with this Jira Task

[mcomella]

Apparently the script executes a ./mach command each time it is run: https://searchfox.org/mozilla-central/rev/33641307ec33033f5129826d8d6eda19feb8a01f/substitute-local-geckoview.gradle#44-50

Maybe we're just waiting for the python interpreter to start a hundred-ish times.

[mcomella]

I made a proof of concept that takes build times from 5m+ to 30s. I changed the mozconfig load to happen once by moving it from mc to ac. Here is my change to mc:

diff --git a/substitute-local-geckoview.gradle b/substitute-local-geckoview.gradle
--- a/substitute-local-geckoview.gradle
+++ b/substitute-local-geckoview.gradle
@@ -31,36 +31,17 @@ def log(message) {
 def warn(message) {
     logger.warn("[substitute-local-geckoview] Warning: ${message}")
 }
 
 if (!project.ext.has('topsrcdir')) {
     throw new GradleException("ext.topsrcdir must be specified to substitute for a local GeckoView")
 }
 
-apply from: "${topsrcdir}/mobile/android/gradle/mach_env.gradle"
-
-// Cribbed from https://hg.mozilla.org/mozilla-central/file/tip/settings.gradle.  When run in
-// topobjdir, `mach environment` correctly finds the mozconfig corresponding to that object
-// directory.
-def commandLine = ["${topsrcdir}/mach", "environment", "--format", "json", "--verbose"]
-def proc = commandLine.execute(
-      machEnv(topsrcdir),
-      new File(ext.has('topobjdir') ? ext.get('topobjdir') : topsrcdir))
-def standardOutput = new ByteArrayOutputStream()
-proc.consumeProcessOutput(standardOutput, standardOutput)
-proc.waitFor()
-
-// Only show the output if something went wrong.
-if (proc.exitValue() != 0) {
-    throw new GradleException("Process '${commandLine}' finished with non-zero exit value ${proc.exitValue()}:\n\n${standardOutput.toString()}")
-}
-
-def slurper = new JsonSlurper()
-def mozconfig = slurper.parseText(standardOutput.toString())
+def mozconfig = ext.mozconfig
 
 if (topsrcdir != mozconfig.topsrcdir) {
     throw new GradleException("Specified topsrcdir ('${topsrcdir}') is not mozconfig topsrcdir ('${mozconfig.topsrcdir}')")
 }
 
 if (!ext.has('topobjdir')) {
     ext.topobjdir = mozconfig.topobjdir
     log("Found topobjdir ${topobjdir} from topsrcdir ${topsrcdir}")

Here's my change to a-c: mcomella@de5e777

[mcomella]

The root cause of the issue is in the script in mozilla-central so I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1744336. Let's keep this open so we can apply and verify the fix.

[mcomella]

I pushed a patch upstream that shouldn't require any changes from consumers such as ac and tested it locally to confirm it was working.

[mcomella]

The upstream patch landed and I verified with the latest mc and ac that the build times are improved: closing.