Let’s Rewrite Buildroot

[chinmaygarde]

The current buildroot was forked from Chromium many years ago. Due to different project priorities, a lot of features were tacked on in an entirely ad-hoc fashion. Changing project priorities and the lack of a clear owner has left a lot of unused cruft in the source tree.

For instance, there are definitions for platforms that don’t exist (fnl), or, have no users in the engine source tree (cros). Many toolchain definitions reference toolchains that don’t exist at all (GCC toolchains and many of the toolchains for architectures that we don’t target). There are also tools and templates for features that we no longer use (like preparing APKs in the source tree). Many unused python and shell utilities reference Chromium infrastructure.

Many of the features we want to add support for are not fully implemented and the existing definitions cause confusion. This includes adding support for sanitizers or supporting custom toolchains and sysroots. This technical debt is currently limiting development velocity and developer productivity.

The Flutter engine is a relatively simple project that would be better served with a simpler buildroot. I propose we write one from scratch.

[jonahwilliams]

The technical debt in the current buildroot is preventing increasing our Android SDK version - either blocking or requiring worse implementations of APIs introduced in newer android versions.

I found that many of the android support libraries we depend on our now packaged as .aar files in newer android SDKs. Our current buildroot doesn't have support for .aar files, though they are only a zip archive of the jar and some resource files. While attempting to patch in an aar rule from chromium buildroot I found that this rule in turn depended on several other newer rules, all of which referred to flags which don't currently exist in ours or required slightly different versions of tool files like gn_helpers.py.

The solution I discussed offline with @chinmaygarde and @jason-simmons is to use reflection to invoke critical accessibility APIs - but this in turn is creating more technical debt for us to manage. Furthermore, less critical Android APIs like system dividers or password auto-fill can't be implemented until this is fixed without also relying on reflection.

[Hixie]

I strongly endorse this product and/or service.

[jf]

Tagging this issue (via #20717 (comment)) with the latest update: this is getting worked on by @jason-simmons. Thank you!

[dnfield]

@jonahwilliams please see flutter/buildroot#194 - that seems to successfully roll the Android SDK to 28.

I did initially pull in the AAR by unzipping it, but realized we don't actually need the support library it references - seems to be a stale ref from Chromium. It'd be great if you can review/confirm.

[dnfield]

One caveat to this - the Chromium team has solved a lot of problems - as I've worke dthrough rolling the Android SDK, for example, some problems were solved simply by taking the latest version of a file from the Chromium buildroot that corresponds to a particular file in our buildroot.

It'd be nice to have a more streamlined buildroot, but I'd hate to lose some of the work they've solved for (particularly around things like what work arounds are needed to install the right dependencies for Version XYZ of Ubuntu). But then, maybe there's a better solution out there that's more focused on that specific problem - or perhaps ther'es some way we can keep tapping into the latest and greatest that Chromium does for this stuff while not taking on a lot of cruft and baggage that Flutter doesn't acutally need.

[dnfield]

Currently, updating the Android SDK requires the following steps, for each supported platform:

1. Manually run sdkmanager to download some SDK assets
2. Manually run scripts to package those assets up into a tarball.
3. Manually run scripts to upload them to a mojo bucket.
4. Manually update the hashes for the mojo bucket in platform specific files.
5. Update various references to the support library for class path references/javadoc

I've added some documentation for this in flutter/buildroot#202 - which is mainly just taken from Chromium's build root with some tweaks.

A better solution would be to have a (dart? python? I'm assuming dart) script that can just download these assets based on some version numbers for us, and gclient sync them for your platform without resorting to the weird mojo bucket.

I'm working on a quick prototype to see if we can recreate the expected files this way, if so I think we should look to replace the current process with that.

[dnfield]

Created https://github.com/dnfield/android_sdk_downloader as a prototype for achieving a bit of this.

Need to figure out the following:

1. Where does this live (buildroot? engine?) and how do we bootstrap Dart for it?
2. The archive package is bad at handling large zip files - in particular, the NDK zip file is really heavy and eats up tons of ram to unzip. I'm shelling out for this, similar to how pub does it.
3. This doesn't get the Android support library, but we already have other logic handling that and we may want to separate out the approach for it anyway (it comes from Maven, not from the Android repo).
4. This is missing a few files that we currently get, but I'm pretty sure it we don't need them.
5. It needs to detect if the files exist already to avoid grabbing the same (large) assets every time we do a gclient sync.

[stuartsoft]

Any updates on this? I'm interested in getting the AutoFill feature working, but it's blocked by this issue.

[dnfield]

@stuartsoft what exactly is blocking you on this? We did end up migrating the Android SDK, we're not quite migrated to Android X for the engine though.

[stuartsoft]

@dnfield The AutoFill ticket was blocked by this ticket. #13015
Is there something special I need to do to enable LastPass autofill support for password fields?

[dnfield]

The part of this ticket that was blocking it is resolved.

[dnfield]

(we're now using Android 28)

[dnfield]

The Android build no longer depends on the old Chromium rules - except for the config, which we could easily move into the engine repo if we wanted.

However, the ICU package does still depend on those rules. If we want to strip them out, we'll need to rework how ICU is built.