Postmortem: AndroidX plugin migration
Flutter postmortem: AndroidX Plugin Migration
Owners: @mklim, @amirh, @dnfield, @hixie
Description: The flutter/plugins repo was updated with a breaking change to
change migrated and deprecated Android dependencies in the plugins'
build.gradle files and required any apps using the plugins to also have Gradle
files that were either migrated or compatible with the AndroidX. Gradle errors
in hard-to-debug ways on Flutter apps broken by the change, without surfacing a
clear explanation or migration path to affected users. In addition this was a
known breaking change but originally incorrectly incremented the minor semantic
version number in 6/32 of the affected packages.
Date/time: 2019-01-24 13:24 -0800
Duration: 4 days for the semantic versioning problem. 2 weeks for the confusing Gradle error.
User impact: Users pinned to the major version of firebase_analytics, firebase_database, firebase_messaging, firebase_storage, google_sign_in, and url_launcher were broken by pub despite deliberately trying to avoid any breaking changes. All other first party plugin users that try to use the latest major version are affected by the confusing Gradle error.
Timeline (all times in PST/PDT)
- 13:24 - First breaking change with incorrect versioning merged, flutter/plugins#1103.
- 14:20 - flutter/plugins#1103 uploaded to pub. <START OF OUTAGE>
- 09:44 - flutter/plugins#1115, the final commit with breaking changes and incorrect semver, is merged to master.
- 10:55 - Final batch of incorrectly versioned plugins is published to pub.
- 13:53 - @hixie files
the obscure Gradle error following a fresh run of
flutter createon the
stablechannel. Links the error to the AndroidX migration and tags @mklim, @amirh, and @dnfield.
- 14:03 - @mklim confirms the root cause of the error as an incompatibility between the app and plugin's Gradle files. (The semantic versioning issue is not known at this point.)
- 14:45 - @dnfield, @mklim, and @amirh discuss mitigating the user unfriendly
flutter createerror on
stableoffline. Decide to republish the AndroidX migration with a constraint so that users are on a version of flutter that produces AndroidX-compatible Gradle files. pub maintainers are contacted to remove the broken packages.
- 00:30 - Email back and forth begins with pub maintainers.
- 08:07 - flutter/flutter#27128 filed about upgrading a >= 1.0.0 plugin's minor version number.
- 12:33 - @mklim briefly explains the problem in
recommends to downgrade plugin versions since a revert for
flutter stableis incoming. However his comment doesn't have details on all of the plugins to downgrade and what versions to downgrade them to. User discussion continues on this bug over the next 48 hours.
- 14:00 - Discussion with pub maintainers concludes. The affected packages can't be replaced because pub has safeguards against removing a "stable" package version. Decided to replace the packages with a patch+1 revert, and then a patch+2 migration with the minimum Flutter SDK constraint.
- 09:00 - Semantic versioning issue discovered. Offline discussion scheduled with @mklim, @amirh, @kevmoo, @hixie, and @dnfield to re-assess a fix.
- 11:00 - @mklim, @amirh, @kevmoo, @hixie, and @dnfield re-review the issue with
the semantic versioning issue also in mind. Decide that the minimum Flutter
constraint is useless to address the problem since it really has to do with
build.gradle, which could have been generated by any Flutter version including the current one and which also could have been manually edited. Decide that the semver problem should be fixed by patching the 6 affected plugins with a revert and then incrementing them with a major bump and the migration.
- 11:45 - @mklim updates flutter/flutter#27128 with a detailed description of the bug, root cause, way to fix via migration or downgrade, and plan to fix the semantic versioning for the affected plugins.
- 14:28 - flutter/plugins#1127 (the patch revert) is merged to master.
- 15:05 - flutter/plugins#1128 (the correct forward roll) is merged to master.
- 15:08 - @dnfield alerts users following the AndroidX migration (flutter/flutter#23995) of the semantic version issues and that there's going to be a major version bump.
- 15:46 - The last affected package is updated on pub with both the patch revert and the forward roll with correct semver bump. <END OF SEMVER OUTAGE>
- 15:58 - @mklim updates flutter/flutter#27128 and flutter/flutter#23995 with info on the 6 packages that have had their semantic versions updated.
- 17:09 - @mklim updates flutter/flutter#27106 with more details on the bug and the fixed semantic versioning problem.
- 10:27 - @mklim updates the Flutter website via flutter/website#2310 to explain how to import a Flutter app into the Android Studio IDE for automatic migration.
- 11:28 - @mklim opens flutter/plugins#1138 so that the plugins warn about the AndroidX migration when Gradle fails to compile.
- 13:41- @mklim opens flutter/flutter#27566 to detect gradle error messages and link to a Flutter AndroidX migration guide on the Flutter website in Flutter itself.
- 17:35 - @mklim opens flutter/website#2349 to add a migration guide to AndroidX for Flutter apps and plugins.
- 13:23 - flutter/website#2349 is merged.
- 17:30 - flutter/plugins#1138 and flutter/flutter#27566 are merged. Plugins are uploaded with the change. <END OF GRADLE ERROR OUTAGE>
Some first party plugin users were broken in a minor version upgrade
It looks like the only issue filed for this directly has been flutter/flutter#27128.
All first party plugin users were broken in a confusing way
An explanation and migration path exists in the CHANGELOG.md of each plugin, but this information is hard to find and the actual Gradle error is not helpful. Several bugs on Github have been filed from affected users.
flutter/plugins#1103, flutter/plugins#1115 migrated the plugins to support AndroidX instead of the original Android support libraries. See #23995 for more details, but in short the original support libraries are deprecated and not recommended for Android app development anymore.
This was a known breaking change and the changelog links to the official
migration steps. BUT
plugins that were greater than or equal to
1.0.0 weren't updated according to
pubspec's semantic versioning
pub for packages <
1.0.0 the versioning is
0.major.minor+patch, and for >=
major.minor.patch. The AndroidX PRs universally incremented the
middle number, so any plugin that was already at or above
1.0.0 is being
incorrectly treated as non-breaking by pub. This caused the semantic versioning
In addition the breaking change was only surfaced in the CHANGELOG.md of each plugin, but resulted in breaking Gradle errors at runtime. These Gradle errors did not provide any obvious ways to debug the incompatibility.
- The plugins were migrated to AndroidX!
Where we got lucky
- @hixie diagnosed the problem with standard
flutter createtemplates relatively quickly and cc'ed the relevant devs.
What didn't work
- We understood that the AndroidX migration was breaking, but underestimated the
scale of the breakage. We didn't realize that it applied to all untouched
build.gradlefiles produced by released versions of
flutter create. Also failed to verify that the Gradle errors mentioned AndroidX or were otherwise easily understable.
flutter/pluginsCI failed to verify that the new plugins could be used with the template from
flutter createon stable.
- No manual sanity checks were recommended and insufficient ones were performed
before publishing the packages to pub. There could have been manual checks to
flutter createand semantic versioning issues.
- The packages were pushed Thursday/Friday (24-25) and remained broken over the weekend until a fix was decided communally when everyone was in the office Monday morning (28).
- The breaking change and suggested migration were only surfaced in the CHANGELOG of the plugins, not in the actual Gradle compile error. The changelogs are not very findable and the Gradle error is confusing.
- There was no way to mark the incorrect semver packages as broken in pub, so we needed to push a breaking revert along with a forward migration.
Test the plugins from a blank, stable
flutter createtemplate on the plugins CI
- flutter/flutter#27258 - The plugins releasing process should be improved to help detect regressions.
- flutter/flutter#14818 - Translate Gradle errors into actionable steps for our users when possible.
- dart-lang/pub#2038 - Show a migration prompt and ask for confirmation on updating a breaking change in pub.
- dart-lang/pub-dartlang-dart#378 - Tag specific pub releases as broken.
- flutter/flutter#27258 - Have a better/more formalized releasing process for flutter/plugins.