☂️Merge flutter/buildroot into flutter/engine and retire the repository. #67373
Assignees
Labels
c: proposal
A detailed proposal for a change to Flutter
c: tech-debt
Technical debt, code quality, testing, etc.
e: buildmoot
Final steps to remove `flutter/buildroot`.
engine
flutter/engine repository. See also e: labels.
P2
Important issues not at the top of the work list
team
Infra upgrades, team productivity, code health, technical debt. See also team: labels.
team-engine
Owned by Engine team
triaged-engine
Triaged by Engine team
Comments
chinmaygarde
added
team
|
Definitely in favour of merging buildoot into the engine repo. When we do this, we'll need to update the engine build wiki to eliminate the relevant instructions. One thing I run across relatively frequently are patches that have landed in the buildroot that were never rolled (or can't roll) to the engine repo. Whenever we hit this situation, it typically involves a minor archaeological expedition to figure out what the change was intended to fix and why it wasn't rolled as well as a decision on whether to revert those changes before applying my own or try to roll them through. |
|
The split is also super annoying when trying to maintain any relatively long-lived engine branch that involves buildroot changes (as a current example, the UWP development fork), because keeping up with current master involves maintaining two parallel branches, updating them both at the same time, and updating the DEPS in the fork. |
markusaksli-nc
added
passed first triage
c: proposal
flutter-triage-bot
bot
added
team-engine
cbracken
changed the title
6 tasks
matanlurey
added a commit
to flutter/engine
that referenced
this issue
Oct 10, 2023
…re`. (#46729) As part of flutter/flutter#67373, we'll be adding, for example `third_party/glfw`. This PR will by default ignore folders, except for ones that have internal repo-sourced code (which are unlikely to change much, if at all during this transition). _/cc @Hixie, @chinmaygarde FYI only_.
|
This is now, once again, work in progress. I'll update the original post (thanks @chinmaygarde) with details. |
matanlurey
added
P2
|
I am assigning myself since I'm actively working on a part of this, but this effort has room for a lot of parallelisation so I encourage others who want to actively work on this to self-assign as well. |
This was referenced Oct 17, 2023
chinmaygarde
added a commit
to flutter/buildroot
that referenced
this issue
Oct 17, 2023
Part of flutter/flutter#67373. Pulled into the engine in flutter/engine#46989.
chinmaygarde
added a commit
to chinmaygarde/flutter_engine
that referenced
this issue
Oct 17, 2023
Now, both the DEPS and the custom GN rules for them are in the same repo and won't need a buildroot sibling patch + roll. Part of flutter/flutter#67373. Pending landing of the buildroot patch in flutter/buildroot#781 (the DEPS will be updated before landing and after the presubs pass).
auto-submit bot
pushed a commit
to flutter/engine
that referenced
this issue
Oct 17, 2023
Now, both the DEPS and the custom GN rules for them are in the same repo and won't need a buildroot sibling patch + roll. Part of flutter/flutter#67373. Pending landing of the buildroot patch in flutter/buildroot#781 (the DEPS will be updated before landing and after the presubs pass).
cbracken
added a commit
to flutter/buildroot
that referenced
this issue
Mar 7, 2024
In flutter/engine#51258, all existing build_overrides from the buildroot were copied into the engine. This replaces each existing file under build_overrides in the buildroot with a shim that just imports the files landed in that patch. This allows the third-party dependencies to continue hardcoding the `//build_overrides/foo.gni` path, but provides a seamless path forward when we drop the buildroot. Issue: flutter/flutter#144790 Part of: flutter/flutter#67373 ## Pre-launch Checklist - [X] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [X] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [X] I read the [Flutter Style Guide] _recently_, and have followed its advice. - [X] I signed the [CLA]. - [X] I listed at least one issue that this PR fixes in the description above. - [X] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [X] Do not start, operate, or service Flutter CI infra until you read and understand the operator's manual. Failure to do so could result in serious injury or death. - [X] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/wiki/Tree-hygiene#overview [Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene [test-exempt]: https://github.com/flutter/flutter/wiki/Tree-hygiene#tests [Flutter Style Guide]: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/wiki/Chat
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 7, 2024
In flutter#51258, all existing build_overrides from the buildroot were copied into the engine. In flutter/buildroot#832, all buildroot build_overrides were converted into simple forwarding files that point to the ones in the engine such that we have a seamless path to eliminating the buildroot, while allowing third-party dependencies to continue to hardcode `//build_overrides/foo.gni`. This rolls the buildroot to the engine and switches everthing over to the new engine-based build_overrides. Issue: flutter/flutter#144790 Part of: flutter/flutter#67373
9 tasks
cbracken
added a commit
to flutter/engine
that referenced
this issue
Mar 7, 2024
In #51258, all existing build_overrides from the buildroot were copied into the engine. In flutter/buildroot#832, all buildroot build_overrides were converted into simple forwarding files that point to the ones in the engine such that we have a seamless path to eliminating the buildroot, while allowing third-party dependencies to continue to hardcode `//build_overrides/foo.gni`. This rolls the buildroot to the engine and switches everthing over to the new engine-based build_overrides. Fixes: flutter/flutter#144790 Part of: flutter/flutter#67373 ## Pre-launch Checklist - [X] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [X] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [X] I read and followed the [Flutter Style Guide] and the [C++, Objective-C, Java style guides]. - [X] I listed at least one issue that this PR fixes in the description above. - [X] I added new tests to check the change I am making or feature I am adding, or the PR is [test-exempt]. See [testing the engine] for instructions on writing and running engine tests. - [X] I'm pretty sure that despite no real changes that should affect the licence script, that it'll probably fail for some reason anyway and I'll spend 2 hours fixing it. - [X] I updated/added relevant documentation (doc comments with `///`). - [X] I signed the [CLA]. - [X] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/wiki/Tree-hygiene#overview [Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene [test-exempt]: https://github.com/flutter/flutter/wiki/Tree-hygiene#tests [Flutter Style Guide]: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style [testing the engine]: https://github.com/flutter/flutter/wiki/Testing-the-engine [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/wiki/Chat
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 7, 2024
This moves Flutter out of the buildroot's third_party directory and into //flutter/third_party and updates all BUILD and gni files. Issue: flutter/flutter#144786 Part of: flutter/flutter#67373
This was referenced Mar 7, 2024
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 7, 2024
Migrate perfetto from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144808 Part of: flutter/flutter#67373
9 tasks
cbracken
added a commit
to flutter/engine
that referenced
this issue
Mar 8, 2024
This moves Flutter out of the buildroot's third_party directory and into //flutter/third_party and updates all BUILD and gni files. Issue: flutter/flutter#144786 Part of: flutter/flutter#67373 ## Pre-launch Checklist - [X] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [X] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [X] I read and followed the [Flutter Style Guide] and the [C++, Objective-C, Java style guides]. - [X] I listed at least one issue that this PR fixes in the description above. - [X] I added new tests to check the change I am making or feature I am adding, or the PR is [test-exempt]. See [testing the engine] for instructions on writing and running engine tests. - [X] I updated/added relevant documentation (doc comments with `///`). - [X] The odds of this passing on its first CI run is 0%, because there's no way the licence script isn't going to have a fit over it, at which point I promise on my honour that I will dutifully fix all the things it complains about and re-run, probably more than once. - [X] This PR fulfils the minimum [Canadian Content](https://www.youtube.com/watch?v=h9NMnPm5kXc&list=PLJ7QPuvv91JuoPWrLhqX0zDFUoLdynMEk&index=3) regulations mandated by the CRTC. - [X] I signed the [CLA]. - [X] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/wiki/Tree-hygiene#overview [Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene [test-exempt]: https://github.com/flutter/flutter/wiki/Tree-hygiene#tests [Flutter Style Guide]: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style [testing the engine]: https://github.com/flutter/flutter/wiki/Testing-the-engine [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/wiki/Chat
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 8, 2024
Migrate perfetto from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144808 Part of: flutter/flutter#67373
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 8, 2024
Migrates vulkan_memory_allocator from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144812 Part of: flutter/flutter#67373
8 tasks
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 8, 2024
Migrates vulkan_memory_allocator from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144812 Part of: flutter/flutter#67373
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 8, 2024
Migrates vulkan_memory_allocator from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144812 Part of: flutter/flutter#67373
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 8, 2024
Migrates vulkan_memory_allocator from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144812 Part of: flutter/flutter#67373
auto-submit bot
pushed a commit
to flutter/engine
that referenced
this issue
Mar 8, 2024
Migrates vulkan_memory_allocator from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144812 Part of: flutter/flutter#67373 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
1 task
cbracken
added a commit
to cbracken/dart_sdk
that referenced
this issue
Mar 8, 2024
As part of merging Flutter's flutter/buildroot and flutter/engine repos, Flutter is migrating all third-party dependencies from //third_party to //flutter/third_party. This change allows consumers of the Dart SDK (notably Flutter), to override the path of their third_party/perfetto directory. Issue: flutter/flutter#144204 Part of: flutter/flutter#67373
copybara-service bot
pushed a commit
to dart-lang/sdk
that referenced
this issue
Mar 9, 2024
As part of merging Flutter's flutter/buildroot and flutter/engine repos, Flutter is migrating all third-party dependencies from //third_party to //flutter/third_party. This change allows consumers of the Dart SDK (notably Flutter), to override the path of their third_party/perfetto directory by setting perfetto_root_path in //build_overrides/build.gn. The Dart SDK does not override perfetto's path, but does override other variables (perfetto_build_with_embedder) in the equivalent overrides file secondary/build_overrides/build.gn. TEST=None. Refactor only; no semantic changes. Issue: flutter/flutter#144204 Change-Id: Ie3835a5507a39bf2d68b1c86d42c626093358b17 Part of: flutter/flutter#67373 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/356622 Reviewed-by: Zach Anderson <zra@google.com> Commit-Queue: Chris Bracken <cbracken@google.com> Reviewed-by: Alexander Aprelev <aam@google.com>
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 9, 2024
Migrate perfetto from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144808 Part of: flutter/flutter#67373
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 11, 2024
Migrate perfetto from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144808 Part of: flutter/flutter#67373
copybara-service bot
pushed a commit
to dart-lang/sdk
that referenced
this issue
Mar 11, 2024
In https://dart-review.googlesource.com/c/sdk/+/356622 we updated the perfetto include directory in libprotozero_config to use `perfetto_root_path` rather than hardcoding `//third_party/perfetto`. Some Perfetto headers, however, #include a generated header, which is located at `$root_gen_dir/$perfetto_root_path/build_config`. See declaration of perfetto_build_flags.h and buildflag_gen_dir_ at: https://flutter.googlesource.com/third_party/perfetto/+/refs/heads/main/gn/BUILD.gn#26 This was missed in the first patch since while the default local engine build of Flutter DOES do all gn processing, it DOES NOT do a full Dart build, our presubmits however, revealed the gap. This updates the include directories for the config to use the correct value as declared in perfetto. TEST=None. Refactor only; no semantic changes. Issue: flutter/flutter#144204 Change-Id: I44c1269af85e40eab8e397c3d099f5dc1bca1229 Part of: flutter/flutter#67373 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/356960 Commit-Queue: Chris Bracken <cbracken@google.com> Reviewed-by: Alexander Aprelev <aam@google.com>
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 12, 2024
Migrate perfetto from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144808 Part of: flutter/flutter#67373
cbracken
added a commit
to cbracken/flutter_engine
that referenced
this issue
Mar 12, 2024
Migrate perfetto from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144808 Part of: flutter/flutter#67373
auto-submit bot
pushed a commit
to flutter/engine
that referenced
this issue
Mar 12, 2024
Migrate perfetto from the buildroot's third_party directory to the engine's. Issue: flutter/flutter#144204 Part of: flutter/flutter#67373 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
Remaining work is now tracked by this label:
EDITED by @matanlurey so it's more visible to others. Original post is below:
Background on buildroot by @chinmaygarde
The Flutter Buildroot and the Engine used to live in a common repository. In 2016, in order to facilitate building the Flutter Engine in the Fuchsia as well Flutter buildroots, the buildroot used by Flutter was separated into its own repository with the engine DEPS file referencing the version of the buildroot to use. The Flutter Engine GN rules had to be written such that they worked when the engine was present in either buildroot.
Flutter no longer builds in the Fuchsia buildroot. Instead, Fuchsia artifacts are built from the Flutter buildroot using the Fuchsia SDK (which was not present at the time the split occurred) and vended to Flutter application authors. So, the original use case of the split is no longer present.
Even though there is no use-case served by the split, there are numerous downsides to this approach:
Personally, I dread instances where I have to update the buildroot. Not because it is hard, but because it is incredibly tedious. And quite unnecessarily so. The buildroot itself is a significant source of technical debt and having build rules split across two repos also makes incrementally tackling this debt hard. Merging the repos would get rid of one source of unnecessary complexity in the build process.
The buildroot has no presubmit or its own issue tracker, so I don't expect any infra related issues with this switch.
The original split occurred by preserving no history. The team has grown significantly since then and the Git history may be valuable. So maybe we should consider a subtree import instead of just copying over the files. But, I don't have a strong opinion either way.
The first PR is WIP, flutter/engine#46733, which moves
third_party/glfw.Roughly, the process is:
Update
DEPSforsrc/third_partydependencies:Update source references, such as any
BUILD.gnfiles (but other files are also possible):Update the license checker (WIP, @cbracken is hacking on this now).
Here is a list of all dependencies that need to be migrated.
Some of these (i.e.
skia) will be more difficult, and might require a few incremental PRs, or closer work with folks on the engine team. Others, like the Dart packages, or leaf-level C++ packages that aren't highly used (likeglfw) are simpler, and can be done more simply (see flutter/engine#46733).To volunteer, edit this table and replace
Nonewith<link-to-GH-issue>, and assign yourself to the GH issue.You can follow this template:
C++ or Utilities
src/third_party/abseil-cppsrc/flutter/third_party/abseil-cppsrc/third_party/anglesrc/flutter/third_party/anglesrc/third_party/benchmarksrc/flutter/third_party/benchmarksrc/third_party/boringsslsrc/flutter/third_party/boringsslsrc/third_party/expatsrc/flutter/third_party/expatsrc/third_party/flatbufferssrc/flutter/third_party/flatbufferssrc/third_party/freetype2src/flutter/third_party/freetype2src/third_party/glfwsrc/flutter/third_party/glfwsrc/third_party/googletestsrc/flutter/third_party/googletestsrc/third_party/gtest-parallelsrc/flutter/third_party/gtest-parallelsrc/third_party/harfbuzzsrc/flutter/third_party/harfbuzzsrc/third_party/icusrc/flutter/third_party/icusrc/third_party/imguisrc/flutter/third_party/imguisrc/third_party/injasrc/flutter/third_party/injasrc/third_party/jsonsrc/flutter/third_party/jsonsrc/third_party/khronossrc/flutter/third_party/khronossrc/third_party/libcxxsrc/flutter/third_party/libcxxsrc/third_party/libcxxabisrc/flutter/third_party/libcxxabisrc/third_party/libjpeg-turbosrc/flutter/third_party/libjpeg-turbosrc/third_party/libpngsrc/flutter/third_party/libpngsrc/third_party/libtess2src/flutter/third_party/libtess2src/third_party/libwebpsrc/flutter/third_party/libwebpsrc/third_party/libxmlsrc/flutter/third_party/libxmlsrc/third_party/ocmocksrc/flutter/third_party/ocmocksrc/third_party/perfettosrc/flutter/third_party/perfettosrc/third_party/protobufsrc/flutter/third_party/protobufsrc/third_party/pyyamlsrc/flutter/third_party/pyyamlsrc/third_party/rapidjsonsrc/flutter/third_party/rapidjsonsrc/third_party/shadercsrc/flutter/third_party/shadercsrc/third_party/skiasrc/flutter/third_party/skiasrc/third_party/stbsrc/flutter/third_party/stbsrc/third_party/sqlitesrc/flutter/third_party/sqlitesrc/third_party/tinygltfsrc/flutter/third_party/tinygltfsrc/third_party/swiftshadersrc/flutter/third_party/swiftshadersrc/third_party/vulkan-depssrc/flutter/third_party/vulkan-depssrc/third_party/vulkan_memory_allocatorsrc/flutter/third_party/vulkan_memory_allocatorsrc/third_party/wuffssrc/flutter/third_party/wuffssrc/third_party/yapfsrc/flutter/third_party/yapfAndroid and Java
TBD.
Dart Packages
src/third_party/pkg/archivesrc/flutter/third_party/pkg/archivesrc/third_party/pkg/equatablesrc/flutter/third_party/pkg/equatablesrc/third_party/pkg/flutter_packagessrc/flutter/third_party/pkg/flutter_packagessrc/third_party/pkg/gcloudsrc/flutter/third_party/pkg/gcloudsrc/third_party/pkg/googleapissrc/flutter/third_party/pkg/googleapissrc/third_party/pkg/platformsrc/flutter/third_party/pkg/platformsrc/third_party/pkg/processsrc/flutter/third_party/pkg/processsrc/third_party/pkg/process_runnersrc/flutter/third_party/pkg/process_runnersrc/third_party/pkg/vector_mathsrc/flutter/third_party/pkg/vector_mathGeneral build infra
src/build_overridessrc/flutter/build_overridesThe text was updated successfully, but these errors were encountered: