-
Notifications
You must be signed in to change notification settings - Fork 27.2k
Adding a new Test Shard
keyonghan edited this page Jan 29, 2024
·
11 revisions
Further documentation on Flutter's build infrastructure can be found in https://github.com/flutter/flutter/blob/master/dev/bots/README.md.
A general outline of the requirements that a Flutter CI test shard has:
- On LUCI, test shards map to builders. Each test shard must have its own LUCI builder. For the Framework, these are defined in framework_config.star. Generally you will need to have both a pre-submit ("try" in LUCI terminology) builder and a post-submit ("prod") builder.
- This LUCI builder will specify a "recipe" to run. These are starlark scripts that determine the actual CI steps to run, and are defined in flutter.googlesource.com/recipes. Most Framework tests use the flutter/flutter_drone.py recipe. To learn how to edit these, see https://github.com/flutter/flutter/blob/master/dev/bots/README.md#editing-a-recipe.
- Builders are then added to .ci.yaml. These files are read by Flutter's build dashboard, and are used for scheduling builds.
It is important to land these changes in order to prevent any failing builds during the migration period:
- Framework tests are run by a Dart test runner called test.dart that lives in the framework repository. Any new test shards must first be added to this file. Merge this framework change. Note that sharding an existing test doesn't need to update test.dart.
- Update .ci.yaml in the Framework tree to include the newly added builder following CI_YAML.md. Ensure that the "shard" and "subshard"/"subshards" properties match what was added to test.dart in the previous step. Verify that the entry is marked as
bringup: true
. New shards should always be marked in bringup to verify they are passing on master before being able to block the tree. Merge this change. Note that the new shard will not run in presubmit at this point as the target is withbringup: true
. - Monitor the CI results of the new shard on the Flutter build dashboard. After 50 consecutive passing builds without any flakes, the flake bot will create a PR to remove the
bringup: true
parameter from.ci.yaml
in the Framework tree. This will allow the test to block the tree, preventing breakages. With this change, the new shard will start running in presubmit automatically, unless specifypresubmit: false
. Note the flake bot runs once a week on Weds.
Note: if a new post-submit target is renamed from an existing target, there is no need to follow the bringup process.
- Home of the Wiki
- Roadmap
- API Reference (stable)
- API Reference (main)
- Glossary
- Contributor Guide
- Chat on Discord
- Design documents
- Code of Conduct
- Issue triage reports (latest)
- Our Values
- Tree hygiene
- Issue hygiene and Triage
- Style guide for Flutter repo
- Project teams
- Contributor access
- What should I work on?
- Popular issues
- Running and writing tests
- Release process
- Flutter Framework Gardener Rotation
- Rolling Dart
- Manual Engine Roll with Breaking Commits
- Updating Material Design Fonts & Icons
- Postmortems and Retrospectives
- Hotfix Documentation Best Practices
- In case of emergency
- Landing Changes With Autosubmit
- Setting up the Framework development environment
- The Framework architecture
- API Docs code block generation
- Running examples
- Using the Dart analyzer
- The flutter run variants
- Test coverage for package:flutter
- Writing a golden-file test for package:flutter
- Managing template image assets
- Setting up the Engine development environment
- Compiling the engine
- Debugging the engine
- Using Sanitizers with the Flutter Engine
- Testing the engine
- The Engine architecture
- Flutter's modes
- Crashes
- more...
- Setting up the Packages development environment
- Plugins and Packages repository structure
- Contributing to Plugins and Packages
- Understanding Packages tests
- Plugin Tests
- Releasing a Plugin or Package
- more...