Add coverableLineCache param to collect
#466
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
liamappelbe
changed the title
coverableLineCache param to collectcoverableLineCache param to collect
liamappelbe
changed the title
coverableLineCache param to collectcoverableLineCache param to collect
bkonyi
reviewed
Oct 17, 2023
The purpose of force compiling all the libraries is that it tells the VM which lines are coverable, so it can generate a complete list of all the missed lines (particularly for functions that haven't been called). Each collection that the flutter tester does is in a totally fresh VM, so the VM doesn't have any knowledge about which lines are coverable left over from previous collections. So when we set the |
bkonyi
approved these changes
Oct 17, 2023
auto-submit bot
pushed a commit
to flutter/flutter
that referenced
this pull request
Oct 23, 2023
…age (#136851) One of the reasons gathering coverage information is expensive is that we have to force compile every function in the libraries we're interested in. Without this, functions that haven't been invoked (so haven't been compiled) won't have any line number information, so the coverage tool doesn't know which lines to add to the list of misses. In flutter's case, the test infra spawns many VMs, and each of these needs to recompile all those libraries. To fix this, we need a way of skipping force compilation for libraries we've already seen in previous tests, without losing the information about which lines in each library are coverable. So I [added](dart-lang/coverage#466) the `coverableLineCache` to `coverage.collect` in package:coverage v1.7.0. This cache starts out empty, but fills up with lists of all the lines that are coverable for every library as coverage is gathered. package:coverage can then tell the VM not to force compile any libraries in this cache (using `getSourceReport`'s `librariesAlreadyCompiled` param). So the first test suite will still have to compile everything, but subsequent test suites will be much faster. This speeds up coverage collection significantly, for large test suites: | Running flutter/packages/flutter tests... | Time | Overhead | | --- | --- | --- | | without coverage | 8:53 | - | | with coverage | 20:25 | 130% | | with `coverableLineCache` | 12:21 | 40% | Bug: #100751
auto-submit bot
added a commit
to flutter/flutter
that referenced
this pull request
Oct 24, 2023
… up coverage" (#137121) Reverts #136851 Initiated by: CaseyHillers This change reverts the following previous change: Original Description: One of the reasons gathering coverage information is expensive is that we have to force compile every function in the libraries we're interested in. Without this, functions that haven't been invoked (so haven't been compiled) won't have any line number information, so the coverage tool doesn't know which lines to add to the list of misses. In flutter's case, the test infra spawns many VMs, and each of these needs to recompile all those libraries. To fix this, we need a way of skipping force compilation for libraries we've already seen in previous tests, without losing the information about which lines in each library are coverable. So I [added](dart-lang/coverage#466) the `coverableLineCache` to `coverage.collect` in package:coverage v1.7.0. This cache starts out empty, but fills up with lists of all the lines that are coverable for every library as coverage is gathered. package:coverage can then tell the VM not to force compile any libraries in this cache (using `getSourceReport`'s `librariesAlreadyCompiled` param). So the first test suite will still have to compile everything, but subsequent test suites will be much faster. This speeds up coverage collection significantly, for large test suites: | Running flutter/packages/flutter tests... | Time | Overhead | | --- | --- | --- | | without coverage | 8:53 | - | | with coverage | 20:25 | 130% | | with `coverableLineCache` | 12:21 | 40% | Bug: #100751
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
coverableLineCacheis a map from library URI to the set of all lines that are coverable. If users are doing multiplecollectcalls over the same set of libraries, they should pass an empty map to this param in the first call, and then pass the same map to all subsequent calls.If
coverableLineCacheis provided, we setVmService.getSourceReport'slibrariesAlreadyCompiledparam to all its keys. This disables forced compilation for each of these libraries. The effect of this is that no misses are returned in the response for these libraries (and coverage collection is faster). To fill in the missing misses, we use the values of thecoverableLineCachemap. Finally, we fill this cache with all the hit and missed lines we've seen for each library.So
coverableLineCachecaches all the lines that have been hit or missed in previouscollectcalls, and uses it to skip recompiling those libraries, and to fill out the misses of subsequent calls.