Flutter test coverage will not report untested files

[100cm]

I run the flutter test --coverage as follows

and here are my project's files;

the generated Icov.info looks like this:

SF:lib/components/button.component.dart
DA:12,1
DA:29,1
DA:45,1
DA:47,1
DA:48,1
DA:49,1
DA:50,1
DA:51,2
DA:52,1
DA:53,1
DA:56,2
DA:57,1
DA:59,2
DA:60,1
DA:61,1
DA:62,1
DA:63,3
DA:64,2
DA:65,1
DA:66,1
DA:67,1
DA:70,1
DA:71,2
DA:72,1
LF:24
LH:24
end_of_record
SF:lib/components/button.css.dart
DA:4,1
DA:6,2
DA:7,2
DA:8,1
DA:10,1
DA:16,3
DA:53,1
DA:73,0
LF:8
LH:7
end_of_record
SF:lib/environment.dart
DA:6,1
DA:8,2
DA:9,1
DA:11,1
DA:13,2
DA:14,2
DA:15,2
DA:16,1
LF:8
LH:8
end_of_record

why coverage didn't include the untested files?

i know how to show coverage by tools like genHtml locv-sumary

I just want to know how to generate the Locv.info that includes untested files in my projects

[100cm]

i searched the same question on stack overflow:

https://stackoverflow.com/questions/54602840/how-can-i-generate-test-coverage-of-untested-files-on-my-flutter-tests

but none answer it.

[BigBl4ckW0lf]

I have found a temporary workaround for this issue.

If you create a "fileLoader_test.dart" file like this:

// ignore_for_file: unused_import
import 'package:my_package/FileA.dart';
import 'package:my_package/folder_one/lib.dart';
import 'package:my_package/folder_two/lib.dart';
void main() { }

then "flutter test --coverage" will recognize all files specified in the "lib.dart" files.

[100cm]

yep @BigBl4ckW0lf I got this, but i must import all files in my project.

[jonahwilliams]

If a file is not reachable by the entrypoint that executes coverage, then because the VM has never loaded the library it doesn't return any coverage information for us.

The tester process could walk the filesystem and insert the remaining file URIs into the coverage report - it's not clear that would always be meaningful information or the right thing to do.

[dangilbert]

I'm using @BigBl4ckW0lf's idea but didn't want to manually add them in case I forget. I've got a CI step which creates the file before running the coverage using this script

#!/bin/sh
file=test/coverage_helper_test.dart
echo "// Helper file to make coverage work for all dart files\n" > $file
echo "// ignore_for_file: unused_import" >> $file
find lib -name '*.dart' | cut -c4- | awk -v package=$1 '{printf "import '\''package:%s%s'\'';\n", package, $1}' >> $file
echo "\nvoid main(){}" >> $file

Just call it with your package name as an argument and it'll generate a test including all dart files in the lib folder

[SBNTT]

Here is a version which exclude .g.dart files

#!/bin/sh
file=test/coverage_helper_test.dart
echo "// Helper file to make coverage work for all dart files\n" > $file
echo "// ignore_for_file: unused_import" >> $file
find lib -not -name '*.g.dart' -and -name '*.dart' | cut -c4- | awk -v package=$1 '{printf "import '\''package:%s%s'\'';\n", package, $1}' >> $file
echo "void main(){}" >> $file

[mbilawisdom]

Here is a version which exclude .g.dart files

#!/bin/sh
file=test/coverage_helper_test.dart
echo "// Helper file to make coverage work for all dart files\n" > $file
echo "// ignore_for_file: unused_import" >> $file
find lib -not -name '*.g.dart' -and -name '*.dart' | cut -c4- | awk -v package=$1 '{printf "import '\''package:%s%s'\'';\n", package, $1}' >> $file
echo "void main(){}" >> $file

were do I have to add thuis

[vrnvorona]

any way to avoid part libraries? Or just ignore the warning?

[Kiruel]

Hi,
If it helps I make my own script for that:

#!/bin/sh
file=test/coverage_helper_test.dart
printf "// Helper file to make coverage work for all dart files\n" > $file
printf "// **************************************************************************\n" >> $file
printf "// Because of this: https://github.com/flutter/flutter/issues/27997#issue-410722816\n" >> $file
printf "// DO NOT EDIT THIS FILE USE: sh scripts/import_files_coverage.sh YOUR_PACKAGE_NAME\n" >> $file
printf "// **************************************************************************\n" >> $file
printf "\n" >> $file
printf "// ignore_for_file: unused_import\n" >> $file
find lib -type f \( -iname "*.dart" ! -iname "*.g.dart" ! -iname "*.freezed.dart" ! -iname "generated_plugin_registrant.dart" \) | cut -c4- | awk -v package="$1" '{printf "import '\''package:%s%s'\'';\n", package, $1}' >> $file
printf "\nvoid main(){}" >> $file

Don't forget to change YOUR_PACKAGE_NAME by your package.

And you can also ignore this file on your .gitignore.

/test/coverage_helper_test.dart

[saurabh-sablok]

any way to avoid part libraries? Or just ignore the warning?

@vrnvorona do you found any way to avoid part libraries from coverage_helper_test.dart file?

[vrnvorona]

any way to avoid part libraries? Or just ignore the warning?

@vrnvorona do you found any way to avoid part libraries from coverage_helper_test.dart file?

Didn't try anything yet sadly, so no.

[danhunsaker]

any way to avoid part libraries? Or just ignore the warning?

@vrnvorona do you found any way to avoid part libraries from coverage_helper_test.dart file?

Add a -not -iname '*.part.dart' to the find command.

[humblerookie]

Its been quite some time and yet there seems to be little progress here, @jonahwilliams anything particular that's blocking this issue? Is it slated to be fixed in the next dart release?

[pdblasi]

To add to the workarounds previously provided, I wrote one up in dart that also gets rid of files with a part of directive. Since I've already got dart anywhere I need to run it, it was easier to write and use a dart script.

Just replace the packageName value with your actual package name and run with dart run in the root of your repo/package. You can also add in additional checks to the where clause around line 15 if you need to customize the files that are included.

import 'dart:io';

const packageName = 'YOUR_PACKAGE_NAME_HERE';

void main() async {
  final cwd = Directory.current.uri;
  final libDir = Directory.fromUri(cwd.resolve('lib'));
  final buffer = StringBuffer();

  var files = libDir
      .listSync(recursive: true)
      .whereType<File>()
      .where((file) =>
          file.path.endsWith('.dart') &&
          !file.path.contains('.freezed.') &&
          !file.path.contains('.g.') &&
          !file.path.endsWith('generated_plugin_registrant.dart') &&
          !file.readAsLinesSync().any((line) => line.startsWith('part of')))
      .toList();

  buffer.writeln('// ignore_for_file: unused_import');
  buffer.writeln();

  for (var file in files) {
    final fileLibPath =
        file.uri.toFilePath().substring(libDir.uri.toFilePath().length);
    buffer.writeln('import \'package:$packageName/$fileLibPath\';');
  }

  buffer.writeln();
  buffer.writeln('void main() {}');
  buffer.writeln();

  final output =
      File(cwd.resolve('test/coverage_helper_test.dart').toFilePath());
  await output.writeAsString(buffer.toString());
}

[kbublitz]

Just came across this issue as well and I found there is also a flutter package that does the exact same thing:
full_coverage
It only mentions flutter files but I see no reason why it shouldn't work for dart projects that are not flutter apps.

We added it as a dev dependency to our pubspec.yaml and just call flutter pub run full_coverage before running flutter test

[PiotrFLEURY]

Hi everyone

I used a simple workaround on my side.
Just make a fake test file and include the main.dart in it.
Every dart fil will be reachable by the coverage.

In the test dir create a fake_test.dart (name it as you want)

// ignore: unused_import
import 'package:example/main.dart';

void main() {
  // Fake test in order to make each file reachable by the coverage
}

Every file used by the app will be scanned by the coverage tool.

🎉 Enjoy 🎉

[akiller]

To add to the workarounds previously provided, I wrote one up in dart that also gets rid of files with a part of directive. Since I've already got dart anywhere I need to run it, it was easier to write and use a dart script.

Just replace the packageName value with your actual package name and run with dart run in the root of your repo/package. You can also add in additional checks to the where clause around line 15 if you need to customize the files that are included.

Thanks for this.

Just a note, if you're using Windows (boo hiss, I know!) you need to replace \ with / for the generated imports, e.g.:

final fileLibPath = file.uri
        .toFilePath()
        .substring(libDir.uri.toFilePath().length)
        .replaceAll('\\', '/'); 

[jayudey-wf]

I've used the approach suggested above in the comments and it's worked great for most of my use cases, however I was wondering if there was a way to perform this action (include untested files in the coverage calculation) when the package uses the build_runner package for generating files?

[PiotrFLEURY]

I think the better approach is to really test the main.

By default any new Flutter app already includes a Widget test for the counter app.
Most of us just deleted it once we change the main.dart content. In my opinion this is our first mistake.

The workaround I suggested can provide metrics about uncovered sources but a cleaner way to do should be to create a simple app should start Widget test like Spring boot does when you generate a new backend from start.spring.io

I am wondering if we should include this test by default in the flutter create template.

[CBeening]

What if there is no main?

I have a project that is just a collection of many widgets and some logic used by different apps.
They are not used within the project.

Flutter test will not pick them up, therefore the coverage is off by a lot.

[PiotrFLEURY]

The current coverage engine works fine.

The real problem is files not included in the coverage because no Dart file import it.

What if we create a Dart CLI package in order to add missing files in the lcov file juste after coverage execution ?

The process could be

1. run flutter test --coverage
2. scan source files and add missing ones in the lcov file as "not covered"

Both steps could be included in a single CLI command

The most complicated part will be to scan Dart sources in order to get an lcov format output.

This way, real coverage value could be reported.

I'll try to create a package and see if it works or not

[PiotrFLEURY]

Hi

Can someone give a try on https://pub.dev/packages/discover ?

I think it's a good start.

Open any Flutter projet and run:

dart pub global activate discover
discover scan

Discover will

• scan your project sources
• search for a coverage file
• try to generate coverage if not exists
• search for source files not listed in coverage file
• create another coverage file listing missing source files
• generate HTML report including actual computed coverage including not tested and not imported files

Let me know how I can improve it.

@CBeening let me know if it works on your Flutter widgets package.

[tekneurt]

I have always used a solution based on this comment by pdblasi which has given good results, basically generating a huge import file including all relevant project files (except generated ones), but having to update this file all the time is sub-optimal, so looking forward to this new package, which is way cleaner.

I did a few tests with a small projects and the only thing I noticed so far is, that that my barrel files (which only contain export 'file_name.dart' lines are being taken into account in the coverage report, while in the normal coverage reports, they are ignored/absent. This was the only thing I found so far, so looking good!

EDIT: To not spam this thread... @PiotrFLEURY
Some additional remarks:
lines that"import","part" or "part of" a file should be ignored
lines that are comments should be ignored
lines that declare a class, mixin or extension should be ignored
lines that are closing statements with ), ], ;, should be ignored
lines that return a variable or contain parameters for a function/constructor should be ignored

All these lines contribute to a lower coverage percentage when using discover, compared to the regular way.

Another thing I noticed is that when a file is missing, "discover scan" will fix the coverage AND open the lcov-report. However when all files are already referenced, I get a message "All Dart files are listed in the coverage file." and the lcov-report needs to be manually generated. It would be nice (either through a setting or by default) that the result (and side effects)of "discover scan" is always the same: either always output the result of scan or generate the report and open it. This way it's easy to include the command in a shell script. Normally I run a shell script by typing dcov which will execute the following. It would be nice if "discover scan" could either fit in the script or replace it completely.

dart pub global activate coverage;
dart run test --coverage=coverage;
dart pub global run coverage:
format_coverage --lcov --check-ignore --in=coverage --out=coverage/lcov.info --report-on=lib;
genhtml -o coverage coverage/lcov.info;open coverage/index.html

Either way, the package is already really useable, thanks!

[PiotrFLEURY]

Thanks for testing @tekneurt

Barrel files are now ignored by default. Discover package version 0.1.0 has been published including this feat.

I guess I should manage a .discoverignore file in order to let users ignore files as they want.

EDIT: Version 0.2.0 has been publish including .discoverignore file management. Juste add globs to this file and the coverage will ignore them

EDIT: @tekneurt version 0.3.0 has just been published containing better code filtering. Last edit here to not spam this thread about this package. Let's continue on the discover repo. If anything need to be fixed or improved please open an issue. Thanks for testing again ! Happy to help the community with my little contributions.