GHA: Measure the code coverage also on MS-Windows #6795
Conversation
Codecov Report
@@ Coverage Diff @@
## master #6795 +/- ##
==========================================
- Coverage 88.51% 85.83% -2.69%
==========================================
Files 147 159 +12
Lines 160762 173353 +12591
==========================================
+ Hits 142305 148804 +6499
- Misses 18457 24549 +6092
Continue to review full report at Codecov.
|
|
Hi,
On Tue, Aug 25, 2020 at 10:00 PM K.Takata ***@***.***> wrote:
And some improvements to the GHA environment.
- Add COVERAGE=yes|no option to Make_cyg_ming.mak.
- Enable to measure the code coverage on MinGW HUGE (x86, x64) on GHA
and upload the data to codecov.
(This doesn't upload the data to coveralls. Not sure it is worth
doing.)
Unfortunately, this will drop the code coverage about 3%. :-(
Why does enabling additional code coverage on MS-Windows decrease the
overall
code coverage? If this will decrease the code coverage by 3%, then this
change should
not be merged.
Regards,
Yegappan
|
|
Because the code coverage of the Windows-specific files like gui_w32.c, os_mswin.c, os_win32.c, etc. were not measured. The coverage of those files is lower than the average. So, they decrease the overall coverage. Of course, it doesn't mean that merging this PR decreases total covered lines. The number of covered lines are increased. |
|
Hi,
On Wed, Aug 26, 2020 at 8:52 AM K.Takata ***@***.***> wrote:
Because the code coverage of the Windows-specific files like gui_w32.c,
os_mswin.c, os_win32.c, etc. were not measured. The coverage of those files
is lower than the average. So, they decrease the overall coverage. Of
course, it doesn't mean that merging this PR decreases total covered lines.
The number of covered lines are increased.
Makes sense. This is similar to the lack of sufficient coverage for the GUI
code in Linux
and other OSes (gui.c, gui_beval.c, gui_dwrite.cpp, gui_gtk.c,
gui_gtk_x11.c, gui_w32.c
and gui_xim.c).
Regards,
Yegappan
|
|
If a line of code is tested under Linux, does it mean it will work under MS-Windows? I wonder if we should have two separate coverage projects: One for Linux and one for MS-Windows. Otherwise we cannot see what code is actually tested in the MS-Windows environment and what is tested in the Linux environment. Most tests will run equally on both platforms, but not everything. Having two projects may help discovering what we are not testing, while it won't really increase the effort for writing tests much. |
Yes, I also think of that, but I couldn't find the way yet. Need more investigation. |
|
I enabled the Flags feature. The percentage of the coverage is still calculated from total of linux and windows. E.g. The coverage on linux: Do you think this is acceptable?
Creating two separate codecov projects doesn't seem possible. |
|
For a larger file switching between Windows and Linux is very slow. Also, the UI doesn't give any feedback, thus it looks like it doesn't work. |
I think so. |
k-takata
force-pushed
the
mingw-coverage
branch
from
869b75d
to
c1215db
Compare
Extracted from vim#6795. This adds `COVERAGE=yes` option to Make_cyg_ming.mak. This makes someone measure the code coverage easily on MS-Windows on a local PC.
k-takata
force-pushed
the
mingw-coverage
branch
from
c1215db
to
e73fb31
Compare
|
Close in favor of #9750. |
And some improvements to the GHA environment.
COVERAGE=yes|nooption toMake_cyg_ming.mak.(This doesn't upload the data to coveralls. Not sure it is worth doing.)
Unfortunately, this will drop the code coverage about 3%. :-(
To fix this, copy all the files in
src/(not onlysrc/testdir/) tosrc2/, and fix some tests.(The tests for gvim.exe is run at
src/testdir/, and the tests for vim.exe is run atsrc2/testdir/.)Some tests are flaky on GHA.
Test_BufWrite_lockmarks()andTest_bufunload_all()are added to the list.Actually, there are some more flaky tests on GHA, but at least these two tests are often failed.
(I tried to fix them, but I couldn't. They don't reproduce on my local environment.)