No coverage report data on windows-latest workflows with tox >=4

[stephane-caron]

Issue

Since the release of tox 4 the qpsolvers project has been experiencing windows-latest workflow issues: qpsolvers/qpsolvers#141. The symptom is:

py3*-windows: commands[7]> coverage report --include=\"qpsolvers/*\"
  No data to report.

Which makes the CI fail. Version-pinning tox to 3.28.0 removes the symptom, as checked in qpsolvers/qpsolvers@d9b4ee7 by this job.

This issue only happens only on windows-latest workflows (ubuntu-latest or macos-latest workflows are fine). It seems to have appeared between tox 3.28.0 and tox 4.0.0 (see the bisection in Additional context below).

Environment

Provide at least:

• OS: windows-latest from GitHub Actions
• pip list of the host Python where tox is installed:

Package            Version
------------------ ---------
cachetools         5.2.0
certifi            2022.12.7
chardet            5.1.0
charset-normalizer 2.1.1
colorama           0.4.6
coverage           6.5.0
coveralls          3.3.1
distlib            0.3.6
docopt             0.6.2
filelock           3.8.2
idna               3.4
packaging          22.0
pip                22.3.1
platformdirs       2.6.0
pluggy             1.0.0
pyproject_api      1.2.1
requests           2.28.1
setuptools         56.0.0
tomli              2.0.1
tox                4.0.17
tox-gh-actions     3.0.0
urllib3            1.26.13
virtualenv         20.17.1

Output of running tox

Provide the output of tox -rvv: windows-latest-tox-rvv.log

Minimal example

If possible, provide a minimal reproducer for the issue:

• GitHub Actions configuration: test.yml
• tox configuration: tox.ini
• Link to a GitHub Actions job that fails: this one

One way to reproduce the issue is to:

• Fork qpsolvers
• Edit .github/workflows/test.yml to remove the version pin from the following step:

            - name: "Install dependencies"
              run: |
                  python -m pip install --upgrade pip
                  # tox version: https://github.com/tox-dev/tox/issues/2778
                  python -m pip install coveralls tox==3.28.0 tox-gh-actions

• Enable and run GitHub Actions in the fork

Additional context

• The issue started happening two days ago with this job, which was a minor change following a successful build.
• Windows jobs fail with tox 4.0.14 as checked in qpsolvers/qpsolvers@e923f90 by e.g. this job
• Windows jobs fail with tox 4.0.0 as checked in qpsolvers/qpsolvers@41b5026 by e.g. this job
• Windows jobs succeed with tox 3.28.0 as checked in qpsolvers/qpsolvers@3882478 by e.g. this job

[gaborbernat]

This is a big heavyweight. Can you please create a minimal reproducer? Also, tox 4 was released on Dec 6; how come only started two days ago?

[stephane-caron]

This is a big heavyweight. Can you please create a minimal reproducer?

Done: https://github.com/stephane-caron/tox-windows-issue

Also, tox 4 was released on Dec 6; how come only started two days ago?

I have no idea. I'm just reporting all the observations I made in the hope it may help 🙂

[stephane-caron]

Checked in the minimal reproducer that:

• Jobs succeed with tox==3.28.0: checked with this commit by this job.
• Jobs fail with tox==4.0.18: checked with this commit by this job.

[gaborbernat]

Sounds like a duplicate of #2635. Did you try without quoting the include param?

[stephane-caron]

The job succeeds without the quotes: https://github.com/stephane-caron/tox-windows-issue/actions/runs/3800268866/jobs/6463582301

Not allowing the quotes breaks compatibility with e.g. zsh. Here is what happens in my shell when running the command from this repro without quotes:

src:tox-windows-issue$ coverage report --include=tox_windows_issue/*
zsh: no matches found: --include=tox_windows_issue/*
src:tox-windows-issue$ coverage report --include="tox_windows_issue/*"
Name                            Stmts   Miss  Cover
---------------------------------------------------
tox_windows_issue/__init__.py       5      0   100%
---------------------------------------------------
TOTAL                               5      0   100%

Feel free to close this issue if it is indeed a duplicate. I'll chime in in #2635.