Not uploading coverage reports when there are tests failures/errors

[evandrocoan]

Even with failure, the coverage is generated, but after that its results are not uploaded. The repository is: https://github.com/evandrocoan/PackagesManager

It must always load the results because there is nothing wrong with some tests failing. It is well expected that some tests are going to fail, for some time or even for ever, but just because they fail does not mean they should be moved from the application or block something else.

----------------------------------------------------------------------
Ran 60 tests in 8.894s
FAILED (failures=19)
Name                                                            Stmts   Miss  Cover
-----------------------------------------------------------------------------------
1_reloader.py                                                      61     33    46%
2_bootstrap.py                                                    268    165    38%
PackagesManager.py                                                 48     21    56%
...
packages_manager/upgraders/hg_upgrader.py                          49     38    22%
packages_manager/upgraders/vcs_upgrader.py                          7      4    43%
packages_manager/versions.py                                       61      9    85%
-----------------------------------------------------------------------------------
TOTAL                                                           11537   7185    38%
UnitTesting: Done.
The command "sh travis.sh run_tests --coverage" exited with 1.
Done. Your build exited with 1.

[randy3k]

I believe that it is a common practice not to upload the report upon failures. However, you could modify your travis/appveyor configuration such that the report is still uploaded even upon failures.

[evandrocoan]

Yes, but I do think it is trivial because the upload seems to be done after the https://github.com/SublimeText/UnitTesting/blob/master/sbin/run_tests.py python script finishes.

It seems that run_tests.py return a error code when there are errors/failures on the build. This causes the .travis.yml build to halt and not run the codecov:

UnitTesting/.travis.yml

Lines 48 to 54 in 9b938ee

	after_success:
	- if [ "$TRAVIS_OS_NAME" == "linux" ]; then
	pip install codecov;
	elif [ "$TRAVIS_OS_NAME" == "osx" ]; then
	pip3 install codecov;
	fi
	- codecov

I think I could trick it by forcing the run_tests.py always to return a success code, however it should causes the build badge to always report successful. However, it seems that .travis.yml is cannot report parcial results on its badge, then it should not help too much.

[randy3k]

Instead of putting the codecov code under after_success in .travis.yml, it could be placed under script so that it still uploads the report no matter what the status of the build is.

[evandrocoan]

Thanks, it kinda worked. But I have a hard time trying to edit the .travis.yml. This was the one which seemed to work:

env:
    global:
        - PACKAGE="PackagesManager" # Package name
        - SUBLIME_TEXT_VERSION="3"
        # use UNITTESTING_TAG to specific tag of UnitTesting
        # - UNITTESTING_TAG="master"

# mutliple os matrix
# https://docs.travis-ci.com/user/multi-os/#Python-example-(unsupported-languages)
matrix:
    include:
        - os: linux
          language: python
          python: 3.3
        - os: osx
          language: generic

before_install:
    - curl -OL https://raw.githubusercontent.com/SublimeText/UnitTesting/master/sbin/travis.sh
    - if [ "$TRAVIS_OS_NAME" == "osx" ]; then
            brew update;
            brew install python3;
            pip3 install python-coveralls;
            pip3 install codecov;
      fi
    - if [ "$TRAVIS_OS_NAME" == "linux" ]; then
            pip install python-coveralls;
            pip install codecov;
      fi
    # enable gui, see https://docs.travis-ci.com/user/gui-and-headless-browsers
    - if [ "$TRAVIS_OS_NAME" == "linux"  ]; then
            export DISPLAY=:99.0;
            sh -e /etc/init.d/xvfb start;
      fi

install:
    # bootstrap the testing environment
    - sh travis.sh bootstrap
    # install Package Control and package denepdencies
    # - sh travis.sh install_package_control

script:
    # run tests with test coverage report
    - sh travis.sh run_tests --coverage
    # testing syntax_test files
    # - sh travis.sh run_syntax_tests
    - coveralls
    - codecov

after_success:
    # remove the following if `coveralls` is not needed
    - echo

notifications:
    email: false

[evandrocoan]

I tried the same for AppVeyor, but the script just halted:

UnitTesting: Done.
ScriptHalted

https://ci.appveyor.com/project/evandrocoan/packagesmanager/build/1.0.11

This were the changes to appveyor.yml:

environment:
    # The package name
    PACKAGE: "PackagesManager"
    SUBLIME_TEXT_VERSION : "3"

install:
    - ps: appveyor DownloadFile "https://raw.githubusercontent.com/SublimeText/UnitTesting/master/sbin/appveyor.ps1"
    - ps: .\appveyor.ps1 "bootstrap" -verbose
    # install Package Control
    # - ps: .\appveyor.ps1 "install_package_control" -verbose

build: off

test_script:

    # run tests with test coverage report
    - ps: .\appveyor.ps1 "run_tests" -coverage -verbose
    - "SET PYTHON=C:\Python33"
    - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
    - pip install codecov
    - codecov

after_test:
    - echo

[randy3k]

you may try putting the codecov under after_test for appveyor (not tested).

[evandrocoan]

under after_test for appveyor

It was originally under there. It seems that after_test only runs if all tests are successful. I searched about AppVeyor and I found a documentation page within all commands:

1. https://www.appveyor.com/docs/build-configuration/

Then I tried putting it under some other sections. Hence, it seems to be running when I put it under on_failure section. But something seems stranger. It throws up an not find error on the beginning of the process:

codecov
The filename, directory name, or volume label syntax is incorrect.
FIND: Parameter format not correct
      _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.0.12
==> Detecting CI provider
    AppVeyor Detected
==> Preparing upload
    Error running `cd 'C:\projects\packagesmanager' && git ls-files`: None
==> Processing gcov (disable by -X gcov)
    Executing gcov (find 'C:\projects\packagesmanager' -not -path './bower_components/**' -not -path './node_modules/**' -not -path './vendor/**' -type f -name '*.gcno'  -exec gcov -pb  {} +)
    Error running `find 'C:\projects\packagesmanager' -not -path './bower_components/**' -not -path './node_modules/**' -not -path './vendor/**' -type f -name '*.gcno'  -exec gcov -pb  {} +`: None
==> Collecting reports
XX> Searching for reports disabled
    Generating coverage xml reports for Python
    + C:\projects\packagesmanager\coverage.xml bytes=486379
==> Uploading
    .url https://codecov.io
    .query service=appveyor&package=py2.0.12&job=evandrocoan%2Fpackagesmanager%2F1.0.14&build=5b2ee8r23c64cxvk&branch=master&commit=0899fafeef36aa11a77984cbac7967c1929b4a18&slug=evandrocoan%2FPackagesManager
    Pinging Codecov...
    Uploading to S3...
    https://codecov.io/github/evandrocoan/PackagesManager/commit/0899fafeef36aa11a77984cbac7967c1929b4a18

https://ci.appveyor.com/project/evandrocoan/packagesmanager/build/1.0.14

When it runs under Travis the output is a little different (i.e., with no not find error):

$ codecov
      _____          _
     / ____|        | |
    | |     ___   __| | ___  ___ _____   __
    | |    / _ \ / _  |/ _ \/ __/ _ \ \ / /
    | |___| (_) | (_| |  __/ (_| (_) \ V /
     \_____\___/ \____|\___|\___\___/ \_/
                                    v2.0.12
==> Detecting CI provider
    Travis Detected
==> Preparing upload
==> Processing gcov (disable by -X gcov)
    Executing gcov (find /home/travis/build/evandrocoan/PackagesManager -not -path './bower_components/**' -not -path './node_modules/**' -not -path './vendor/**' -type f -name '*.gcno'  -exec gcov -pb  {} +)
==> Collecting reports
    Generating coverage xml reports for Python
    + /home/travis/build/evandrocoan/PackagesManager/coverage.xml bytes=440303
==> Appending environment variables
    + TRAVIS_OS_NAME
    + TRAVIS_PYTHON_VERSION
==> Uploading
    .url https://codecov.io
    .query slug=evandrocoan%2FPackagesManager&pr=false&package=py2.0.12&service=travis&build=13.1&job=326004718&branch=master&commit=0899fafeef36aa11a77984cbac7967c1929b4a18
    Pinging Codecov...
    Uploading to S3...
    https://codecov.io/github/evandrocoan/PackagesManager/commit/0899fafeef36aa11a77984cbac7967c1929b4a18
The command "codecov" exited with 0.
Done. Your build exited with 1.

https://travis-ci.org/evandrocoan/PackagesManager/jobs/326004718