Configure protected trunk for Codecov

[adiroiban]

Since codecov correctness is on/off I have created this issue to track enabling / disable codecov status protection
#146 can be used for tracking Travis-CI status

[adiroiban]

I have this PR twisted/twisted#268

and for commit 458167d I can see that codecov/patch — 99.24% of diff hit (target 100%) ... but it reports the status for the merge commit and not the actual patch for this PR

[adiroiban]

I have another commit with the same error
https://github.com/twisted/twisted/compare/trunk...8552-pydoctor-positioning?diff=unified&name=8552-pydoctor-positioning

It reports the coverage report for the merge commit itself (7ebdec7) instead of reporting the coverage for the branch diff

I will disable the codecov checking until the issue is resolved

[glyph]

@adiroiban How are we tracking whether the codecov issue is resolved enough? Is there a ticket open with @codecov somewhere?

[adiroiban]

So for push only builds, which are not attached with any PR, it looks like codecov is reporting the patch coverage for the commit itself, rather then the diff with master.... I am not sure if this is a bug or a feature ... there is little documentation about how the patch coverage should behave (https://codecov.io/docs#yaml_default_commit_status)

codecov issues are here https://github.com/codecov/support/issues

I have not yet reported an issue.

There is also this PR for which the project coverage is not reported ... twisted/twisted#223 ... and in buildbot I can see that codecov command was successful ... and I force another round of builds

I saw that @rodrigc is reporting some of the errors over https://gitter.im/codecov/support

[rodrigc]

https://gitter.im/codecov/support does not seem to be very effective for reporting codecov issues. There seems to be only one person from codecov monitoring that gitter channel, and he seems to be overloaded.

[adiroiban]

I have also disabled codecov/project commit status as it looks like it stays in our way.

will monitor it and we might want to enforce it again in 1 or 2 months when we have better documentation for the development process.

[glyph]

It seems like the problem with these statuses is merely that we haven't set up a proper CI environment (i.e.: we lack PyPy buildbots and therefore PyPy build coverage isn't reported), not an issue with documentation of the development process. What issue would documentation solve here?

[adiroiban]

I was thinking at documenting the unexpected behavior for our coverage reporting + general info about what is this coverage thing

For example if some builder fails to publish the report, codecov will wait longer ... or forever before sending the aggregated report... this is now improved as we now send coverage from Travis even when tests are failing.

I am still monitoring codecov.io activity

Also, I have observed that people are not familiar with coverage reports.

Over the IRC or tickets I saw that people are asking about how to read the coverage reports and how to make sense of them ...

for example wsanchez was not aware about what are the yellow markers

for example we don't have coverage for the 'setup.py' script, and rodrigc was suggesting to have coverage reported by running coverage run setup.py

I have also discussed with non-twisted developers and I found out that they are not familiar with branch coverage thing

codecov.io has some nice features like diff / changes / files and I am not sure if people are aware of these great tools and how to interpret what is presented there :)

We might need to find a few links to help contributors understand what this coverage is all about.

but maybe is just my impression and in fact everything is clear :)

but yes, we now also have public Windows CI ... and I hope that OSX CI will be accepted soon, so we should have a better coverage reporting for PR submitted by non-team members.

[rodrigc]

This is outdated