Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Merge multiple builds from different CI but same commit hash #613
I have a setup with two CI servers, one using Linux (Travis) and the other Windows (Appveyor); however, the coverage reports appears to be separated even though the commit hash is the same.
Shouldn't reports for the same commit, regardless of the CI service or job ID, be merged ?
I recently added Windows support and AppVeyor CI builds to one of my projects (paylogic/pip-accel#61) and ran into this same issue.
I actually forked the coveralls-python project in an attempt to add proper AppVeyor support, only to find out that it appears to be impossible to get merged coverage statistics from multiple CI environments, because now I just get two separate coverage reports even though the branch names, build numbers and commit hashes all match :-(. It looks quite confusing in the Coveralls web interface because everything matches except for the "Via" column (which is now correctly populated due to my fork of coveralls-python).
I think I've gone through all of the available Coveralls API documentation but there appears to be no documented way to get these reports combined. The only nasty hack I haven't seriously considered yet is to get Travis CI and AppVeyor to just generate
A tangentially related observation: Coverage reports originating from Travis CI use regular (forward) slashes but coverage reports originating from AppVeyor use backslashes. Before encountering this issue I was half anticipating that reports from multiple CI environments could be merged, but would clash due to the different path separators. Well, I guess that potential issue isn't relevant for now :-).
added a commit
Oct 31, 2015
I wonder if anyone from the Coveralls-team is reading this, because I think this would be a feature request.
I've noticed that while multiple builds of the same commit are grouped nicely if they are coming from Travis, reports from Appveyor are coming one build at a time. At https://coveralls.io/github/nknapp/thoughtful-release, build
One could certainly build a service that collects coveralls-reports, merges the reports with the same commit-ish and sends the combined report to Coveralls.
The real solution however would be to group reports by the commit-ish within Coveralls, so that the Appveyor builds would just be added to the TravisCI builds. But this is something that only the Coveralls-Team can do...
I made https://github.com/Robpol86/appveyor-artifacts while I wait for Coveralls to support multiple CIs. It's Python only though (haven't tested it with any other coverage file format).
It's similar to what xolox suggested. I have Travis CI wait for my AppVeyor tests to finish and then merge its coverage results into a Travis' .coverage file. This lets Coveralls see full coverage from Linux and Windows tests. I have to modify the downloaded AppVeyor .coverage file to change the file paths to Linux paths.
I really hope to see Coveralls support simultaneous CIs some day though. I've been waiting since November 2014 :(