Merge multiple builds from different CI but same commit hash

[Snaipe]

I have a setup with two CI servers, one using Linux (Travis) and the other Windows (Appveyor); however, the coverage reports appears to be separated even though the commit hash is the same.

Shouldn't reports for the same commit, regardless of the CI service or job ID, be merged ?

[smootoo]

I have same setup as @Snaipe (travis and appveyor). This would be useful for me too.

[xolox]

I recently added Windows support and AppVeyor CI builds to one of my projects (paylogic/pip-accel#61) and ran into this same issue.

I actually forked the coveralls-python project in an attempt to add proper AppVeyor support, only to find out that it appears to be impossible to get merged coverage statistics from multiple CI environments, because now I just get two separate coverage reports even though the branch names, build numbers and commit hashes all match :-(. It looks quite confusing in the Coveralls web interface because everything matches except for the "Via" column (which is now correctly populated due to my fork of coveralls-python).

I think I've gone through all of the available Coveralls API documentation but there appears to be no documented way to get these reports combined. The only nasty hack I haven't seriously considered yet is to get Travis CI and AppVeyor to just generate .coverage data (Python specific example), find some place where a web hook can be made responsible for fetching and combining the raw coverage data and submitting that to Coveralls.

A tangentially related observation: Coverage reports originating from Travis CI use regular (forward) slashes but coverage reports originating from AppVeyor use backslashes. Before encountering this issue I was half anticipating that reports from multiple CI environments could be merged, but would clash due to the different path separators. Well, I guess that potential issue isn't relevant for now :-).

[nknapp]

I wonder if anyone from the Coveralls-team is reading this, because I think this would be a feature request.

I've noticed that while multiple builds of the same commit are grouped nicely if they are coming from Travis, reports from Appveyor are coming one build at a time. At https://coveralls.io/github/nknapp/thoughtful-release, build #72 Collect output in my house is a Travis build with two sub-builds. Build #79 and #80 are Appveyor builds of the same commit. They are not grouped, but seperate.

One could certainly build a service that collects coveralls-reports, merges the reports with the same commit-ish and sends the combined report to Coveralls.

The real solution however would be to group reports by the commit-ish within Coveralls, so that the Appveyor builds would just be added to the TravisCI builds. But this is something that only the Coveralls-Team can do...

[Robpol86]

I made https://github.com/Robpol86/appveyor-artifacts while I wait for Coveralls to support multiple CIs. It's Python only though (haven't tested it with any other coverage file format).

It's similar to what xolox suggested. I have Travis CI wait for my AppVeyor tests to finish and then merge its coverage results into a Travis' .coverage file. This lets Coveralls see full coverage from Linux and Windows tests. I have to modify the downloaded AppVeyor .coverage file to change the file paths to Linux paths.

I really hope to see Coveralls support simultaneous CIs some day though. I've been waiting since November 2014 :(

[felixfbecker]

Does anyone know how other coverage sites handle this? Eg Codecov, Codeclimate? Would be a reason to switch

[felixfbecker]

codecov.io supports this:
http://docs.codecov.io/docs/merging-reports

[jkoritzinsky]

Any updates on this?

[felixfbecker]

@jkoritzinsky I recommend you to switch to codecov, I never looked back

[paralin]

Still want this, but will now go look at codecov because coveralls can't seem to be able to merge multiple reports on the same hash.

[andfoy]

Is there any update on this issue?

[JaimieMurdock]

Any updates on this? About to switch several projects to Codecov as a result.

[plroebuck]

Over three years now since opened. Any progress or wontfix?

[Pr0methean]

I'm having the same problem using a build matrix in Azure Pipelines, since jacoco:report-aggregate doesn't seem to be working properly there -- the merged report comes out empty, and the merged jacoco.exec is missing altogether.

[stevenvachon]

Anyone give a shit?

[stale]

This issue has been automatically marked for closure because it has not had recent activity. It will be closed if no further activity occurs. If your issue is still active please add a comment and we’ll review as soon as we can. Thank you for your contributions.

[Pr0methean]

Keep open

[stale]

This issue has been automatically marked for closure because it has not had recent activity. It will be closed if no further activity occurs. If your issue is still active please add a comment and we’ll review as soon as we can. Thank you for your contributions.

[rivy]

Not stale.

[afinetooth]

Hi @rivy. This feature request precedes me, but I have created a new card for it in our backlog and linked it to this issue and other, potentially similar requests.

In the meantime, I'm afraid coveralls does not officially support (parallel) builds across multiple, different Ci systems.

I can begin to imagine some approaches to work around this until we have such support, but I would like to ask if anyone here has explored and accomplished any such workarounds.

I would be happy to add their approach to documentation to help other users.

[stale]

This issue has been automatically marked for closure because it has not had recent activity. It will be closed if no further activity occurs. If your issue is still active please add a comment and we’ll review as soon as we can. Thank you for your contributions.

[ljharb]

bump

[afinetooth]

Thanks @ljharb. I have bumped the internal card on this. Our engineering team has grown recently, so more of a chance we could tackle this. If it gets picked up, I'll update here.

Still interested in anyone's configuration details who managed to get this working.