New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge multiple builds from different CI but same commit hash #613

Open
Snaipe opened this Issue Sep 3, 2015 · 12 comments

Comments

Projects
None yet
@Snaipe

Snaipe commented Sep 3, 2015

I have a setup with two CI servers, one using Linux (Travis) and the other Windows (Appveyor); however, the coverage reports appears to be separated even though the commit hash is the same.

Shouldn't reports for the same commit, regardless of the CI service or job ID, be merged ?

@smootoo

This comment has been minimized.

smootoo commented Sep 30, 2015

I have same setup as @Snaipe (travis and appveyor). This would be useful for me too.

@xolox

This comment has been minimized.

xolox commented Oct 31, 2015

I recently added Windows support and AppVeyor CI builds to one of my projects (paylogic/pip-accel#61) and ran into this same issue.

I actually forked the coveralls-python project in an attempt to add proper AppVeyor support, only to find out that it appears to be impossible to get merged coverage statistics from multiple CI environments, because now I just get two separate coverage reports even though the branch names, build numbers and commit hashes all match :-(. It looks quite confusing in the Coveralls web interface because everything matches except for the "Via" column (which is now correctly populated due to my fork of coveralls-python).

I think I've gone through all of the available Coveralls API documentation but there appears to be no documented way to get these reports combined. The only nasty hack I haven't seriously considered yet is to get Travis CI and AppVeyor to just generate .coverage data (Python specific example), find some place where a web hook can be made responsible for fetching and combining the raw coverage data and submitting that to Coveralls.

A tangentially related observation: Coverage reports originating from Travis CI use regular (forward) slashes but coverage reports originating from AppVeyor use backslashes. Before encountering this issue I was half anticipating that reports from multiple CI environments could be merged, but would clash due to the different path separators. Well, I guess that potential issue isn't relevant for now :-).

xolox added a commit to paylogic/pip-accel that referenced this issue Oct 31, 2015

Replace AppVeyor/Coveralls wrapper with coveralls-python fork
Until my pull request against coveralls-python is merged [1] or an update
from Coveralls is posted on an issue I replied to [2] I guess this is my
last change with regards to Windows support [3] (unless bug reports come
in - let's hope I didn't just jinx that :-).

[1] coveralls-clients/coveralls-python#97
[2] lemurheavy/coveralls-public#613
[3] #61
@nknapp

This comment has been minimized.

nknapp commented Feb 26, 2016

I wonder if anyone from the Coveralls-team is reading this, because I think this would be a feature request.

I've noticed that while multiple builds of the same commit are grouped nicely if they are coming from Travis, reports from Appveyor are coming one build at a time. At https://coveralls.io/github/nknapp/thoughtful-release, build #72 Collect output in my house is a Travis build with two sub-builds. Build #79 and #80 are Appveyor builds of the same commit. They are not grouped, but seperate.

One could certainly build a service that collects coveralls-reports, merges the reports with the same commit-ish and sends the combined report to Coveralls.

The real solution however would be to group reports by the commit-ish within Coveralls, so that the Appveyor builds would just be added to the TravisCI builds. But this is something that only the Coveralls-Team can do...

@Robpol86

This comment has been minimized.

Robpol86 commented May 2, 2016

I made https://github.com/Robpol86/appveyor-artifacts while I wait for Coveralls to support multiple CIs. It's Python only though (haven't tested it with any other coverage file format).

It's similar to what xolox suggested. I have Travis CI wait for my AppVeyor tests to finish and then merge its coverage results into a Travis' .coverage file. This lets Coveralls see full coverage from Linux and Windows tests. I have to modify the downloaded AppVeyor .coverage file to change the file paths to Linux paths.

I really hope to see Coveralls support simultaneous CIs some day though. I've been waiting since November 2014 :(

@felixfbecker

This comment has been minimized.

felixfbecker commented Sep 5, 2016

Does anyone know how other coverage sites handle this? Eg Codecov, Codeclimate? Would be a reason to switch

@felixfbecker

This comment has been minimized.

felixfbecker commented Sep 5, 2016

codecov.io supports this:
http://docs.codecov.io/docs/merging-reports

@jkoritzinsky

This comment has been minimized.

jkoritzinsky commented Sep 22, 2016

Any updates on this?

@felixfbecker

This comment has been minimized.

felixfbecker commented Sep 23, 2016

@jkoritzinsky I recommend you to switch to codecov, I never looked back

Scondo added a commit to Scondo/purepng that referenced this issue Sep 27, 2016

@paralin

This comment has been minimized.

paralin commented Oct 8, 2016

Still want this, but will now go look at codecov because coveralls can't seem to be able to merge multiple reports on the same hash.

@andfoy

This comment has been minimized.

andfoy commented Jul 11, 2017

Is there any update on this issue?

@JaimieMurdock

This comment has been minimized.

JaimieMurdock commented Nov 2, 2017

Any updates on this? About to switch several projects to Codecov as a result.

@plroebuck

This comment has been minimized.

plroebuck commented Nov 1, 2018

Over three years now since opened. Any progress or wontfix?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment