Test coverage on conditional builds

[mrcasals]

Hi all!

I contacted CodeClimate support and they told me to come here and open an issue, so here I am 😄

I'm contacting you regarding your Test Coverage beta feature. I'm currently not using it, but would like feedback before working on it as my setup might be complex to handle.

Repo: https://github.com/decidim/decidim
RubyGems page: https://rubygems.org/gems/decidim
Ci: CircleCI (https://circleci.com/gh/decidim/workflows/decidim)
Builds: in parallel, using CircleCI's workflows
CI config file: https://github.com/decidim/decidim/blob/master/.circleci/config.yml
CodeClimate config file: https://github.com/decidim/decidim/blob/master/.codeclimate.yml

My repo is a monorepo handling multiple Ruby gems at once. Decidim is a metagem that includes all other subgems found in the repo. On CircleCI we're using their workflows feature to run each gem tests on a different job:

Workflow setup:

This means that every time we push some commits we run the full test suite, running the tests for all gems. Even if we modify only a gem without side effects. Even if we modify the README. To avoid this, I started working on a PR to skip job builds if the commit didn't modify any relevant file of a given gem:

decidim/decidim#2028

This would let us have quicker builds in general. But I realized that CodeCov, our current Test Coverage provider, doesn't know how to calculate the coverage for those gems that did not run their test suite.

Is there any way to solve this with your Test coverage feature? I see you have a way to handle multiple parallel builds, but as far as I know it is based on the fact that all test suites are actually run, which wouldn't be my case.

The only workaround I can think of is separating the project into multiple sub-projects (one project for each gem + one project for the metagem) on CodeClimate, can I have multiple projects on CodeClimate from a single GitHub repo?

Thank you so much for your support!

[maxjacobson]

Hi @mrcasals, thanks for the detailed issue.

Do you suppose it's possible to cache the simplecov artifacts between builds using Circle CI? If so, you could re-use existing results when the tests don't run. When they do run, the new results would overwrite the old ones.

can I have multiple projects on CodeClimate from a single GitHub repo?

This is only possible with private repos. For OSS repos, the URL is based on the GitHub repo slug, and so you can only add an OSS repo once.

[mrcasals]

Hi @maxjacobson!

I guess we would need to save only those reports created on master, and use either those or the ones created in the build. I guess the docs are these:

https://circleci.com/docs/2.0/caching/

I'll try to work on it, see if we get some sane results and report back. Thanks!