solidity-coverage doesn't work correctly when parallel mode is enabled

[fvictorio]

This issue is not exactly actionable, because a decent solution is probably a breaking change, but I'm opening it anyway because it's useful input for whenever we tackle the whole test runner thing.

For details, see this comment

[hellwolf]

any chance to reconsider this?

[hellwolf]

Our test harness is still using hardhat, kudos to the hardhat developers, that now it runs around 6m, using github default runner (https://github.com/superfluid-finance/protocol-monorepo/actions/runs/10631733383/job/29473286607) where we use parallel mode and 4 runners.

In contrast, our coverage test with the exact same set of tests takes more than one hour (https://github.com/superfluid-finance/protocol-monorepo/actions/runs/10631733383/job/29473287853) !

Is there anything we can help to get this resolved?

[cgewecke]

Hi @hellwolf, I help maintain solidity-coverage.

A partial workaround for this problem is to set up CI to run coverage in parallel. The basic idea is to:

• partition the set of test files passed to the coverage task
• split coverage into several concurrent jobs, passing test file targets as arguments using the --testfiles command line flag
• cache the coverage results in shared storage as each job completes
• combine results in a final step by either:
  • using the instanbul-combine-updated tool
  • passing them as a batch to Codecov CI which automatically merges them on their end

There's a nice example of this being done in CircleCI at Synthetix:

• job-unit-tests-coverage.yml
• job-unit-tests-coverage-report.yml

There is a github action (with recipe) for splitting a test suite into equal time groups for parallel execution here:

• https://github.com/marketplace/actions/split-tests

(If you try this at superfluid and get it working in Github Actions could you open an issue about it at solidity-coverage? I will add it as an example to the documentation there.)

[cgewecke]

Thinking about this again, there might be a path to supporting --parallel from within the solidity-coverage plugin - have left a note about that in solidity-coverage 691#comment