Discussion: Improve gcovr project management/tooling

[chrta]

I would like to start a discussion on how we can improve gcovr on a project level. Maybe we can have some kind of realtime discussion on gitter, teams, matrix, jitsi or anywhere else.

Some ideas i would like to talk about:

• use github milestones to manage versions
  • this should reduce the required work once we do a release
• maybe have a very simple roadmap on https://github.com/gcovr/gcovr/projects
• check if we can improve some tooling to ensure certain quality level for contributions (though it is already pretty good)
  • use https://jorisroovers.com/gitlint/ to check commit messages
  • document rules in contribution guide
  • document what we expect in commits, see e.g. https://docs.zephyrproject.org/latest/contribute/index.html#commit-guidelines
• check if we can include different compiler versions in the matrix build and automated tests (e.g. gcc-9 and clang)

[latk]

Those are some very good ideas, and since you have write access you can work towards what you consider important.

Assorted remarks:

• chat: there's a Gitter room that's also bridged into Matrix (#gcovr:matrix.org)
• milestones: I don't know, they seem more useful if you have fixed releases that you're working towards, whereas gcovr has no system or schedule. On the other hand, this could make it easier to collect issues/PRs that should be mentioned in the changelog.
• roadmap: go for it, create a project!
• tooling: tooling is good. The current problem is that tooling results like linter errors are often hidden in the Travis CI log. Porting tooling to Github Actions can help with that.
  • one thing I've considered to keep debt low is to have a check that ensures the changelog is updated, unless the PR indicates that no mention is necessary.
• documenting rules, commit guidelines, …: being explicit about expectations is generally good, but requiring new contributors to read too much is counterproductive

And on the matter of matrix builds: yes, being shackled to GCC 5 for the test suite is a real problem, see also #134. In the forseable future, it will become unreasonably difficult to install this version on vanilla systems. We also have no automated way to test upcoming GCC releases.

The solution I expect is to JSONify our test suite further, so that there can be separate reference files for different compiler versions, to the degree that there are differences. The other reports are independent from compilers and can be generated from canonical JSONs with the add-tracefile feature (very fast!). This prevents us from having to run a full (test × output format × compiler × Python version) matrix, we only need test × (output format + compiler + Python version).

It's also not necessary to run every test with every compiler, as long as CI runs the full matrix to prevent surprises. The difficult part is tagging the test cases to figure out what test should run in which environment, and finding a way to efficiently update references when the tests change.

[Spacetown]

I agree with you. It will also be good to have temlates for new issues and PR's in the project.
But before adding other tooling I prefer to port the existing CI to GitHub Actions to have a single place for the CI. After this the CI can be extended with additional checks.