Make code coverage CI take filecheck tests into account

[webmiche]

Maybe something with https://llvm.org/docs/CommandGuide/llvm-cov.html can be done.

[webmiche]

AFAIU, the provided link only helps in showing code coverage data. Looking at the lit documentation directly, they don't seem to be able to report code coverage metrics. While a package like coverage.py might be helpful, it does not work in this case as lit is it's own executable.

We might be able to run a sufficiently smart pytest program, that basically does what lit does atm (crawl over files, run the right commands), while also being introspectable. Though I feel that it might actually be easier to just expand our pytest suite to a point where we won't need the filecheck tests to get a reasonable high coverage.

[georgebisbas]

Though I feel that it might actually be easier to just expand our pytest suite to a point where we won't need the filecheck tests to get a reasonable high coverage.

Exactly! :-)

[webmiche]

What I am not 100% sure about is whether this is actually possible/reasonable, in particular around xdsl-opt, as this is really intended to be run from the command line. This becomes particularly true for things around the mlir-converter. I guess we can write some weird tests to circumvent this though...