Environment groups

[sampsyo]

In Pollen, we have built up a long list of Turnt testing environments that all do differential testing on separate subcommands:
https://github.com/cucapra/pollen/blob/main/test/turnt.toml

So, the chop_oracle environment goes with the chop_test environment in the sense that they share the same output file—i.e., they are doing "differential testing" together. But there are other such pairs too in the same configuration: crush_oracle goes with crush_test, and they do not really relate to chop_*.

Perhaps we should introduce a concept of named "environment groups." Groups of environments would do two things:

• Save some typing in the config file. They could share some options that they have in common.
• Easier invocation together. Perhaps you could do turnt -e crush, for instance, as a shorthand for turnt -e crush_test -e crush_oracle.

[sampsyo]

There's another quality-of-life problem with differential testing setups currently: if you really only want to do differential testing (checking that two commands have the same output) and not expect testing (checking their output against a checked-in saved result file), it's sort of annoying. You have to turnt --save one command, ignore the failures, and then turnt the other command. Maybe some kind of configuration setup could enable this kind of differential testing without manual saving on the part of the user.

[anshumanmohan]

I'm a big fan of something to make differential testing easier somehow! Adrian has covered the pain point precisely: half of the output is of the form not ok... # missing and needs to be ignored.

I just want to point out that, at least in Pollen's situation, the inputs to chop_oracle and chop_test are also different. The Makefile makes this clear, but the situation is:

• There exist something.gfa and something.og. The latter was created from the former, but expensively. I don't want to do it repeatedly.
• chop_oracle receives something.og and I save the output as something.chop.
• chop_test receives something.gfa and I test the output against something.chop created above.

The way to wrestle this into submission would be:

• There exists something.gfa.
• chop_oracle receives something.gfa, creates something.og, runs against the .og, and I save the output as something.chop.
• chop_test receives something.gfa and I test the output against something.chop created above.

But it is wasteful to create the .og repeatedly in this way: most of the other differential tests would have been happy to reuse it. There is currently only one differential test that makes its .og afresh, and that is because it absolutely needs to.