remove or retain tests in CRAN vegan?

[jarioksa]

We have removed tests from CRAN branch of vegan in the 2.4-series. I am about to branch the current master to cran-2.5 to make a new release, and I planned to remove the tests again. However, I would like to have your comments.

The major reason for removing tests is that tests can complicate merging fixes from master to cran-2.5. There will be no trouble if all changes in tests and their reference output in master are made in separate commits which change nothing but tests. The tests cannot be merged after a while when the code bases diverge, because there will be conflicts. The reference output of tests must be generated by the current cran-2.5 branch instead of merging. If there are commits that mix tests and changes in other files, we have conflicts that must be sorted out by hand. Having a good discipline in master commits, these problems can be avoided.

A minor reason is that running tests takes time in CRAN. However, tests are currently pretty fast and this is not an issue.

The major reason for having tests is quality control in cran-2.5. We do now have that control in master, and we hope it precipitates in cran-2.5 as long as this is based on a sound subset of changes in master. However, missing tests has been an issue (#181) in some derived works.

Another issue is that the test results are now somewhat fragile numerically, and exact sixteenth decimal place results are dependent on platform and environment (OS, hardware, compilers and their settings, other tools, moon phases etc.). This can cause some hassle with us.

[EDiLD]

Would it make sense to have the tests in a separate branch, with a one-way merge from master to tests?

[psolymos]

The cran-2.5 could also drop any *.Rout.save files, because those cause the conflicts. Tests would still run and flag any issues, except for numerically identical outputs (which will be still available in the master).

[jarioksa]

Actually the conflicts are really expected from tests/Examples/vegan-Ex.Rout.save that are compared to the results of running examples. These examples can change with man/*.Rd , and can be difficult to maintain between branches. The numerical fragilities at 16th decimal also crop out with running man/*.Rd examples. The proper tests (tests/vegan-tests.*, tests/cca-object-tests.*, tests/oecosimu-tests.*) should only change when we change the tests/*.R files, and these can also be made (slowly?) tolerant to last bit numerical instability.

So one option could be to remove tests/Examples, but keep the tests in the main tests/ directory.

[jarioksa]

removed tests/Examples reference ouput, but kept the proper tests.