PROTEUS test reliability

[nichollsh]

While the tests are now a lot more reliable, largely due to our move from OSF to Zenodo, they still fail somewhat stochastically. This can be due to network issues in getting data from Zenodo, and sometimes due to differences in nitrogen partitioning found by the outgassing calculations.

We often mitigate this by just re-running the tests until they work. It would be great if the tests were consistent, since we could then take it more seriously any time they fail.

How should this be mapped to an issue? Is it part of #507?

[nichollsh]

@timlichtenberg @stuitje @planetmariana @EmmaPostolec Would be great to hear your thoughts/experiences with this, and potential suggestions.

[stuitje]

I am not deep into the code yet, but if there are network issues, is it possible to decouple the zenodo download from the tests? Pre-download all necessary data from zenodo before running the tests, so that the tests themselves can be run in offline-mode – and potentially retry the download immediately if it fails?

[nichollsh]

Yeah I suppose we should distinguish between testing the download functionality, and testing the rest of the code. It would be ideal to test the downloader functions explicitly, warn if they fail, and then test the physics (etc.) when all the required data are reliably available.

[timlichtenberg]

This seems like a good potential solution. This and the Aragog documentation issue seem like two good project to start with Karen. :)

[stuitje]

@timlichtenberg Is it possible to make this discussion into an issue, assigned to me?

[timlichtenberg]

You are already. :> It's here: FormingWorlds/MORS#33. It's a MORS issue, but it is globally relevant. I think this should not be your first issue to tackle though. Let's discuss this.

[timlichtenberg]

Yes I agree, this is quite annoying. I have no good solution idea that immediately solves the problem. Two related ideas would be: (i) enhance the test printout like we discussed. Then we see at least which specific test fails and perhaps already with a more direct printout from what is going on. (ii) Automatically re-run tests once if they fail after 5 min. Not sure how, but this is probably possible I could imagine.

[nichollsh]

We could readily implement something to re-run the utilities which download data, if they happen to fail. This would then also help if the downloads fail on a user's computer in addition to on the GitHub runners.