Flaky tests using MPI

[renefritze]

It looks like the 'MPI' CI job sometimes fails due to some intermittent system condition. I have not been able to reproduce it yet. A restart of the job fixes it.

[sdrave]

The job you have linked actually succeeded. Do you have an example for a failing job?

[renefritze]

Not atm. This is basically just an FYI for when you come across a failed MPI job.

[sdrave]

One example of a failing test is here, where the problem seems to be sqlite related:

https://zivgitlab.uni-muenster.de/pymor/pymor/-/jobs/32104

What is kind of strange: by default there is a timeout of 5 seconds on aquiering a lock, and I don't see that there is any operation which might take any longer ..

[renefritze]

Suggestion 1: I replace the SQLiteRegion implementation with a new DiskRegion based on https://github.com/grantjenks/python-diskcache
They claim they're process safe.

Suggestion 2: use an ORM like sqlachemy to abstract to a SQLRegion implementation where the actual database connection is a configuration detail.

[sdrave]

python-diskcache looks kind of neat. Using it would allow us to remove that nasty SQL code from pyMOR. If it will solve the issues with the MPI tests I am not sure. sqlite should be process safe as well. Maybe it's worth giving it a try.

[sdrave]

We haven't seen MPI tests failing since the adoption of diskcache. Let's hope for the best and close this issue.

[renefritze]

Since I haven't been able to quickly determine what's going on with the newly failing MPI tests, I propose to disable them until I've found a solution. Which might take a couple of weeks until I really have time for that.
Objections, @pymor/pymor-devs ?

[renefritze]

If there are none, I'll merge #735 when ready and rebase open PR branches on that

[pmli]

None from me.

[renefritze]

I've rebased the active PRs where jobs were failing in MPI on master now.