Fix-ish hyperopt leak

[scarlehoff]

I haven't been able to actually fix the memory leak, so I took a different route. Instead of fixing the leak, now each hyperopt trial is run as forked process that dies once the trial finishes.

With this we can run hyperopt for as many trials we want without hitting OOM. I'm bypassing it when running hyperopt in parallel because in that case one can just leave the database running and run many small jobs and then it would be better not to have the overhead of these changes.

Not very beautiful but functional*. I've left also my previous attempts (in internal_state.py) at cleaning up tensorflow which were not enough but they did clean up things that were dangling so I think it might be worth having them.

(tested by submitting many jobs at BSC and none of them failed due to an OOM, which was the case before! I'll add an actual benchmark)

[Radonirinaunimi]

Thanks @scarlehoff for this. I will have a look ASAP before the end of this week.

[scarlehoff]

Thanks! Turns out this is going to be important because we need to redo the hyperopt with MHOU (which certainly requires be very light on memory :__ )

[scarlehoff]

The promised benchmark, running in montblanc in Milan, with the memory peaks:

A first run small run (so take the times with a grain of salt, since due to the computer and the size of the runcard, the overhead is irrelevant, the memory was certainly the cause of the difference)

master: 22 GB, 2.5 hours

this branch:  16 GB, 30 minutes

A second run, with a greater number of datasets and replicas

master: nothing, OOM, i got a peak of 28 GB before the OOM which happened after 1 hours

this branch: 24 GB, it finished after 6 hours

[Radonirinaunimi]

This looks generally fine to me. I only have very minor comments/suggestions below and they are mainly pedantic aspects. What I think are really missing though are unit tests, especially for _run_trial_in_subprocess.

We could for example have simple tests that:

• the actual forks really have different PID than the parent
• in case of no forks/macos it falls back directly to the current process
• the timeout is enforced
• ...