Benchmark Early Stopping for the GapEncoder

[simonamaggio]

This PR adds a script to benchmark the use of early stopping for GapEncoder.
The early stopping, implemented in a modified version of GapEncoder and enabled with early_stop=True, monitors the GapEncoder score every monitor_steps batches. As soon as the score increases (worsen) it triggers a counter with a limit given by patience batches. If after patience batches the score has not been decreased, the process is ended.

This PR is related to #593, where a previous benchmark is run to investigate GAPEncoder slowness.

[simonamaggio]

Results of the comparison for a small dataset in skrub: employee_salaries:

[simonamaggio]

Remarks:

• Should we add an early stopping to the GapEncoder class directly and modify the bench script ?
• Should we add the min_delta parameter (like in pytorch), i.e. a decrease of more than or equal to min_delta, will count as improvement? For now there's no such parameter, so equivalent to min_delta=0.
• Should we record the best encoder status so far (score, matrices, ...) so to restore it in case the score worsen instead? Here we always take the last score obtained.
• The outer stopping tol should still be tuned to higher values according to Benchmark GapEncoder divergence #593 results.

[LeoGrin]

Should we record the best encoder status so far (score, matrices, ...) so to restore it in case the score worsen instead? Here we always take the last score obtained.

I think taking the last should be okay, though we can check when I run the benchmarks on more datasets. (also it decreases the memory usage).

Should we add the min_delta parameter (like in pytorch), i.e. a decrease of more than or equal to min_delta, will count as improvement? For now there's no such parameter, so equivalent to min_delta=0.

I think this would be useful! (I'll do it)

[simonamaggio]

Here is the updated result for commit 785082e

[LeoGrin]

Right now the version with early stopping is actually slower than the original version, I think because the score method is so long. I'll try to see if it can be sped up.

[LeoGrin]

See #681