Big variance of initial delay

[zhengying-liu]

Describe the bug
This issue concerns the competitions:
https://autodl.lri.fr/competitions/3
https://autodl.lri.fr/competitions/32
When one makes the same submission, there is a chance to get very long delay. Even though the algorithm, VM configuration, ingestion + scoring, datasets etc are all the same.

The Output Log of ingestion for the submission 6272:

2019-08-23 12:22:34,493 INFO ingestion.py: ************************************************
2019-08-23 12:22:34,493 INFO ingestion.py: ******** Processing dataset Apollon ********
2019-08-23 12:22:34,493 INFO ingestion.py: ************************************************
2019-08-23 12:22:34,493 INFO ingestion.py: Reading training set and test set...
2019-08-23 12:22:34,735 INFO ingestion.py: ===== Start core part of ingestion program. Version: v20190820 =====
2019-08-23 12:22:34,735 INFO ingestion.py: Creating model...
2019-08-23 12:23:22,627 INFO ingestion.py: Begin training the model...
2019-08-23 12:23:41,877 INFO ingestion.py: Finished training the model.
2019-08-23 12:23:41,877 INFO ingestion.py: Begin testing the model by making predictions on test set...
2019-08-23 12:23:46,053 INFO ingestion.py: Finished making predictions.
2019-08-23 12:23:46,204 INFO ingestion.py: [+] 1 predictions made, time spent so far 71.47 sec
2019-08-23 12:23:46,204 INFO ingestion.py: [+] Time left 1128.53 sec

We see that making the first prediction took 72 seconds.

The Output Log of ingestion for the submission 6386:

2019-08-23 14:56:01,055 INFO ingestion.py: ************************************************
2019-08-23 14:56:01,055 INFO ingestion.py: ******** Processing dataset Apollon ********
2019-08-23 14:56:01,055 INFO ingestion.py: ************************************************
2019-08-23 14:56:01,055 INFO ingestion.py: Reading training set and test set...
2019-08-23 14:56:01,299 INFO ingestion.py: ===== Start core part of ingestion program. Version: v20190820 =====
2019-08-23 14:56:01,300 INFO ingestion.py: Creating model...
2019-08-23 14:56:07,437 INFO ingestion.py: Begin training the model...
2019-08-23 14:56:25,904 INFO ingestion.py: Finished training the model.
2019-08-23 14:56:25,904 INFO ingestion.py: Begin testing the model by making predictions on test set...
2019-08-23 14:56:30,127 INFO ingestion.py: Finished making predictions.
2019-08-23 14:56:30,273 INFO ingestion.py: [+] 1 predictions made, time spent so far 28.97 sec
2019-08-23 14:56:30,273 INFO ingestion.py: [+] Time left 1171.03 sec

We see that this time making the first prediction only took 29 seconds.

The Output Log of ingestion for the submission 6506:

2019-08-23 17:41:37,020 INFO ingestion.py: ************************************************
2019-08-23 17:41:37,021 INFO ingestion.py: ******** Processing dataset Apollon ********
2019-08-23 17:41:37,021 INFO ingestion.py: ************************************************
2019-08-23 17:41:37,021 INFO ingestion.py: Reading training set and test set...
2019-08-23 17:41:37,271 INFO ingestion.py: ===== Start core part of ingestion program. Version: v20190820 =====
2019-08-23 17:41:37,271 INFO ingestion.py: Creating model...
2019-08-23 17:41:43,085 INFO ingestion.py: Begin training the model...
2019-08-23 17:42:00,111 INFO ingestion.py: Finished training the model.
2019-08-23 17:42:00,111 INFO ingestion.py: Begin testing the model by making predictions on test set...
2019-08-23 17:42:04,084 INFO ingestion.py: Finished making predictions.
2019-08-23 17:42:04,240 INFO ingestion.py: [+] 1 predictions made, time spent so far 26.97 sec
2019-08-23 17:42:04,240 INFO ingestion.py: [+] Time left 1173.03 sec

We see that this time making the first prediction only took 27 seconds.

So the normal level should be around 28 seconds. In addition, if we look at the same submission for other datasets, this delay is common.

This variance affects the final ALC score a lot since for the above 3 submissions, the ALC scores are respectively:
0.5583, 0.6963, 0.6646

Screenshots
Comparison of different learning curves:

To Reproduce
Steps to reproduce the behavior:

1. Go to https://autodl.lri.fr/competitions/32
2. Make exact the same submissions twice
3. Look at the difference of scores for the two submissions
   As the delay varies in a unpredictable way, it's hard to reproduce exactly the same situation.

Expected behavior
The variance should be a lot less. The normal level standard deviation of ALC score should be less than 0.03.

Desktop (please complete the following information):

• Docker: Ubuntu

Smartphone (please complete the following information):
[Not relevant]

Additional context
None