BUG missing bagged scores.csv when collecting the data [AWS workers]

[tomMoral]

The dispatcher fails sometime complaining about missing bagged_score.csv when collecting the results.
We get the following tracceback:

Traceback (most recent call last):
  File "/opt/miniconda3/bin/ramp-launch", line 8, in <module>
    sys.exit(start())
  File "/opt/miniconda3/lib/python3.7/site-packages/ramp_engine/cli.py", line 122, in start
    main()
  File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/miniconda3/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.7/site-packages/ramp_engine/cli.py", line 90, in dispatcher
    disp.launch()
  File "/opt/miniconda3/lib/python3.7/site-packages/ramp_engine/dispatcher.py", line 297, in launch
    self.update_database_results(session)
  File "/opt/miniconda3/lib/python3.7/site-packages/ramp_engine/dispatcher.py", line 263, in update_database_results
    set_bagged_scores(session, submission_id, path_predictions)
  File "/opt/miniconda3/lib/python3.7/site-packages/ramp_database/tools/submission.py", line 736, in set_bagged_scores
    index_col=[0, 1])
  File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 676, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 448, in _read
    parser = TextFileReader(fp_or_buf, **kwds)
  File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 880, in __init__
    self._make_engine(self.engine)
  File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 1114, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 1891, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 374, in pandas._libs.parsers.TextReader.__cinit__
  File "pandas/_libs/parsers.pyx", line 673, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File /home/ramp/ramp_deployment/events/human_locomotion_datacamp2020/predictions/submission_000003805/bagged_scores.csv does not exist: '/home/ramp/ramp_deployment/events/human_locomotion_datacamp2020/predictions/submission_000003805/bagged_scores.csv'

[kegl]

Maybe it's a timing issue? The submission was pulled before bagging finished at the end? How does the dispatcher know when the submission has finished training?

[kegl]

Perhaps _training_succesful in https://github.com/paris-saclay-cds/ramp-board/blob/master/ramp-engine/ramp_engine/aws/api.py#L800 should check if bagged_scores.csv is there, not only the prediction files. It's just a guess.