Set number of retries per error code

[severo]

See https://huggingface.slack.com/archives/C04L6P8KNQ5/p1714480880576699?thread_ts=1714033939.429809&cid=C04L6P8KNQ5 (internal)

Errors like CreateCommitError should always be retried, because they correspond to the Hub being down, which should always be a temporary situation. I set the limit to 30 in that case (instead of 3). I set 30 to other error codes as well: HfHubError, LockedDatasetTimeoutError, PreviousStepStillProcessingError

[severo]

CI error is unrelated, see #2759

[severo]

It's working as expected: for the first backfill on retryable errors, a lot of new jobs

[severo]

And the number of CreateCommitErrors is decreasing, soon zero

[severo]

Is there an issue with the metrics? We never get back to zero:

[severo]

For PreviousStepStillProcessingError, it seems it's due to the previous step being JobManagerCrashedError. When we detect that a job has crashed, maybe we're not creating jobs for its children?

I opened an issue: #2765

[severo]

Also: the metrics include details.copied_from_artifact: false | true, ie: we currently have only 4 datasets with the CreateCommitError, but 40 cache entries for all the following steps, and the metrics above show 40.

For the 4 datasets, we reached failed_runs: 30, so we must have a bug here. One of these is https://huggingface.co/datasets/venetis/VMMRdb_make_model_test. I'm not sure what happens. Opening an issue: #2766