Retry mechanism for sbt

[hagay3]

A retry mechanism for sbt

Resolves coursier/coursier#2733 and sbt/sbt#7200

Gradle implementation
https://docs.gradle.org/current/userguide/dependency_resolution.html#sub:http-retries

Documentation and tests will be added with more guidance

@alexarchambault @eed3si9n let me know how to proceed :)

[hagay3]

someone can please approve the workflow? :)
@alexarchambault @eed3si9n

[hagay3]

someone can please approve the workflow? :) @alexarchambault @eed3si9n

trying again ..

[eed3si9n]

Overall LGTM

[eed3si9n]

Would you like to squash your commits? Otherwise, I'll squash and merge.

[alexarchambault]

Sorry for blocking the merge until further discussion here, but there are a number of issues in the approach of that PR. I don’t think retrying the whole resolution whenever it failed is the right thing to do. As your link to the Gradle docs describes, Gradle only retries things for individual downloads, and only so for specific error types. Coursier already does that for some TLS-related errors, and that’s the approach that should be extended IMO (I recall several @eed3si9n’s comments going in the same direction)

[eed3si9n]

Here are my comments:

• Transient ResolveExceptions are not properly retried sbt/sbt#6685 (comment)
• Retry when fetching artifact from remote resolver (maven for example) coursier#2733 (comment)
• Transient java.net.ConnectException are not properly retried coursier#2217 (comment)

[alexarchambault]

@hagay3 if ever you feel… too intimidated by the coursier codebase or don’t have too much time to dive into it, and really want to add that here, that might work, but the errors that trigger retries really have to be more specific. Currently, it seems any kind of error triggers a retry, which is too broad (we don’t want to retry after 404 errors for example, which is what worried me here).

I’d advice maybe to have a look at the output of a resolution that you would have liked to see retried, look for the specific exception that made it fail (like a ConnectException for example), and make only those trigger a retry here.

[hagay3]

@hagay3 if ever you feel… too intimidated by the coursier codebase or don’t have too much time to dive into it, and really want to add that here, that might work, but the errors that trigger retries really have to be more specific. Currently, it seems any kind of error triggers a retry, which is too broad (we don’t want to retry after 404 errors for example, which is what worried me here).

I’d advice maybe to have a look at the output of a resolution that you would have liked to see retried, look for the specific exception that made it fail (like a ConnectException for example), and make only those trigger a retry here.

Thank you for your feedback
I added a code that matches CantDownloadModule error, in addition, I'm checking if the error contains "not found" string and then deciding not to retry.
I'm wondering if we want to treat also java exceptions as the list of exceptions is quite large.

[hagay3]

Would you like to squash your commits? Otherwise, I'll squash and merge.

yes please squash my branch :)

[guniatais]

looks good, would love to have that!

[hagay3]

@alexarchambault
hi, added a code regarding specific errors to retry, wdyt?

[alexarchambault]

Sorry for the delay, I'll try to find time for this in the coming days

[hagay3]

Sorry for the delay, I'll try to find time for this in the coming days

Hi, any updates regarding that?
I’ll appreciate any feedback from your side 👍🏻

[hagay3]

it's been a while since there was any update here.
is there a way to push this PR further? @alexarchambault @eed3si9n

[hagay3]

@eed3si9n @alexarchambault
bump ^

[hagay3]

@eed3si9n @alexarchambault
bump 2^

[alexarchambault]

@hagay3 I rebased, tweaked many small things, and changed the way the error is detected (ensuring we retry iff a connection timeout is detected)

Should be good to merge to me, but for the last commit that I need to revert

[alexarchambault]

Merging, thanks @hagay3!

[hagay3]

Thanks @alexarchambault !
@eed3si9n
Can you refer me to the place where documentation can be added for the retry configuration?
Im just not sure how to configure the number of retries and the retry interval in build.sbt