feat: Add retry logic when downloading plugins from GitHub

[cychiang]

Summary

Fixes #291

• Add retry with retry-go library
• Start retry when status code != 200
• By default, it will retry 5 times and wait for 1 second between each attempt.

Here is the example output if status code != 200:

failed downloading: https://github.com/cloudquery/cloudquery/releases/download/plugins-source-my-test-v1.1.5/test_darwin_amd64.zip: All attempts fail:
#1: statusCode != 200
#2: statusCode != 200
#3: statusCode != 200
#4: statusCode != 200
#5: statusCode != 200

---

Use the following steps to ensure your PR is ready to be reviewed

• Read the contribution guidelines 🧑‍🎓
• Run go fmt to format your code 🖊
• Lint your changes via golangci-lint run 🚨 (install golangci-lint here)
• Update or add tests 🧪
• Ensure the status checks below are successful ✅

[erezrokah]

Thanks for the PR @cychiang! WDYT about using https://github.com/avast/retry-go for the retry logic?

I think it could simplify the code and also avoid the need to write test cases as we rely on an already tested library.

Please let me know that you think

[cychiang]

Thanks for the PR @cychiang! WDYT about using https://github.com/avast/retry-go for the retry logic?

I think it could simplify the code and also avoid the need to write test cases as we rely on an already tested library.

Please let me know that you think

Yes, I do also thinking about the library you mentioned, focusing on the retry itself. But go-resty supports save response into file in the other hand might be helpful to simplify the codebase? But if the project would like to support downloading plugins other than GitHub in the future then I think would be good to consider which one would be good in the next step. Maybe I just make this issue more complicate and confuse 😶‍🌫️

[erezrokah]

Yes, I do also thinking about the library you mentioned, focusing on the retry itself. But go-resty supports save response into file in the other hand might be helpful to simplify the codebase? But if the project would like to support downloading plugins other than GitHub in the future then I think would be good to consider which one would be good in the next step. Maybe I just make this issue more complicate and confuse 😶‍🌫️

That's a very good point about go-resty. How about we try with retry-go as that would be a smaller change and will resolve #291, and open a new issue to consider using go-resty so we can discuss the benefits of that approach?

I can open the issue to use go-resty or you can do that too and explain the benefits. Please let me know what you prefer

[cychiang]

Yes, I do also thinking about the library you mentioned, focusing on the retry itself. But go-resty supports save response into file in the other hand might be helpful to simplify the codebase? But if the project would like to support downloading plugins other than GitHub in the future then I think would be good to consider which one would be good in the next step. Maybe I just make this issue more complicate and confuse 😶‍🌫️

That's a very good point about go-resty. How about we try with retry-go as that would be a smaller change and will resolve #291, and open a new issue to consider using go-resty so we can discuss the benefits of that approach?

I can open the issue to use go-resty or you can do that too and explain the benefits. Please let me know what you prefer

Sounds good, let's put go-resty to another issue and focus on retry-go for the #291 😄

[erezrokah]

This looks great, thanks for the follow up @cychiang 👍

I'll let @yevgenypats comment too

[erezrokah]

I'll go ahead and merge (and release this) as it fixes the bug. We can do any follow in future PRs.

One thing I noticed is that the error is printed at the end of all the retry attempts. It would be better to print the attempt on each iteration, which I fixed in a4a4f5f, see:

Loading spec(s) from examples/config-azure.yml
Failed downloading https://github.com/cloudquery/cloudquery/releases/download/plugins-source-azure-v1.2.0/azure_darwin_arm64.zip with status code <code>. Retrying
Failed downloading https://github.com/cloudquery/cloudquery/releases/download/plugins-source-azure-v1.2.0/azure_darwin_arm64.zip with status code <code>. Retrying
Failed downloading https://github.com/cloudquery/cloudquery/releases/download/plugins-source-azure-v1.2.0/azure_darwin_arm64.zip with status code <code>. Retrying
Failed downloading https://github.com/cloudquery/cloudquery/releases/download/plugins-source-azure-v1.2.0/azure_darwin_arm64.zip with status code <code>. Retrying
Failed downloading https://github.com/cloudquery/cloudquery/releases/download/plugins-source-azure-v1.2.0/azure_darwin_arm64.zip with status code <code>. Retrying
Error: failed to get source plugin client for azure: failed to download plugin: failed downloading: https://github.com/cloudquery/cloudquery/releases/download/plugins-source-azure-v1.2.0/azure_darwin_arm64.zip

[erezrokah]

Thanks for this PR @cychiang 🎸