Feature/retry config

[Minimonium]

Changelog: Feature: Implement conan.conf retry and retry-wait and CONAN_RETRY and CONAN_RETRY_WAIT to configure all retries for all transfers, including upload, download, and tools.download().
Docs: conan-io/docs#1295

Closes #1215

The feature to pull up download/upload retry and retry_wait configuration to conan.conf and environment variables CONAN_RETRY and CONAN_RETRY_WAIT.

By using them as config variables the user can customize the Conan network behaviour specific to the machine in a unified manner instead of defining the flags in each related command.
Parameters of the upload command and the tools.download force override the default values for customizable behaviour.

It requires to change the default values of parameters of the upload command to None, so we'd fall back to the configuration values.
Also, because of the unified value system default values are now consistent across the API. Because of that, it's required to change old tests since messages on the default behaviour on retries has changed.

The problem of pushing them via an API for the general download (and related) commands is that we'd need to propagate them all across the codebase:

uploader_downloader [Downloader]
rest_client_v1, rest_client_v2, rest_client_common? rest_client, net (default is fine)
auth_manager
remote_manager
installer [BinaryInstaller], cmd/download, source [complete_recipe_sources]
manager [install], graph_manager_base [build_graph?]
conan_api [workspace_install, install, install_reference, download]      
command [install, workplace, download]

And it would clutter the code with forwarded parameters which is not really worth it.

[Minimonium]

Consider a different name for retry and retry_wait though

[lasote]

I like the idea, but you are changing the default retry attempts, and that is breaking the behavior. Changing that shouldn't be needed, right?

[memsharded]

I'd prefer another approach for implementing this, what do you think @lasote

[memsharded]

I'd prefer a different approach.
As there are arguments existing, use them, and if another value is to be used from the conan.conf, lets inject it much earlier in the call stack, as soon as possible.
In other words, better to have it coming always from argument than having to read another env-var here.

[Minimonium]

I like the idea, but you are changing the default retry attempts, and that is breaking the behavior. Changing that shouldn't be needed, right?

It's needed to detect when the value is forced and when the value is the defaulted one. I agree though that it's better to preserve existing defaults. Will see how to pull extraction up.

[Minimonium]

So, I've pulled up the fallback to the config values for the upload in conan_api because config values are applied only from that level, but left fall back values in uploader_downloader to keep the old behaviour (I left it in for non-obvious calls to preserve the old behaviour with old defaults).
The only place where I'm not sure it makes sense to have default ones is in conans/client/cmd/uploader.py (and we don't have tests to check for that one):
https://github.com/conan-io/conan/pull/5174/files#diff-ad1674306ad35622cdaa15f2eca15c81

Also, I have a problem with the tools.download. Since it makes sense to apply the config to everything, but the call here isn't wrapped by config environment variables, so how should one provide the context?
https://github.com/conan-io/conan/blob/develop/conans/client/tools/net.py#L59

[Minimonium]

I guess something like #5204? For the tools part.

[memsharded]

Please have a look to those minor comments, and I think it could be merged after that.

[Minimonium]

Note my comment about tools.download.

[Minimonium]

Docs: conan-io/docs#1295

[memsharded]

Hi @Minimonium

Each conan_api call is wrapped with:

with tools.environment_append(api._cache.config.env_vars):
                # Patch the globals in tools
                return f(*args, **kwargs)

So I'd say tools.download usage of env-vars will be fine.

[memsharded]

Merged, will be in next 1.16, many thanks!

Check the first 2 lines I added in your first comment:

Changelog: Feature: Implement conan.conf retry and retry-wait and CONAN_RETRY and CONAN_RETRY_WAIT to configure all retries for all transfers, including upload, download, and tools.download().
Docs: conan-io/docs#1295

They are used to automatically generated the changelog each release :)