Add --remote-cache-warnings and make remote cache warnings less chatty (Cherry-pick of #11859)
#11863
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tty (pantsbuild#11859) pantsbuild#11740 added new aggregation of remote cache logs. It was a fine start, but did not work properly for several error messages because we were dumping the headers, which included unique values like timestamps or the digest value: > 23:16:17.28 [WARN] Failed to write to remote cache (1 occurrences so far): Error from server in response to find_missing_blobs_request: Status { code: PermissionDenied, message: "insufficient permissions", metadata: MetadataMap { headers: {"server": "awselb/2.0", "date": "Mon, 05 Apr 2021 23:16:17 GMT", "content-type": "application/grpc", "content-length": "0"} } } Further, these headers were extremely noisy and most users don't care about them. They're now short: > 23:16:17.28 [WARN] Failed to write to remote cache (1 occurrences so far): PermissionDenied: "insufficient permissions" Likewise, for most use cases, it was still too noisy to dump every increment. Our average use case should only log the very first time so that people are aware of the problem, but don't get flooded. In CI, users can set to log more frequently, which is now implemented via exponential backoff based on the exponent of 2. [ci skip-build-wheels]
tdyas
approved these changes
Apr 8, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.