Skip to content

Increase monitoring retry deadline.#1290

Merged
inferno-chromium merged 1 commit into
masterfrom
d29
Dec 17, 2019
Merged

Increase monitoring retry deadline.#1290
inferno-chromium merged 1 commit into
masterfrom
d29

Conversation

@inferno-chromium

Copy link
Copy Markdown
Collaborator

Speculative fix for borg bot exceptions.

Failed to flush metrics.
Traceback (most recent call last):
  File "/mnt/scratch0/clusterfuzz/src/python/metrics/monitor.py", line 107, in run
    create_time_series(project_path, time_series)
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/api_core/retry.py", line 206, in retry_target
    last_exc,
  File "/mnt/scratch0/clusterfuzz/src/third_party/six.py", line 740, in raise_from
    raise value
RetryError: Deadline of 90.0s exceeded while calling <functools.partial object at 0x7ec30a301c00>, last exception: 504 Deadline Exceeded

Speculative fix for borg bot exceptions.
```
Failed to flush metrics.
Traceback (most recent call last):
  File "/mnt/scratch0/clusterfuzz/src/python/metrics/monitor.py", line 107, in run
    create_time_series(project_path, time_series)
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/api_core/retry.py", line 286, in retry_wrapped_func
    on_error=on_error,
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/api_core/retry.py", line 206, in retry_target
    last_exc,
  File "/mnt/scratch0/clusterfuzz/src/third_party/six.py", line 740, in raise_from
    raise value
RetryError: Deadline of 90.0s exceeded while calling <functools.partial object at 0x7ec30a301c00>, last exception: 504 Deadline Exceeded
```
@googlebot googlebot added the cla: yes CLA signed. label Dec 17, 2019
@inferno-chromium

Copy link
Copy Markdown
Collaborator Author

/gcbrun

@inferno-chromium

Copy link
Copy Markdown
Collaborator Author

TBR

@inferno-chromium inferno-chromium merged commit d2d7693 into master Dec 17, 2019
@oliverchang

Copy link
Copy Markdown
Collaborator

Thanks! Looks like this is still happening occasionally (although I see some metrics on dashboard). Filed an internal bug.

@oliverchang oliverchang deleted the d29 branch January 30, 2020 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: yes CLA signed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants