Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle exceptions thrown from TesApi gracefully with error codes and exception messages #36

Closed
jbagga opened this issue Feb 6, 2020 · 5 comments
Closed

Handle exceptions thrown from TesApi gracefully with error codes and exception messages #36

jbagga opened this issue Feb 6, 2020 · 5 comments
Assignees
@jbagga
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
Bug fixes and feature enhancements
Milestone
2.3

Comments

@jbagga
Copy link
Contributor

jbagga commented Feb 6, 2020 •
edited

Some Batch exceptions are transient and the task should be re-queued/retried
Also need to investigate when to fail permanently
Added Polly for retries - DONE
@jbagga jbagga added the Bug Something isn't working label Feb 6, 2020
@jbagga
Copy link
Contributor Author

jbagga commented Apr 15, 2020 •
edited

As part of this bug fix, also cache quotas for the batch account
DONE

@tracykard tracykard moved this from To do to In progress in Bug fixes and feature enhancements Apr 15, 2020
@jbagga jbagga self-assigned this Apr 15, 2020
@jbagga jbagga mentioned this issue Apr 28, 2020
Merged
@jbagga
Copy link
Contributor Author

jbagga commented Apr 28, 2020 •
edited

Add a TesApiException type to handle exceptions with error codes and proper exception messages when thrown from TesApi

@jbagga jbagga changed the title Handle Batch quota exceptions gracefully and continue to next period Handle exceptions thrown from Tes.Api gracefully with error codes and exception messages Apr 28, 2020
@jbagga jbagga changed the title Handle exceptions thrown from Tes.Api gracefully with error codes and exception messages Handle exceptions thrown from TesApi gracefully with error codes and exception messages Apr 28, 2020
@jbagga jbagga moved this from In progress to To do in Bug fixes and feature enhancements Apr 28, 2020
@jbagga jbagga added the documentation Improvements or additions to documentation label Apr 30, 2020
@tracykard tracykard mentioned this issue Aug 10, 2020
Closed
@jbagga
Copy link
Contributor Author

jbagga commented Aug 10, 2020

Add failure reasons from TES to CosmosDB and surface to trigger files that end up in failed workflows directory

@jbagga jbagga added enhancement New feature or request and removed Bug Something isn't working labels Aug 14, 2020
@jbagga jbagga mentioned this issue Aug 24, 2020
Merged
@tracykard tracykard added this to the 2.3 milestone Oct 14, 2020
@tonybendis
Copy link
Contributor

TesTask object now includes FailureReason and CromwellResultCode properties. In CosmosDB, the actual values are serialized to Metadata dictionary inside the last item in TesTask.Logs[] array. Next step is for TriggerService to analyze the TesTasks related to a failed workflow and surface the root cause in the trigger file.

@jbagga
Copy link
Contributor Author

jbagga commented Nov 2, 2020

This issue has been addressed partly with PR #166
The other part is covered by issue #62

@jbagga jbagga closed this as completed Nov 2, 2020
Bug fixes and feature enhancements automation moved this from To do to Done Nov 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jbagga jbagga

Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
No open projects
Bug fixes and feature enhancements
  
Done
Milestone
2.3
Development

No branches or pull requests
3 participants
@tracykard @tonybendis @jbagga