Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make 404 an optionally recoverable error when calling rest api endpoint #2072

Closed
benedeki opened this issue May 24, 2022 · 1 comment · Fixed by #2087
Closed

Make 404 an optionally recoverable error when calling rest api endpoint #2072

benedeki opened this issue May 24, 2022 · 1 comment · Fixed by #2087
Assignees
@lsulak
Labels
Conformance Conformance Job affected feature New feature priority: medium Important but not urgent Standardization Standardization Job affected under discussion Requires consideration before a decision is made whether/how to implement
Milestone
Features for 3.0....

Comments

@benedeki
Copy link
Collaborator

Background

When call from Spark Job to Menas REST API fails, if the error is considered recoverable another of the configured servers is tried.

Feature

404 is not considered recoverable. Make it configurable that 404 can be considered recoverable to try a different server.

Bonus [Optional]

Maybe it could be done generally, that a list of error status codes can be configured, that will be considered recoverable.
@benedeki benedeki added Conformance Conformance Job affected feature New feature priority: medium Important but not urgent Standardization Standardization Job affected under discussion Requires consideration before a decision is made whether/how to implement labels May 24, 2022
lsulak added a commit that referenced this issue Jun 16, 2022
@lsulak
#2072 adding retryableException parameter into CrossHostApiCaller and…
95c61ab 
… changing DaoException not to be retryable by default
lsulak added a commit that referenced this issue Jun 16, 2022
@lsulak
#2072 adding unit tests for retryableException functionality
f7bcaec
lsulak added a commit that referenced this issue Jun 16, 2022
@lsulak
#2072 adding retryableException parameter spark-jobs
22b3b7d
lsulak added a commit that referenced this issue Jun 16, 2022
@lsulak
#2072 adding retryableExceptions into reference configuration of the …
aebf732 
…spark-jobs
lsulak added a commit that referenced this issue Jun 16, 2022
@lsulak
#2072 zip only iterates over the shorter Set, so we must check the size
d862908
lsulak added a commit that referenced this issue Jun 16, 2022
@lsulak
#2072 improving/fixing potentially duplicated items in the input retr…
848c756 
…yableExceptions string
lsulak added a commit that referenced this issue Jun 16, 2022
@lsulak
#2072 adding type to an empty Set for unit test (it should work even …
e255e5c 
…without it, we compare the content of the sets in our tests)
lsulak added a commit that referenced this issue Jun 17, 2022
@lsulak
#2072 renaming CustomException to MenasException for the abstract cus…
a32913c 
…tom exception that's being used for the DAO module and removing var canBeRecovered
lsulak added a commit that referenced this issue Jun 17, 2022
@lsulak
#2072 adding AutoRecoverableException for ResourceAccessException | R…
364f03a 
…estClientException and fixing matching
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 changing optionally retryable load process according to code re…
55bab24 
…views
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 only 404 as optionally retryable in the config file for now
fa68337
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 reflecting changes related to custom exceptions in DAO module
624577e
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 fixing long line
3d179b4
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 adding optionallyRetryableExceptions param into examples module
35765f3
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 propagating change related to custom exception in DAO module in…
739f38c 
…to spark-jobs module
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 removing unnecessary module, this was replaced by our config utils
3b7aff1
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 reflecting changes related to custom exceptions in DAO module
3ffa5d6
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 fixing long line
c338120
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 adding optionallyRetryableExceptions param into examples module
9d4acba
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 propagating change related to custom exception in DAO module in…
c540217 
…to spark-jobs module
lsulak added a commit that referenced this issue Jun 27, 2022
@lsulak
#2072 removing unnecessary module, this was replaced by our config utils
6d01c05
lsulak added a commit that referenced this issue Jun 28, 2022
@lsulak
#2072 adding optionallyRetryableExceptions into conformance and stand…
d407b89 
…ardization code
lsulak added a commit that referenced this issue Jun 28, 2022
@lsulak
#2072 removing unnecessary scalastyle check
19ec54d
lsulak added a commit that referenced this issue Jul 18, 2022
@lsulak
#2072 refactoring Menas Exceptions - now categorizing them into 3 gro…
4601622 
…ups: Not Retryable, Retryable, and Optionally Retryable errors; this was basically implemented already, but not clearly expressed
lsulak added a commit that referenced this issue Jul 18, 2022
@lsulak
#2072 post-review changes, splitting behavior of NotFound exception i…
7af2c48 
…nto two individual ones for clarity
lsulak added a commit that referenced this issue Jul 18, 2022
@lsulak
#2072 changing unit tests so that they reflect recent changes
9352298
lsulak added a commit that referenced this issue Jul 18, 2022
@lsulak
#2072 minor typo
f03dca1
lsulak added a commit that referenced this issue Jul 18, 2022
@lsulak
#2072 making 'optionally retryable exceptions' not required for Hyper…
2a31585 
…conformance's config (as recently is implemented in HyperConformanceAttributes)
lsulak added a commit that referenced this issue Jul 18, 2022
@lsulak
#2072 bullet-proofing the config loading process for menas.rest.optio…
1790c48 
…nallyRetryableExceptions
lsulak added a commit that referenced this issue Jul 18, 2022
@lsulak
#2072 post-review changes
939380c
lsulak added a commit that referenced this issue Jul 18, 2022
@lsulak
#2072 human mistake
dbe7169
@benedeki benedeki linked a pull request Jul 19, 2022 that will close this issue
#2072 make 404 recoverable error #2087
Merged
lsulak added a commit that referenced this issue Jul 25, 2022
@lsulak
#2072 post-review changes
6a7b5b9
lsulak added a commit that referenced this issue Aug 24, 2022
@lsulak
#2072 enhancing getIntList with getListFromString, based on an older …
c82fe02 
…commit from the repo
lsulak added a commit that referenced this issue Aug 24, 2022
@lsulak
#2072 post-review comments, improving error logs
9fb9c14
lsulak added a commit that referenced this issue Aug 24, 2022
@lsulak
#2072 post-review comments, authentization mechanism can fail due to …
29293b7 
…other problems (e.g 404) than just authentization
lsulak added a commit that referenced this issue Aug 24, 2022
@lsulak
#2072 removing dead code
d31b901
lsulak added a commit that referenced this issue Aug 24, 2022
@lsulak
#2072 improving unit tests
80cbe01
lsulak added a commit that referenced this issue Aug 25, 2022
@lsulak
#2072 renaming MenasException -> RestAPIException, since it's much mo…
bf9e613 
…re precise in terms of its responsibilities
lsulak added a commit that referenced this issue Aug 25, 2022
@lsulak
#2072 improving err message
d78c00d
lsulak added a commit that referenced this issue Aug 25, 2022
@lsulak
#2072 improving param name
e4ac41b
lsulak added a commit that referenced this issue Aug 25, 2022
@lsulak
#2072 improving function name
cb4def0
lsulak added a commit that referenced this issue Aug 26, 2022
@lsulak
#2072 post-review changes
a0f21bd
lsulak added a commit that referenced this issue Aug 26, 2022
@lsulak
#2072 obeying our scala overlords
0b2538a
lsulak added a commit that referenced this issue Aug 26, 2022
@lsulak
#2072 minor improvement
e13ca59
lsulak added a commit that referenced this issue Aug 28, 2022
@lsulak
#2072 post-review changes
efd65d7
lsulak added a commit that referenced this issue Aug 28, 2022
@lsulak
#2072 making naming consistent throughout the repository
d50ed1e
lsulak added a commit that referenced this issue Aug 28, 2022
@lsulak
#2072 making naming consistent with PR #2111
fd8fb81
@lsulak
Copy link
Collaborator

lsulak commented Sep 2, 2022

Release notes

A user can specify a list of HTTP error status codes that are allowed to be retried when a REST API call yields such an HTTP error. Currently supported HTTP error status codes: 401, 403, and 404.

lsulak added a commit that referenced this issue Sep 2, 2022
@lsulak @benedeki
#2072 make 404 recoverable error (#2087)
efb4d73 
* #2072 adding retryableException parameter into CrossHostApiCaller, AuthClient, and reference.conf config file
* #2072 enhancing config loading functionality providing support for the retryableException parameter
* #2072 refactoring DAOException module
* #2072 unit tests confirming the correct functionality of this feature



Co-authored-by: David Benedeki <14905969+benedeki@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lsulak lsulak

Labels
Conformance Conformance Job affected feature New feature priority: medium Important but not urgent Standardization Standardization Job affected under discussion Requires consideration before a decision is made whether/how to implement
Projects
None yet
Milestone
Features for 3.0.0 (next major version)
Development

Successfully merging a pull request may close this issue.

#2072 make 404 recoverable error AbsaOSS/enceladus
3 participants
@lsulak @AdrianOlosutean @benedeki