Skip to content

Failed batches processing with dead-letter queue #1713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 32 commits into from
Dec 22, 2020
Merged

Failed batches processing with dead-letter queue #1713

merged 32 commits into from
Dec 22, 2020

Conversation

lapaniku
Copy link
Contributor

@lapaniku lapaniku commented Dec 18, 2020 •
edited by vishalbollu
Loading

Closes #1540 #1504 #1541

checklist:

  • run make test and make lint
  • test manually (i.e. build/push all images, restart operator, and re-deploy APIs)
  • update docs and add any new files to summary.md (view in gitbook after merging)

lapaniku and others added 22 commits December 3, 2020 13:07
@lapaniku
Batch branch recovery
3c875e3
@lapaniku
DL test
763b61f
@lapaniku
Cortex conf
6234111
@lapaniku
Batch.py fix
043ff54
@lapaniku
Batch.py fix 2
8c388e2
@lapaniku
Revert batch.py
76e2955
@lapaniku
SQS region
8f322ff
@vishalbollu
Disable job clean up cron
f9d6a37
@vishalbollu
Set default queue visibility timeout to 60 seconds
307d547
@lapaniku
Batch.py renewal update
a766aed
@lapaniku
Merge branch 'failed_batches' of https://github.com/lapaniku/cortex i…
0ad5fab 
…nto failed_batches
@vishalbollu
Merge branch 'master' into failed_batches
cb4f057
@vishalbollu
Add message renewal and on job complete message retries
2d2a0bc
@lapaniku
Failed predict test
383d583
@lapaniku
Fix sqs client reference
0751891
@vishalbollu
Update batch.py
f9fb08f
@vishalbollu
Cast uuid to str
beb1844
@vishalbollu
Update batch.py
2b05878
@vishalbollu
Update batch.py
d5724ef
@vishalbollu
Add DLQ to job submission and retry batches on exception in Python
252f794
@vishalbollu
Merge branch 'master' into failed_batches
242b811
@vishalbollu
Clean up PR
69a01c3
@CLAassistant
Copy link

CLAassistant commented Dec 18, 2020 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@vishalbollu vishalbollu requested a review from deliahu December 21, 2020 19:36
deliahu
deliahu approved these changes Dec 21, 2020
View reviewed changes
@vishalbollu vishalbollu merged commit 202749a into cortexlabs:master Dec 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deliahu deliahu deliahu approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Retry batches on failure
4 participants
@lapaniku @CLAassistant @deliahu @vishalbollu