New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry not working when a job fails in the export job queue #3295

Open
rowasc opened this Issue Sep 17, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@rowasc
Contributor

rowasc commented Sep 17, 2018

Describe the bug
When a job fails in the export jobs queue, it should retry the export chunk the number of times specified (3 in this scenario).
RIght now if an export chunk fails, it just fails and does not retry.

Where was the bug observed
amurt.ushahidi.io
rominacsvsms.ushahidi.io
Notes:

  • I have not seen this in sad-charlie, but it might happen if there's a failed chunk.
  • Note that this cannot be reproduced in production unless the timeout for exports is set lower than what we have it now (ie 8s). This should be reproduced locally or in QA to verify if it is some config being set incorrectly or a problem in the code.

To Reproduce
Steps to reproduce the behavior:

  1. Go to a deployment and start an export. Important: make sure the chunks will be considered FAILED by lowering the timeout in the lumen_export_queues repo to 1s.
  2. Check that when a chunk fails, it won't be retried

Internal slack thread: https://ushahidi.slack.com/archives/C03UU507R/p1537183988000100
Original issue for context on what was happening #3294

@rjmackay

This comment has been minimized.

Show comment
Hide comment
@rjmackay

rjmackay Sep 19, 2018

Member

If this proves hard to repro, would it be better to test a properly queued version of CSV exports (that works with multisite this time)?

Member

rjmackay commented Sep 19, 2018

If this proves hard to repro, would it be better to test a properly queued version of CSV exports (that works with multisite this time)?

@rowasc

This comment has been minimized.

Show comment
Hide comment
@rowasc

rowasc Sep 27, 2018

Contributor

Do you mean migrating the lumen_queues codebase to the platform codebase?

Contributor

rowasc commented Sep 27, 2018

Do you mean migrating the lumen_queues codebase to the platform codebase?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment