Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save message to S3 before DeleteMessage #39

Closed
k0kubun opened this issue Jul 14, 2017 · 2 comments
Closed

Save message to S3 before DeleteMessage #39

k0kubun opened this issue Jul 14, 2017 · 2 comments

Comments

@k0kubun
Copy link
Collaborator

k0kubun commented Jul 14, 2017

This TODO should be fixed before v1. Delaying DeleteMessage or doing PutObject earlier is needed.

@eagletmt
Copy link
Member

Yeah, I think the best solution for this is saving the message body just after enqueueing.
SQS is good for queueing and long-polling but not good for storing data. Any data should be always stored to database or S3 in my opinion.

queue = Barbeque::JobQueue.find_by!(name: @queue)
message = build_message
response = Barbeque::MessageEnqueuingService.sqs_client.send_message(
  queue_url:    queue.queue_url,
  message_body: message.to_json,
)
Barbeque::ExecutionLog.save_message(response.message_id, message)
response.message_id

We can determine the S3 key for message.json before inserting JobExecution record since the S3 key consists of application name, job name and message_id which are available in the message body.

@eagletmt
Copy link
Member

My idea above breaks SNS integration because SNS integration doesn't go through /v2/job_executions or MessageEnqueueingService. Thus we cannot add extra process before ReceiveMessage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants