New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enqueueing (recovery aware) publishing #184

Open
michaelklishin opened this Issue Jan 23, 2014 · 13 comments

Comments

Projects
None yet
5 participants
@michaelklishin
Member

michaelklishin commented Jan 23, 2014

Currently Bunny::Exchange#publish doesn't try to ensure the message is sent when network connection is down. We should investigate enqueueing messages locally
(possibly with a write-ahead log) and sending them out only when it's safe to do so.
Batching can be an option, too.

This is not a feature everybody needs, and it does not replace publisher confirms but it
can make it easier for people who cannot afford to lose a single message to build
publishers where message loss probability is significantly lower.

Some open questions:

  • If we choose to implement a WAL, what on-disk format should we use?
  • How do we not degrade throughput by a significant amount?
  • How do we synchronize recovery and this publisher? Will it be running in a separate thread?
  • Is an in-memory version practical/interesting? Apps are increasingly deployed into environments (e.g. PaaS) where local disk usage is limited, discouraged or even not an option. Plus memory is a finite resource: do we reject publishes when the buffer is full, keep the most recent N messages or do something else?
@trungpham

This comment has been minimized.

Show comment
Hide comment
@trungpham

trungpham Sep 17, 2014

I don't think we should write to a file on disk, otherwise people who deploy their app on Heroku won't be able to use it. Should make it write to the database or some remote location.

trungpham commented Sep 17, 2014

I don't think we should write to a file on disk, otherwise people who deploy their app on Heroku won't be able to use it. Should make it write to the database or some remote location.

@michaelklishin

This comment has been minimized.

Show comment
Hide comment
@michaelklishin

michaelklishin Sep 18, 2014

Member

@trungpham WAL will be optional. Writing to a database largely has the same issues as not having a WAL in the first place.

Member

michaelklishin commented Sep 18, 2014

@trungpham WAL will be optional. Writing to a database largely has the same issues as not having a WAL in the first place.

@carlhoerberg

This comment has been minimized.

Show comment
Hide comment
@carlhoerberg

carlhoerberg Sep 19, 2014

Contributor

Could still be helpful on Heroku, if it's only an intermediate network issue, and it's able to reconnect before the dyno is restarted, those messages on disk could be republished when the connection is established again.

On Thursday 18 September 2014 at 05:21, Michael Klishin wrote:

@trungpham (https://github.com/trungpham) WAL will be optional. Writing to a database largely has the same issues as not having a WAL in the first place.


Reply to this email directly or view it on GitHub (#184 (comment)).

Contributor

carlhoerberg commented Sep 19, 2014

Could still be helpful on Heroku, if it's only an intermediate network issue, and it's able to reconnect before the dyno is restarted, those messages on disk could be republished when the connection is established again.

On Thursday 18 September 2014 at 05:21, Michael Klishin wrote:

@trungpham (https://github.com/trungpham) WAL will be optional. Writing to a database largely has the same issues as not having a WAL in the first place.


Reply to this email directly or view it on GitHub (#184 (comment)).

@michaelklishin

This comment has been minimized.

Show comment
Hide comment
@michaelklishin

michaelklishin Sep 19, 2014

Member

The problem is that having database adapters is way out of scope for Bunny. A common interface for WAL stores is the only way to go, and with it you can do whatever you want with it.

Member

michaelklishin commented Sep 19, 2014

The problem is that having database adapters is way out of scope for Bunny. A common interface for WAL stores is the only way to go, and with it you can do whatever you want with it.

@carlhoerberg

This comment has been minimized.

Show comment
Hide comment
@carlhoerberg

carlhoerberg Sep 19, 2014

Contributor

To clarify, i did not advocate database persistence, i meant on-disk persistence, even on Heroku, which is totally possible.

Contributor

carlhoerberg commented Sep 19, 2014

To clarify, i did not advocate database persistence, i meant on-disk persistence, even on Heroku, which is totally possible.

@michaelklishin

This comment has been minimized.

Show comment
Hide comment
@michaelklishin

michaelklishin Sep 19, 2014

Member

Ah, interesting. Yeah, I had no idea. Does Heroku documentation cover this?

Member

michaelklishin commented Sep 19, 2014

Ah, interesting. Yeah, I had no idea. Does Heroku documentation cover this?

@trungpham

This comment has been minimized.

Show comment
Hide comment
@trungpham

trungpham Sep 19, 2014

@michaelklishin I vote for a common interface too. That will give us the flexibility of writing to the log file or other data store.

trungpham commented Sep 19, 2014

@michaelklishin I vote for a common interface too. That will give us the flexibility of writing to the log file or other data store.

@carlhoerberg

This comment has been minimized.

Show comment
Hide comment
@JanStevens

This comment has been minimized.

Show comment
Hide comment
@JanStevens

JanStevens Jul 12, 2016

Any update on this issue? We experience sometimes connection issues in our pushing applications (Rails + Unicorn), this means that not all messages are pushed and there is no retry mechanism in place (so the message is lost, gone forever). Really a shame since this prevents us from really using Bunny all over the place.

A start would be to write messages in a file whenver a Timeout/Bunny Connection error happens. If the connection is back up we can reply the file.

JanStevens commented Jul 12, 2016

Any update on this issue? We experience sometimes connection issues in our pushing applications (Rails + Unicorn), this means that not all messages are pushed and there is no retry mechanism in place (so the message is lost, gone forever). Really a shame since this prevents us from really using Bunny all over the place.

A start would be to write messages in a file whenver a Timeout/Bunny Connection error happens. If the connection is back up we can reply the file.

@michaelklishin

This comment has been minimized.

Show comment
Hide comment
@michaelklishin

michaelklishin Jul 12, 2016

Member

@JanStevens no updates. We are working on something very similar in the Objective-C client first, will see how well it works there. That doesn't cover a WAL (writing to durable storage).

Member

michaelklishin commented Jul 12, 2016

@JanStevens no updates. We are working on something very similar in the Objective-C client first, will see how well it works there. That doesn't cover a WAL (writing to durable storage).

@rthouvenin

This comment has been minimized.

Show comment
Hide comment
@rthouvenin

rthouvenin Dec 14, 2017

We also have this problem. If for some reason the connection to RabbitMQ is lost, all messages sent while the connection is down are lost forever. Any update on this topic?

From what I understand using publisher confirms would solve this, right? Is it the recommended way to approach this problem?

rthouvenin commented Dec 14, 2017

We also have this problem. If for some reason the connection to RabbitMQ is lost, all messages sent while the connection is down are lost forever. Any update on this topic?

From what I understand using publisher confirms would solve this, right? Is it the recommended way to approach this problem?

@michaelklishin

This comment has been minimized.

Show comment
Hide comment
@michaelklishin

michaelklishin Dec 14, 2017

Member

@rthouvenin there are no updates. This is not a support forum, please start with the docs.

Member

michaelklishin commented Dec 14, 2017

@rthouvenin there are no updates. This is not a support forum, please start with the docs.

@ruby-amqp ruby-amqp locked and limited conversation to collaborators Dec 14, 2017

@michaelklishin

This comment has been minimized.

Show comment
Hide comment
@michaelklishin

michaelklishin Dec 14, 2017

Member

Publisher confirms per se do not do what's covered in this issue but they allow the developer to keep track of what messages need re-publishing.

Member

michaelklishin commented Dec 14, 2017

Publisher confirms per se do not do what's covered in this issue but they allow the developer to keep track of what messages need re-publishing.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.