Replies: 2 comments 1 reply
-
Hi The data can be sent twice in regular subscription as well, see: https://ravendb.net/docs/article-page/5.3/csharp/client-api/data-subscriptions/what-are-data-subscriptions#documents-processing Regarding the two questions you listed,
|
Beta Was this translation helpful? Give feedback.
-
@georgiosd your questions about the two cases & disconnections are making assumptions that I don't believe to be correct. Consider the following:
Note that at T4 - we are done with the batch, but we didn't report it back. The situation is basically the same. You are correct that this applies to both concurrent and non concurrent subscriptions. If I understand this properly, you have a device that has the following functions:
And you are saving the |
Beta Was this translation helpful? Give feedback.
-
Reading the docs on the new concurrent subscriptions it says:
If I understand this statement correctly, it concerns a disconnection after the batch is sent to the worker. It’s not clear however if the worker will process the documents even if the connection is lost before ack.
So I guess we have two cases:
While this is in the concurrent subscriptions discussion, I presume these concerns are also valid for non-concurrent subscriptions.
Which begs the bigger question about achieving exactly-once processing with RavenDB subscriptions, concurrent or otherwise.
Speaking to @DejanMilicic about this, he sent two good articles (1, 2) about the problem in general, which highlight why the problem is hard to solve and how Kafka takes a 20% performance hit to guarantee order + exactly once semantics and still requires you to store the index position to achieve it.
I wanted to share my approach on a particular subscription consumer I wrote where it's very important to process messages exactly once and get help from the community to pick holes or suggest other alternatives/success stores :)
The use case is for a network device that digitally signs financial transactions and transmits them to the Greek tax authority. Basically, every time someone buys something, it produces a sort of digital receipt that can be validated, and is required in case of a tax audit, to display that you've registered the transaction and will pay the appropriate VAT for it.
So, I think it's clear why we must have exactly-once here; if someone pays 10 EUR at 24% VAT and we generate two signatures, they will have to declare 20 EUR income and pay 4.80 EUR to be valid with the authorities. 🗡️
While the tax device itself it completely basic and doesn't even try to help you with this problem, it does provide two sequence numbers: a batch id (these batches are rolled over every 24 hours or more if no transactions) and sequence id (device-specific). I also have to generate a unique and incremental sequence id that is organization-specific.
The way I went about it is:
a. Produce the organization-specific sequence id using
NextIdentity
as soon as thesign
message is processed by RavenDB, and save it immediately. Order is not important so even if it fails later, I don't have to generate this number again.b. Store the batch + device sequence ids in the device document. After generating the organization sequence id (a above), I query the device for these numbers and if there is a mismatch, I throw an exception and notify for manual intervention.
c. Continue processing and issue the new signature. Finally update the stored sequence numbers and store the signature in one transaction.
The only things I haven't figured out so far are:
How to make sure that a device (there are several of them for different clients) is not used by another worker, since the device cannot accept concurrent connections. I could just let it fail on the network level and retry, but I was also thinking I could create one subscription per device. The least preferred option would be to use a cmp xchg value to "lock" it.
Whether I need to shield myself somehow from the scenario that the SAME command will be processed by TWO workers at the SAME time.
Looking forward to hearing from the community!
Beta Was this translation helpful? Give feedback.
All reactions