Enable reliable submission scenarios using reference_id

[frankebersoll]

Summary

If we want to reliably submit events in order to prevent loss of any log messages, exceptions or whatever a user wants to submit, we probably need to

1. assign a Global Unique ID to each event.
2. filter out duplicate events on the server.

Do we need reliability?

As a developer, I probably won't care if sometimes an event that I didn't manually create somehow disappeared, e.g. an unhandled exception on a production client that I don't know of. After all, I don't expect the event to be there in the first place.
On the other hand, when I create events manually and some of them don't appear on the server, this will lead to confusion - I can't know for sure if there is a problem with Exceptionless or if my code is broken. This gets even more problematic if an event depends on another and one of them is lost. If this happens frequently, I will start questioning the quality of the service. Equally frustrating would be events that should have been sent only once but appear multiple times on the server, because the submission components messed it up.

Those reasons alone make me think that reliable submission shouldn't be opt-in, but default behavior. From a service provider perspective, we should even more care about getting all the events, because that means more events per month for that customer.

Can't the client figure it out?

There are situations where our request-response-pattern just doesn't work. We can't rely on having a useful response from the server if...

• there was a network problem (meaning the events have been submitted either successfully or partially or not at all).
• the application was killed while submitting or updating the persistant queue.
• there was a race condition (queue is already being processed, just when the app exits due to whatever reason and we try to force-submit any new events).

I tried to visualize that last point in the following diagram:

In the last step ("doesn't know what to send"), the queue can't just wait for the current submission operation to finish, because waiting would need to happen asynchronous - but asynchronous operations are not possible in the onExit handler, as the application will terminate right after the method execution ends.

Solutions

All those situations won't profit from client-side deduplication in any way. Either the client needs to ask the server if a message is already there until it gets a positive response, which would be chatty and doesn't feel right. Or the server must figure it out on its own.

In any case, events need an ID:

• We could just use the reference_id field that already seems to be supported by the server and API.
• We could drastically increase the information density of the ID by using ASCII word characters in upper- and lower-case.
• The server would check for the existence of the ID and cancel the pipeline early on collision, or just let ElasticSearch handle it.
• To maximize performance, it could cache recent reference ids in a dictionary or a Trie.

The common solution to this kind of problem is adding an idempotent PUT method to the API, where the client can choose the resource ID and put it multiple times, always getting the same results.

[ejsmith]

Hey Frank, thanks for the time and thought you have put into this!

I completely agree that we should strive to never lose an event. We have tried to balance performance and making it cheap to submit events and quickly put them into a queue for processing the server returns a 202 Accepted response and that is supposed to indicate that we have successfully received and queued the events. Obviously if the events aren't immediately saved, then we can't definitively say whether or not we successfully processed the events. From the client's perspective, once they get a 202 response then they should be able to assume that the events will be processed. The reference id is meant to be something that can be shown to an end user and then we can lookup that event in the system and see what error the user is getting. Also, the reference id is at an individual event level, but the client submits in batches, so don't think we can use reference id from the client side to ensure that events have been submitted successfully. The client currently should not remove events from the queue until it has successfully submitted them.

I'm not sure what more we can do at the client level. Events are submitted in batches, so we could have a batch id, but that doesn't really help because the next time we go to submit we might have a different batch of events that we are trying to submit. I think the best we can do is know that the server has successfully received and queued them. Not sure reference id helps much unless the client is going to go back and check later to see if each individual event exists in the system, but that would be a massive burden for clients to have to do that logic and also a huge amount of extra work for the system in general.

On the server side, we could fairly cheaply check to see if we already have a reference id which means it would be a duplicate and then cancel the event processing. If we just stored a key in redis to see if the event already exists then it would be a pretty cheap thing to do. We could add a distributed lock around the entire event pipeline based on the reference id and that should prevent duplicates.

It sounds like the core problem you are seeing is missing events? Are you seeing this happen frequently?

[frankebersoll]

Hey Eric,

thanks for your quick response!

From the client's perspective, once they get a 202 response then they should be able to assume that the events will be processed.

Yes, I think that's how it should work. This is the default case and works for any type of storage (in-memory or persisted): As soon as we get a 202 for a batch, we delete those messages from the queue.

What I wanted to talk about are edge cases, where we don't have a 202 or don't know if we have one, as in the examples I gave. This shouldn't happen that often, but it obviously can and will happen.

It sounds like the core problem you are seeing is missing events? Are you seeing this happen frequently?

Well, I don't use Exceptionless in production, yet. I'm integrating it in some private Node projects and found out it doesn't handle uncaught exceptions, and that's why I forked it. While implementing the synchronous submission for Node, I got to think about this problem (see sequence diagram above), and that's why I wanted to discuss it, as I think that doesn't only apply to JavaScript but to any clients in general.

Events are submitted in batches, so we could have a batch id, but that doesn't really help because the next time we go to submit we might have a different batch of events that we are trying to submit.

I also thought about this, but this doesn't seem to solve the problem. In case of the sequence diagram above, we would have two batches: One that contains events (A, B, C) currently being sent, and one that contains (A, B, C and D) where D is the uncaught exception, because we will never receive the 202 for (A, B, C) in time. The batch IDs won't help here.

I really think the only possibility is to make the client send the events as often as needed, until they get a 202. So we need a way to remove the duplicates on the server, and that's why every event needs an ID.

On the server side, we could fairly cheaply check to see if we already have a reference id which means it would be a duplicate and then cancel the event processing. If we just stored a key in redis to see if the event already exists then it would be a pretty cheap thing to do. We could add a distributed lock around the entire event pipeline based on the reference id and that should prevent duplicates.

Yes! This sounds great. This would make it easy for clients to support reliable submission:

• On application start, if you have any events in the queue, submit them. They may or may not have been submitted - just retry.
• On exit, we force-submit events. If !queue.isProcessing, no problem anyway, just process. If queue.isProcessing, we submit everything nevertheless - the server will sort out any duplicates.

What do you think?

[ejsmith]

One that contains events (A, B, C) currently being sent, and one that contains (A, B, C and D) where D is the uncaught exception, because we will never receive the 202 for (A, B, C) in time.

The queue is only allowed to process a single batch of events at a time. So you would submit 1 batch and that would succeed or fail and then another batch would be attempted.

Reading and thinking some more... I think you are saying that during exit we may already be processing the queue, but you need to synchronously process the queue right now and can't wait for the other process to finish. So you want to just be able to process anyway which may cause duplicates, but we would handle them on the server side.

I think the problem with this is that not all events have a reference id on them. So we would need to force them to all have one. Which isn't horrible, but seems like a lot of work for the normal use case that doesn't really need them.

Do you think it would be possible for us to wait for the current process to complete? I don't think that is possible in a sync way.

[frankebersoll]

Do you think it would be possible for us to wait for the current process to complete? I don't think that is possible in a sync way.

No, that's not possible. Yes, we would need an ID on every message.

I even see that in other client technologies. Don't the same issues apply to .NET, too? I haven't looked into the code, but I would think this should be done in AppDomain.DomainUnload. If we process the queue there, the situation would be the same.

Which isn't horrible, but seems like a lot of work for the normal use case that doesn't really need them.

I really can't think of a better solution at the moment.

[ejsmith]

Another thing to consider is that we are trying to make the clients as simple / dumb as possible. It should be incredibly easy to submit events with just curl. So we wouldn't be able to rely on reference ids being there, but I guess we could have a plugin that adds reference ids if they don't have one. That would prevent duplicates on the server for ones that do have them. I think checking the reference ids on the server side is pretty reasonable and would be fairly cheap. So I am good with adding that logic.

I am trying to take a step back and figure out what the core issue is here. It seems like the real problem is just wanting to make sure that you don't lose events when the app exits, is that correct? In the C# client we have persistent queue storage so even if the events don't get submitted on exit, they will still get processed the next time you run the app. Do you think that would help with what you are talking about? It would be nice either way for us to add a persistent queue implementation for node and also maybe use localstorage for the browser.

[frankebersoll]

It seems like the real problem is just wanting to make sure that you don't lose events when the app exits, is that correct?

Yes, exactly. During development or debugging, I really appreciate when exceptions show up immediately. During production, I can't really tell if the user will immediately restart the application when he's confronted with a critical error. Sending them when the application restarts should be a fallback in my opinion, after all it needs a persistent storage. I would gladly help with that, too - but to me, it was crucial to make submission on-exit work.

Another thing to consider is that we are trying to make the clients as simple / dumb as possible.

I like that.

I think checking the reference ids on the server side is pretty reasonable and would be fairly cheap. So I am good with adding that logic.

What would we get?

• Dumb clients can stay dumb.
• Sophisticated clients can add reference IDs and retry sending those events, when in doubt.
• Indexing on reference IDs doesn't sound bad.

Then we could add local storage and get:

• Offline support.
• Fallback when on-exit doesn't work.

[ejsmith]

Sending them when the application restarts should be a fallback in my opinion

Agreed.

Sounds like a plan!

[ejsmith]

Want me to add the reference id duplicate checking on the server side or do you want to dig into that side of things as well?

[niemyjski]

@ejsmith @frankebersoll I don't really feel like we should be running an elastic search query on this... I almost think that we should just add a boolean cache item with the key "projectid:referenceid" with an expiration of an hour or a week. and if the key exists we cancel the submission of this context item otherwise we let it continue.. Thoughts?

[niemyjski]

This still would allow for duplicates out of this time period.

[ejsmith]

Yes, it would be a redis cache key.

[niemyjski]

Thing that sucks if the pipeline fails, the key will be set, unless we set the key with a 5 second ttl and then in post processing we set the key for the real period that way it could be retried :)

[ejsmith]

It should use a lock... not just a key. That is what the locks are for.