Make activated jobs which were not send to clients re-activatable

[deepthidevaki]

Description

Sometimes the client connection get closed (long polling timed out, network interruption, client died etc.) before gateway could send the activate job response back to the client. This results in the job being marked as activated in the broker, but never gets picked up by a client until the specified job timeout. This might be acceptable in some cases, but if the job time out is high clients might observe a huge latency between the time a job is created and completed.

This can happen at any time, but this is more frequent with long polling. (see #3585)

Proposal

When gateway realizes that response cannot be sent to the client, it can send a request to cancel activation to the broker.

---

related to SUPPORT-13198

[antoniodfr]

Hi, we are having errors due to this bug with Zeebe 0.26.1 in our production environment, do you know when will it be planned?

Thank you in advance.

[deepthidevaki]

This is currently in the backlog. However to prioritize it could you give use more context?

• How do you verify if it is really this issue?
• Is it occasionally an activated job is stuck, or does it happen frequently?
• How much impact do you have due to this?

[antoniodfr]

We identified it in this forum thread

It happens occasionally, but our business logic doesn't allow retries so we can't decrease the jobTimeout, and when the issue happens, we lose a customer in out e-commerce due to timeout.

[korthout]

@npepinpe I want to bring some additional attention to this issue. I regularly see community members run into some form of #5387, for which this can be a solution.

Having discussed it with @saig0, we think the idea is good, but an explicit JobIntent would be necessary to make it clear that this is not a user sending a FailJob.

[korthout]

I have given some thoughts to potential JobIntents:

• defer - seems like it would not be activatable for some time once its deferred
• recover - seems like it would also affect the number of retries or be able to recover from an error
• reject - seems like it would mean that this client would no longer want to be able to activate this job
• relinquish - seems like it would mean that this client would no longer want to be able to activate this job
• reset - seems like it would also affect the number of retries
• return - seems unclear whether it was completed, or not
• shelve - seems like it would not be activatable once it is suspended
• suspend - seems like it would not be activatable once it is suspended
• yield - IMO feels the most like the job is given back to the engine for re-activation, although it is similar to relinquish and reject in that it might have an expectation that this client won't want to be able to activate the job again, but IMO this feeling is less strong than with relinquish and reject

[npepinpe]

What is the expected impact of fixing this, in concrete terms? Also, I'm not sure how this relates to the call closed. I imagine it is for some calls, but do we know anything more about that?

[korthout]

I believe this is what users experience when they say that jobs sometimes get lost while delivering them to the client. I believe this would resolve that problem.

For example, this would already happen when a client sends an activate jobs request, the gateway has long polling enabled and the client's request times out before a new job was activated in the broker. Once the job is available and is activated, the client no longer has a connection with the gateway and the gateway can't deliver the job.

[npepinpe]

Fair point. I'll bring it up again during planning, but it looks like our engine team is pretty busy this quarter.

[jerry123888]

hi @npepinpe .
At present this problem has a great impact on us and has received many user complaints.
Do you have any plans to start? Is it convenient to tell me the detail start time ？Looking forward to your problem solving

[Young200808]

@npepinpe
any plan for fixing the issue for java client? it has big impact on our production enviroment.