Support Zeebe StreamActivatedJobs API

[jwulf]

SDK Component

Zeebe

The Zeebe Gateway has a new RPC StreamJobs to open a long-lived stream that essentially "pushes" jobs to the worker.

This obviates polling loops.

camunda/camunda#14152

Environment variable to enable it:

ZEEBE_CLIENT_WORKER_STREAMENABLED

See also the documentation about the general concept, including how back pressure works and can be implemented for your custom client: https://stage.docs.camunda.io/docs/next/components/concepts/job-workers/#job-streaming

[aleksander-dytko]

Hi @jwulf, do I assume correctly that this is targeted for 8.5 release?

[jwulf]

Hi @aleksander-dytko, it will probably soon come after the 8.5 release, asynchronously. There is a lot of work across the entire SDK for the 8.5 release.

[aleksander-dytko]

Thanks @jwulf. Do you know the rough timeline for this?
The customer raised a Feature Request here and I wanted to provide them with an update

[aleksander-dytko]

@jwulf @akeller do we know when this item will be delivered? I wanted to update the customer in the feature request

[jwulf]

This feature is under development, I estimate that it will be available at the end of May. 

In the meantime, would you be able to share the current worker creation parameters? There may be some tuning that can be done with the existing long polling that could mitigate the resource exhaustion. 

Some ideas that occur to me are to extend the long poll period, and also to add a randomised offset to it so that different workers that are started at the same time do not all poll at once.

[jwulf]

@aleksander-dytko an update on the status of this. This has been more challenging than I had anticipated. My revised time to deliver support for this feature in the JS SDK is the second week of June.

[jwulf]

At the moment, this feature fails in CI against both SM and SaaS. Example: https://github.com/camunda/camunda-8-js-sdk/actions/runs/9313149620

On my local machine, it fails or succeeds with no discernable pattern.

I am using Wireshark to see if I can debug it at the network level.

The stream is opened, but sometimes nothing comes over it locally. On CI, nothing comes over it.

CI is running Ubuntu. Locally, I am running MacOS. I am installing Ubuntu locally to see what happens there.

[jwulf]

@pepopowitz and I paired on this today. This appears to be an issue with the underlying JS gRPC library. We haven't been able to isolate the circumstances surrounding the failure to get it to stably reproduce.

Today, it worked for some time, and then started failing. I had thought that restarting the SM stack caused it to work for some time, but even that didn't work.

Here is a network packet capture from Wireshark for the failing call from the JS client:

The data packet that is being sent is an End Stream command.

With GRPC trace debugging turned on, this is what happens:

For some reason, the stream is terminated by the client immediately.

Contrast this with the network capture of the same request from the Java client (working, and in parallel to the JS client):

[jwulf]

OK, I got it working outside the test suite. There seems to be some interaction in the way the test suite mechanics are wired up.

[jwulf]

In the test, it needs a two second delay after starting the stream before starting a process instance.

[jwulf]

@aleksander-dytko support for the streaming jobs API is available in the latest packages: 8.5.5 and 8.6.0.

Documentation:

https://camunda.github.io/camunda-8-js-sdk/classes/index.Zeebe.ZeebeGrpcClient.html#streamJobs

Here is a potential "gotcha" using this feature, documented in the README:

Job Streaming

The Zeebe gRPC API supports streaming available jobs, rather than polling for them.

The ZeebeGrpcClient method StreamJobs allows you to use this API.

Please note that only jobs that become available after the stream is opened are pushed to the client. For jobs that were already activatable before the method is called, you need to use a polling worker.

In this release, this is not handled for you. You must both poll and stream jobs to make sure that you get jobs that were available before your application started as well as jobs that become available after your application starts.

In a subsequent release, the ZeebeWorker will transparently handle this for you.