Flows cron triggers multiple times

[stoberbryan]

Describe the Bug

Flows Trigger 3 times no matter what.
Created New Flow to validate:
Only the Create Data functionality causes it.
Emits Events On and Off Does Not Affect it

To Reproduce

1. Create Flow
2. Create Data Step
3. Email Yourself

You will get 3 emails and items

Errors Shown

No response

What version of Directus are you using?

9.15.1

What version of Node.js are you using?

16.16.0

What database are you using?

cloud pg

What browser are you using?

chrome

How are you deploying Directus?

cloud

[stoberbryan]

I forgot to add that it starts with a cron step
if i skip create data, i get 1 email

[rijkvanzanten]

This is a known limitation of the cron setup currently, as multiple instances of directus aren't aware of each other, so each instance of directus will execute the cron job simultaneously.

I'll leave this open as a bug, as it's already a thing we're actively working on 👍🏻

[stoberbryan]

Thank you!

[azrikahar]

Ref #13381 which is the primary discussion on solutions for this known limitation.

[rijkvanzanten]

We should generate a redis key for flow ID + the timestamp the cron is supposed to run (not exact current timestamp) with a TTL of like 3 seconds. We can run a redis increment call on that key, if it returns 1, the current instance is the one to run the flow, otherwise, ignore it.

[rijkvanzanten]

Linear: ENG-209

[colemahatmccreary]

Is there any update here?

I'm running into race conditions in my app that I can't seem to get around without solving this.

Thanks!

[adrs0u]

Hello! directus support refers us here! The problem also (still) occurs in the directus Cloud version.
The cron is triggered twice.

It would be good if this works, otherwise you can not use cron scheduling
Thank you!

[macntech]

We are also highly interested into an update of the fix. We running into the same issue on a scaled system. Thanks!

[metalmarco]

We have the same issue on a single instance installation (no Directus Cloud)

[metaaifactory]

Same here :/

Do you have any news on a fix ?

[BrianJM]

Is there a workaround?

Is it possible to isolate cron tasks to a single instance? For example,

• Single instance for cron tasks and flows
• HA cluster allocated for UI

[nickrum]

We had a little discussion about the best fix for this issue and the solution we came up with is to use a "Synchronized Clock" mechanism via redis.

The way this mechanism should work is as follows:

The supposed next execution time of each flow with a schedule trigger is stored in redis with the UUID (and some prefix like directus:clock:) as the key and the execution time as a timestamp with seconds-precision as the value. We only need seconds-precision because cron only supports intervals down to seconds. We can compute the supposed next execution time from the cron instruction and the current time because each cron instruction clearly specifies at which exact second it should be executed next.
Whenever a schedule flow is triggered on one Directus instance, it checks if the next supposed execution time based on the current time is greater than the value stored in redis. If it is, then the value in redis is updated with the value computed by the Directus instance and the instance executes the flow. If it is not, then the value in redis is neither updated, nor is the flow executed by the instance. It is important that this "check and update if greater than"-operation is performed atomically by redis. This ensures that there are no race conditions where one instance determines that its execution time is greater than the value in redis and another instance also determines that its execution time is greater than the value in redis before the first instance could update the value.

This mechanism ensures that, provided the internal clocks of all instances do not deviate by more than one second, for each supposed execution time of a schedule flow, the flow is executed only by the instance that accesses the value in redis first.
This mechanism doesn't take the case into consideration that a Directus instance updates the execution time in redis and crashes before it can finish executing the flow. This is a tradeof we accept because Directus can not know whether it is save to re-execute a flow that has been executed up until a certain point and there is currently no way for the user to specify this.

[u12206050]

@rijkvanzanten the current cache (Keyv) doesn't support atomic updates correct? So are you switching it out or communicating with redis more directly.

Also, I wonder if having to need redis for this to work is viable, vs just having a directus_queue table in the database like laravel has?

[rijkvanzanten]

@rijkvanzanten the current cache (Keyv) doesn't support atomic updates correct? So are you switching it out or communicating with redis more directly.

Keyv just uses Redis under the hood. Redis is single-threaded, so every update is atomic by default 🙂

Also, I wonder if having to need redis for this to work is viable, vs just having a directus_queue table in the database like laravel has?

Writes to a database aren't reliably atomic or timing-error-free like Redis would be for this use case, so I'm hesitant to go that route. Besides, we're already requiring Redis when you're in a horizontally scaling environment as we're relying on it's pubsub for websockets among other things 👍🏻

[BrianJM]

Also, I wonder if having to need redis for this to work is viable, vs just having a directus_queue table in the database like laravel has?

Writes to a database aren't reliably atomic or timing-error-free like Redis would be for this use case, so I'm hesitant to go that route. Besides, we're already requiring Redis when you're in a horizontally scaling environment as we're relying on it's pubsub for websockets among other things 👍🏻

👍 for redis.

Using relational tables will eventually result in table bloat, especially under heavy use, which will need to be monitored and managed; even so, tables may need to be rotated occasionally.