Dynamic task mapping compatibility with multi-scheduler

[tnyz]

Apache Airflow version

Other Airflow 2 version

What happened

We noticed that tasks using the dynamic task mapping api crashes the scheduler with the error

psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "task_instance_pkey"
80
DETAIL:  Key (dag_id, task_id, run_id, map_index)=(ingestion-scheduler, check_for_success_flag, manual__2022-10-03T15:56:03.241451+00:00, 0) already exists.

We have 2 schedulers so my guess is that both try to schedule the dynamically mapped task

What you think should happen instead

it should be not crash the scheduler

How to reproduce

run a large dynamically mapped task in multi-scheduler environment

Operating System

linux

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

I seriously doubt this is the case - this is an extremely wild and unfounded guess. It's likely an error in your DAG, but I think we need more informationand reproducible case showing exactly what causes it to be sure.

I suggest you do more experimentation with it and trying to provide more information, logs, circumstances, also experimenting with decreasing the number of schedulers to 1 and running it for a while - if you can make a fully reproducible case that can help with diagnosing and result of your analysis, it would be much better.

Converting it into discussion. I am also not going to respond as I am going for 2 weeks vacation, but I hope others will.

[stijndehaes]

We noticed a similar issue, however we are not running multi-scheduler. We are only running one at the same time (however because kubernetes deploys in some cases we might have several schedulers.

The issue just randomly happened, at some point the scheduler started crashing. There was a dynamic task which had a taskinstance with map_index -1, and the task instances with map_index 1 through 43 also already existed.
The scheduler was trying to replace the -1 one with the ones that should be created but always crashed because they already existed.

I'll be trying to reproduce this next week, but it might be our scheduler got killed at an unfurtunate time. Some help on where to look in the code, or some ideas on how to reproduce would be appreciated :)

[stijndehaes]

@potiuk we had this happen 2 extra times over the weekend. I am not sure how this can happen looking into the code, everything should be run nicely into a transaction but what we see is the following:

• There is a task with a -1 index
• There are tasks with index 0 to 43

The scheduler keeps on crashing trying to change the -1 index to 0 index. However that record already exists.

I can think of the following fix. When there is a -1 and a 0 index delete the -1 index instead of changing the number. WDYT @potiuk ?

As to how this might happen I have no clue yet.

[potiuk]

No idea. But since it happens often enough for you, I suggest that you upgrade to Airflow 2.5.0 and check if it is fixed there.

You have not explained in detail what is your Airflow version, but It looks like it's not the latest one. With every release there are many issues fixed, so the fastest way to check if this kind of bug is fixed (when it is not really easy to track where it gets from) is just to upgrade and see if it still happens. There were quite a number of fixes to task mapping in 2.5.0 and it is very likely one of those fixed it.

Those kind of errors will anyhow very likely require an upgrade, so you do not loose anything when you upgrade (and even more - you make your company less vulnerable to security issues found in older versions. I strongly recommend you first upgrade and see if it continues to happen. If it does., we will look further (some logs/circumstance description should be provided).

[stijndehaes]

@potiuk Thank you for the suggestion, in the mean time I have created this PR: #27834
We are currently on a patched version of 2.3.4 where I included the changes in the above PR. So for us the issues are fixed.

This week we will upgrade to 2.4.x, I'll let you know if anything chages. However with the changes from the PR we are likely to see no issues again

[potiuk]

Ah I see. Interesting. Yeah. This was an interesting one.

[potiuk]

Let us know if it works, that woudl be great to see the confirmation.