Add map_index to RenderedTaskInstanceFields

[ashb]

In order to do this "properly" we have also migrated it from using execution_date to run_id in its PK

[ashb]

@uranusjr I'm thinking we should add a mapped_value JSON column here in this migration. When does the mapped value become available?

[uranusjr]

I'm thinking we should add a mapped_value JSON column here in this migration. When does the mapped value become available?

They are populated when XCom is resolved, and should be available when RenderedTaskInstanceFields are written. However, since a RenderedTaskInstanceFields row maps to a TaskInstance row, I think we should simply resolve the mapped values directly instead.

On UI we should also add a marker to indicate a particular field value is mapped, so perhaps add a field to record the field names. Or maybe even this is unnecessary since this information is available on the task object? Or maybe in some scenarios we want to read RenderedTaskInstanceFields without parsing the DAG? (Currently I can only find this being used in /rendered-k8s and /rendered-templates, and in both views we already fetch the original task object and know what fields are mapped anyway.)

[uranusjr]

Assuming we don’t need to add mapping info to RenderedTaskInstanceFields (I don’t think we do), this should be good to go.

[ashb]

Or maybe in some scenarios we want to read RenderedTaskInstanceFields without parsing the DAG?

The use case i am thinking of is for showing a table of all the mapped TIs in a detail view in the UI. And mapping can apply to things that aren't marked as templated

[github-actions]

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

[ashb]

Assuming we don’t need to add mapping info to RenderedTaskInstanceFields (I don’t think we do), this should be good to go.

Even if we do we can alter this migration in a future PR.

[uranusjr]

sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1025, "Error on rename of './airflow/#sql-1_b' to './airflow/rendered_task_instance_fields' (errno: 150 - Foreign key constraint is incorrectly formed)")

MySQL is not happy…

[ashb]

Okay, I'm confused -- the mysql tests are not behaving like this for me locally Oh downgrade.