Update the logic of openlineage.airflow.macros.lineage_parent_id

[blacklight]

The macro openlineage.airflow.macros.lineage_parent_id currently generates the parent_id as namespace/name/run_id, where name however is a UUID generated from the task metadata, and run_id is the concatenation of the task instance timestamp and try number.

This isn't compliant with the OpenLineage conventions. I'd expect name=<dag_id>.<task_id>, as it's a convention in other parts of the codebase, and run_id to be a UUID.

[boring-cyborg]

Thanks for creating your first OpenLineage issue! Your feedback is valuable and improves the project. If you haven't already, please be sure to follow the issue template!

[dolfinus]

May I propose is to just remove (or ar least deprecate) the lineage_parent_id macro, and instead use lineage_run_id with some new macro lineage_namespace. This allows passing values to child job like Spark session without parsing, e.g.:

SparkSubmitOperator(
  ...,
  config={
    "spark.openlineage.parentJobNamespace": "{{macros.OpenLineageProviderPlugin.namespace()}}",
    "spark.openlineage.parentJobName": "{{dag.dag_id}}.{{task.task_id}}",
    "spark.openlineage.parentRunId": "{{macros.OpenLineageProviderPlugin.lineage_run_id(task_instance)}}",
)

[kacpermuda]

There is no need to deprecate the macro, we can simply add two new macros - then users can choose to either pass all the information and parse it in spark or pass everything separately.

[dolfinus]

we can simply add two new macros

Done: #2582