How I can resolve openlineage provider's error ?

[DonghaLim]

Hello, This is dongha.
I have a question about using open lineage providers in airflow.
Although I added proper open lineage and it connected with marquez solution, below error is shown.
Additionally, I can't see the detail hireachy in marquez.

[2024-10-29, 02:09:59 UTC] {listener.py:66} DEBUG - OpenLineage listener got notification about task instance start
[2024-10-29, 02:09:59 UTC] {manager.py:150} DEBUG - extractor for PostgresOperator is <class 'airflow.providers.openlineage.extractors.base.DefaultExtractor'>
[2024-10-29, 02:09:59 UTC] {manager.py:90} DEBUG - Using extractor DefaultExtractor task_type=PostgresOperator airflow_dag_id=biz_energy_service_daily_0.0.1 task_id=sql_script_dim_energy_service_devices airflow_run_id=scheduled__2024-10-25T00:00:00+00:00
[2024-10-29, 02:09:59 UTC] {configuration.py:1050} WARNING - section/key [openlineage/disabled_for_operators] not found in config
[2024-10-29, 02:09:59 UTC] {manager.py:105} WARNING - Failed to extract metadata using found extractor <airflow.providers.openlineage.extractors.base.DefaultExtractor object at 0x7ff9b8c936d0> - section/key [openlineage/disabled_for_operators] not found in config task_type=PostgresOperator airflow_dag_id=biz_energy_service_daily_0.0.1 task_id=sql_script_dim_energy_service_devices airflow_run_id=scheduled__2024-10-25T00:00:00+00:00
[2024-10-29, 02:09:59 UTC] {configuration.py:1050} WARNING - section/key [openlineage/config_path] not found in config
[2024-10-29, 02:09:59 UTC] {utils.py:408} WARNING - section/key [openlineage/config_path] not found in config

Please let me know How I can debug this issue?

thanks.

[potiuk]

What happns when you followed instructions ? You followed them, did you ?

https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/structure.html

[DonghaLim]

I followed User guide :

• https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html

Detail Info:

• Airflow = 2.7.3
• apache-airflow-providers-openlineage = 1.4.0
• Parameter
  • OPENLINEAGE_URL : http://marquez:80
  • AIRFLOW__OPENLINEAGE__NAMESPACE : airflow

Error Log
[2024-10-29, 06:33:00 UTC] {init.py:74} DEBUG - Lineage called with inlets: [], outlets: []
[2024-10-29, 06:33:00 UTC] {taskinstance.py:844} DEBUG - Refreshing TaskInstance <TaskInstance: load_piper_daily.drop_partition_click_tv_smartthings_app scheduled__2024-10-25T00:00:00+00:00 [running]> from DB
[2024-10-29, 06:33:00 UTC] {taskinstance.py:1458} DEBUG - Clearing next_method and next_kwargs.
[2024-10-29, 06:33:00 UTC] {taskinstance.py:1400} INFO - Marking task as SUCCESS. dag_id=load_piper_daily, task_id=drop_partition_click_tv_smartthings_app, execution_date=20241025T000000, start_date=20241029T063251, end_date=20241029T063300
[2024-10-29, 06:33:00 UTC] {taskinstance.py:2430} DEBUG - Task Duration set to 9.217579
[2024-10-29, 06:33:00 UTC] {listener.py:113} DEBUG - OpenLineage listener got notification about task instance success
[2024-10-29, 06:33:00 UTC] {manager.py:150} DEBUG - extractor for RerunnableAthenaOperator is None
[2024-10-29, 06:33:00 UTC] {manager.py:109} DEBUG - Unable to find an extractor task_type=RerunnableAthenaOperator airflow_dag_id=load_piper_daily task_id=drop_partition_click_tv_smartthings_app airflow_run_id=scheduled__2024-10-25T00:00:00+00:00
[2024-10-29, 06:33:00 UTC] {configuration.py:1050} WARNING - section/key [openlineage/config_path] not found in config
[2024-10-29, 06:33:00 UTC] {utils.py:408} WARNING - section/key [openlineage/config_path] not found in config

[potiuk]

No idea. but apparently you miss some configuration settings - this is the warning about.

[DonghaLim]

It seems that inlets and outlets is missing. I'm using airflow in EKS environment.