Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context #20603

gbonazzoli · 2021-12-31T05:53:22Z

Apache Airflow version

2.2.3 (latest released)

What happened

This strange behaviour started with Airflow 2.2.3, Airflow 2.2.2 was OK.

In every DAG that has a PythonOperator declared as:

def execute_Get_VMware_Host_Running_The_VM(ds, **kwargs):

    try:
        __VM_NAME__ = kwargs['dag_run'].conf['VM_NAME']
        logging.info(f"Remotely received value of {__VM_NAME__} for key=VM_NAME")
    except KeyError:
        errorMessage = 'This DAG must be RUN with VM_NAME parameter, es: {"VM_NAME":"lxd-p2c-server"}'
        raise KeyError(errorMessage)

with DAG('VMWARE_BACKUP',
          description = 'VMWARE_BACKUP',
          tags=['system'],
          schedule_interval = None,
          start_date = datetime(2019, 5, 29, tzinfo=local_tz),
          default_args = default_args,
          max_active_runs = 1,   # maximum number of active runs for this DAG
          concurrency = 1,       # The Airflow scheduler will run no more than concurrency task instances for your DAG at any given time. number of active runs for this DAG
          catchup = False
    ) as dag:

    Get_VMware_Host_Running_The_VM = PythonOperator(
        task_id = 'Get_VMware_Host_Running_The_VM',
        python_callable = execute_Get_VMware_Host_Running_The_VM,
        dag = dag
    )

we have the following WARNINGS in the log:

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_start' or 'logical_date' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'next_ds' from the template is deprecated and will be removed in a future version. Please use '{{ data_interval_end | ds }}' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'next_ds_nodash' from the template is deprecated and will be removed in a future version. Please use '{{ data_interval_end | ds_nodash }}' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'next_execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_end' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_ds' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_ds_nodash' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_execution_date' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_execution_date_success' from the template is deprecated and will be removed in a future version. Please use 'prev_data_interval_start_success' instead.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'tomorrow_ds' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'tomorrow_ds_nodash' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'yesterday_ds' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'yesterday_ds_nodash' from the template is deprecated and will be removed in a future version.
  warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))

What you expected to happen

I think it is a bug.

How to reproduce

Run the code posted.

Operating System

Ubuntu 20.04.3 LTS

Versions of Apache Airflow Providers

apache-airflow-providers-celery==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-microsoft-mssql==2.0.1
apache-airflow-providers-microsoft-winrm==2.0.1
apache-airflow-providers-openfaas==2.0.0
apache-airflow-providers-oracle==2.0.1
apache-airflow-providers-samba==3.0.1
apache-airflow-providers-sftp==2.3.0
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.3.0

Deployment

Virtualenv installation

Deployment details

Airflow all in one

Anything else

nope

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

potiuk · 2021-12-31T08:34:58Z

I believe we know it already and we fixed it but I will let @uranusjr confirm when he is back.

mik-laj · 2021-12-31T12:05:48Z

Related: #20213

potiuk · 2021-12-31T13:00:40Z

Hmm #20213 should be part of 2.2.3. Are you sure you have 2.2.3 @gbonazzoli ?

kaxil · 2021-12-31T13:48:17Z

Yeah, this was the last item that was fixed for 2.2.3. Can you double-check the Airflow version please

potiuk · 2021-12-31T14:25:27Z

Yeah @gbonazzoli - maybe you have one of the RCs :)

gbonazzoli · 2021-12-31T16:26:04Z

@potiuk, @kaxil, unfortunately not.

root@new-airflow:~# airflow version
2.2.3

From the UI:

Version: v2.2.3
Git Version: .release:2.2.3+06c82e17e9d7ff1bf261357e84c6013ccdb3c241

potiuk · 2021-12-31T16:45:46Z

Confirmed. Just a presence of **kwargs in Python Callable triggers deprecation warnings for 2.2.3

kaxil · 2021-12-31T16:48:35Z

Thanks for confirming @gbonazzoli , we will fix it in 2.2.4

potiuk · 2021-12-31T16:58:28Z

BTW. It's fixed in main - most likely by #20322 - I marked it for 2.2.4

uranusjr · 2022-01-03T14:41:12Z

Specifically

airflow/airflow/utils/context.py

Lines 218 to 244 in 97261c6

    
           def lazy_mapping_from_context(source: Context) -> Mapping[str, Any]: 
        
               """Create a mapping that wraps deprecated entries in a lazy object proxy. 
        
               This further delays deprecation warning to until when the entry is actually 
        
               used, instead of when it's accessed in the context. The result is useful for 
        
               passing into a callable with ``**kwargs``, which would unpack the mapping 
        
               too eagerly otherwise. 
        
               This is implemented as a free function because the ``Context`` type is 
        
               "faked" as a ``TypedDict`` in ``context.pyi``, which cannot have custom 
        
               functions. 
        
               :meta private: 
        
               """ 
        
               def _deprecated_proxy_factory(k: str, v: Any) -> Any: 
        
                   replacements = source._deprecation_replacements[k] 
        
                   warnings.warn(_create_deprecation_warning(k, replacements)) 
        
                   return v 
        
               def _create_value(k: str, v: Any) -> Any: 
        
                   if k not in source._deprecation_replacements: 
        
                       return v 
        
                   factory = functools.partial(_deprecated_proxy_factory, k, v) 
        
                   return lazy_object_proxy.Proxy(factory) 
        
               return {k: _create_value(k, v) for k, v in source._context.items()}

Initially (including when 2.2.3 was implemented) the intention was to make this the expected behaviour and discourage **kwargs, but ultimately we have been recommending the **kwargs pattern too much, and using lazy-object-proxy (which is not a perfect solution and brings other problems) to cover this specific case is considered worthy.

jedcunningham · 2022-02-15T18:17:43Z

Closing this as the fix is already cherry-pick for 2.2.4 👍

BernhardNiksch · 2022-06-27T07:54:01Z

Still experiencing this in 2.2.5

potiuk · 2022-07-04T21:37:03Z

@BernhardNiksch - please open a new issue and provide evidences - logs, details circumstances, etc. Adding a comment "I have the same issue" on a closed issue brings exactly 0 value in any attempt to diagnose and help the issue. Your problem might look similar but be a different issue. The only way to know it, is to look at the evidences. By not providing it, your not giving anyone a chance to try to diagnose your problem, because .... you have not provided any evidences.

If you REALLY want to solve hte problem and, please help us to help you (even if you have the software for free and we are helping here in our free time). Being mindful of that and spending time on reporting your problem and providing evidences is the best way you can pay back to the community for the free software you get.

gbonazzoli added area:core kind:bug This is a clearly a bug labels Dec 31, 2021

potiuk added this to the Airflow 2.2.4 milestone Dec 31, 2021

kaxil added the pending-response label Dec 31, 2021

kaxil removed the pending-response label Dec 31, 2021

jedcunningham closed this as completed Feb 15, 2022

michaelritsema mentioned this issue Jun 6, 2024

Airflow 2.9.1 : AirflowContextDeprecationWarning: Accessing 'yesterday_ds_nodash' from the template is deprecated and will be removed in a future version. #40100

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context #20603

Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context #20603

gbonazzoli commented Dec 31, 2021

potiuk commented Dec 31, 2021

mik-laj commented Dec 31, 2021

potiuk commented Dec 31, 2021

kaxil commented Dec 31, 2021

potiuk commented Dec 31, 2021

gbonazzoli commented Dec 31, 2021

potiuk commented Dec 31, 2021

kaxil commented Dec 31, 2021

potiuk commented Dec 31, 2021

uranusjr commented Jan 3, 2022

jedcunningham commented Feb 15, 2022

BernhardNiksch commented Jun 27, 2022

potiuk commented Jul 4, 2022

Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context #20603

Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context #20603

Comments

gbonazzoli commented Dec 31, 2021

Apache Airflow version

What happened

What you expected to happen

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else

Are you willing to submit PR?

Code of Conduct

potiuk commented Dec 31, 2021

mik-laj commented Dec 31, 2021

potiuk commented Dec 31, 2021

kaxil commented Dec 31, 2021

potiuk commented Dec 31, 2021

gbonazzoli commented Dec 31, 2021

potiuk commented Dec 31, 2021

kaxil commented Dec 31, 2021

potiuk commented Dec 31, 2021

uranusjr commented Jan 3, 2022

jedcunningham commented Feb 15, 2022

BernhardNiksch commented Jun 27, 2022

potiuk commented Jul 4, 2022