Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add concrete examples for accessing context variables from TaskFlow tasks #33296

Merged
merged 4 commits into from Aug 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 9 additions & 2 deletions docs/apache-airflow/core-concepts/taskflow.rst
Expand Up @@ -66,9 +66,16 @@ If you want to learn more about using TaskFlow, you should consult :doc:`the Tas
Context
-------

When running your callable, Airflow will pass a set of keyword arguments that can be used in your function. This set of kwargs correspond exactly to the :ref:`context variables<templates:variables>` you can use in your Jinja templates.
You can access Airflow :ref:`context variables <templates:variables>` by adding them as keyword arguments as shown in the following example:

.. include:: ../../shared/template-examples/taskflow.rst

Alternatively, you may add ``**kwargs`` to the signature of your task and all Airflow context variables will be accessible in the ``kwargs`` dict:

.. include:: ../../shared/template-examples/taskflow-kwargs.rst

For a full list of context variables, see :ref:`context variables <templates:variables>`.

For this to work, you need to define ``**kwargs`` in your function header, or you can add directly the keyword arguments you would like to get such as ``ti=None`` to have the task instance passed.

Logging
-------
Expand Down
12 changes: 12 additions & 0 deletions docs/apache-airflow/templates-ref.rst
Expand Up @@ -81,6 +81,18 @@ Variable Type Description
The DAG run's logical date, and values derived from it, such as ``ds`` and
``ts``, **should not** be considered unique in a DAG. Use ``run_id`` instead.

Accessing Airflow context variables from TaskFlow tasks
-------------------------------------------------------

While ``@task`` decorated tasks don't support rendering jinja templates passed as arguments,
all of the variables listed above can be accessed directly from tasks. The following code block
is an example of accessing a ``task_instance`` object from its task:

.. include:: ../shared/template-examples/taskflow.rst

Deprecated variables
-------------------------------------------------------

The following variables are deprecated. They are kept for backward compatibility, but you should convert
existing code to use other variables instead.

Expand Down
27 changes: 27 additions & 0 deletions docs/shared/template-examples/taskflow-kwargs.rst
@@ -0,0 +1,27 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

.. http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

.. code-block:: python

@task
def print_ti_info(**kwargs):
ti = kwargs["task_instance"]
print(f"Run ID: {ti.run_id}") # Run ID: scheduled__2023-08-09T00:00:00+00:00
print(f"Duration: {ti.duration}") # Duration: 0.972019

dr = kwargs["dag_run"]
print(f"DAG Run queued at: {dr.queued_at}") # 2023-08-10 00:00:01+02:20
24 changes: 24 additions & 0 deletions docs/shared/template-examples/taskflow.rst
@@ -0,0 +1,24 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

.. http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.

.. code-block:: python

@task
def print_ti_info(task_instance=None, dag_run=None):
print(f"Run ID: {task_instance.run_id}") # Run ID: scheduled__2023-08-09T00:00:00+00:00
print(f"Duration: {task_instance.duration}") # Duration: 0.972019
print(f"DAG Run queued at: {dag_run.queued_at}") # 2023-08-10 00:00:01+02:20