Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Several improvements to the Params doc #29062

Merged
merged 3 commits into from Jan 21, 2023
Merged

Several improvements to the Params doc #29062

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bcdb88a
Several improvements to the Params doc
BasPH Jan 20, 2023
66ab4ba
Rewrite first paragraph for clarity
BasPH Jan 20, 2023
08a22b1
Update docs/apache-airflow/core-concepts/params.rst
BasPH Jan 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
64 changes: 33 additions & 31 deletions docs/apache-airflow/core-concepts/params.rst
View file
Open in desktop
Expand Up @@ -20,18 +20,18 @@
Params
======

Params are how Airflow provides runtime configuration to tasks.
When you trigger a DAG manually, you can modify its Params before the DAG run starts.
If the user-supplied values don't pass validation, Airflow shows a warning instead of creating the DAG run.
(For scheduled runs, the default values are used.)
Params enable you to provide runtime configuration to tasks. You can configure default Params in your DAG
code and supply additional Params, or overwrite Param values, at runtime when you trigger a DAG. Param values
are validated with JSON Schema. For scheduled DAG runs, default Param values are used.

Adding Params to a DAG
----------------------
DAG-level Params
----------------

To add Params to a :class:`~airflow.models.dag.DAG`, initialize it with the ``params`` kwarg.
Use a dictionary that maps Param names to a either a :class:`~airflow.models.param.Param` or an object indicating the parameter's default value.
Use a dictionary that maps Param names to either a :class:`~airflow.models.param.Param` or an object indicating the parameter's default value.

.. code-block::
:emphasize-lines: 6-9

from airflow import DAG
from airflow.models.param import Param
Expand All @@ -42,15 +42,31 @@ Use a dictionary that maps Param names to a either a :class:`~airflow.models.par
"x": Param(5, type="integer", minimum=3),
"y": 6
},
) as the_dag:
):

Task-level Params
-----------------

You can also add Params to individual tasks.

.. code-block::

PythonOperator(
task_id="print_x",
params={"x": 10},
python_callable=print_it,
)

Task-level params take precedence over DAG-level params, and user-supplied params (when triggering the DAG)
take precedence over task-level params.

Referencing Params in a Task
----------------------------

Params are stored as ``params`` in the :ref:`template context <templates-ref>`.
So you can reference them in a template.
Params can be referenced in :ref:`templated strings <templates-ref>`. For example:
BasPH marked this conversation as resolved.
Show resolved Hide resolved

.. code-block::
:emphasize-lines: 4

PythonOperator(
task_id="from_template",
Expand All @@ -66,15 +82,16 @@ Even though Params can use a variety of types, the default behavior of templates
You can change this by setting ``render_template_as_native_obj=True`` while initializing the :class:`~airflow.models.dag.DAG`.

.. code-block::
:emphasize-lines: 4

with DAG(
"the_dag",
params={"x": Param(5, type="integer", minimum=3)},
render_template_as_native_obj=True
) as the_dag:
):


This way, the Param's type is respected when its provided to your task.
This way, the Param's type is respected when it's provided to your task:

.. code-block::

Expand All @@ -93,6 +110,7 @@ This way, the Param's type is respected when its provided to your task.
Another way to access your param is via a task's ``context`` kwarg.

.. code-block::
:emphasize-lines: 1,2

def print_x(**context):
print(context["params"]["x"])
Expand All @@ -102,33 +120,17 @@ Another way to access your param is via a task's ``context`` kwarg.
python_callable=print_x,
)

Task-level Params
-----------------

You can also add Params to individual tasks.

.. code-block::

PythonOperator(
task_id="print_x",
params={"x": 10},
python_callable=print_it,
)

If there's already a DAG param with that name, the task-level default will take precedence over the DAG-level default.
If a user supplies their own value when the DAG was triggered, Airflow ignores all defaults and uses the user's value.

JSON Schema Validation
----------------------

:class:`~airflow.modules.param.Param` makes use of ``json-schema <https://json-schema.org/>``, so you can use the full json-schema specifications mentioned at https://json-schema.org/draft/2020-12/json-schema-validation.html to define ``Param`` objects.
:class:`~airflow.modules.param.Param` makes use of `JSON Schema <https://json-schema.org/>`_, so you can use the full JSON Schema specifications mentioned at https://json-schema.org/draft/2020-12/json-schema-validation.html to define ``Param`` objects.

.. code-block::

with DAG(
"my_dag",
params={
# a int with a default value
# an int with a default value
"int_param": Param(10, type="integer", minimum=0, maximum=20),

# a required param which can be of multiple types
Expand All @@ -147,7 +149,7 @@ JSON Schema Validation
maxLength=255,
),
},
) as my_dag:
):

.. note::
As of now, for security reasons, one can not use Param objects derived out of custom classes. We are
Expand Down