Skip to content

Commit

Permalink
Several improvements to the Params doc (#29062)
Browse files Browse the repository at this point in the history
* Several improvements to the Params doc

* Rewrite first paragraph for clarity

* Update docs/apache-airflow/core-concepts/params.rst

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>

Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
(cherry picked from commit af9143e)
  • Loading branch information
BasPH authored and pierrejeambrun committed Mar 7, 2023
1 parent 746d1ee commit fb9d06f
Showing 1 changed file with 33 additions and 31 deletions.
64 changes: 33 additions & 31 deletions docs/apache-airflow/core-concepts/params.rst
Expand Up @@ -20,18 +20,18 @@
Params
======

Params are how Airflow provides runtime configuration to tasks.
When you trigger a DAG manually, you can modify its Params before the dagrun starts.
If the user-supplied values don't pass validation, Airflow shows a warning instead of creating the dagrun.
(For scheduled runs, the default values are used.)
Params enable you to provide runtime configuration to tasks. You can configure default Params in your DAG
code and supply additional Params, or overwrite Param values, at runtime when you trigger a DAG. Param values
are validated with JSON Schema. For scheduled DAG runs, default Param values are used.

Adding Params to a DAG
----------------------
DAG-level Params
----------------

To add Params to a :class:`~airflow.models.dag.DAG`, initialize it with the ``params`` kwarg.
Use a dictionary that maps Param names to a either a :class:`~airflow.models.param.Param` or an object indicating the parameter's default value.
Use a dictionary that maps Param names to either a :class:`~airflow.models.param.Param` or an object indicating the parameter's default value.

.. code-block::
:emphasize-lines: 6-9
from airflow import DAG
from airflow.models.param import Param
Expand All @@ -42,15 +42,31 @@ Use a dictionary that maps Param names to a either a :class:`~airflow.models.par
"x": Param(5, type="integer", minimum=3),
"y": 6
},
) as the_dag:
):
Task-level Params
-----------------

You can also add Params to individual tasks.

.. code-block::
PythonOperator(
task_id="print_x",
params={"x": 10},
python_callable=print_it,
)
Task-level params take precedence over DAG-level params, and user-supplied params (when triggering the DAG)
take precedence over task-level params.

Referencing Params in a Task
----------------------------

Params are stored as ``params`` in the :ref:`template context <templates-ref>`.
So you can reference them in a template.
Params can be referenced in :ref:`templated strings <templates-ref>` under ``params``. For example:

.. code-block::
:emphasize-lines: 4
PythonOperator(
task_id="from_template",
Expand All @@ -66,15 +82,16 @@ Even though Params can use a variety of types, the default behavior of templates
You can change this by setting ``render_template_as_native_obj=True`` while initializing the :class:`~airflow.models.dag.DAG`.

.. code-block::
:emphasize-lines: 4
with DAG(
"the_dag",
params={"x": Param(5, type="integer", minimum=3)},
render_template_as_native_obj=True
) as the_dag:
):
This way, the Param's type is respected when its provided to your task.
This way, the Param's type is respected when it's provided to your task:

.. code-block::
Expand All @@ -93,6 +110,7 @@ This way, the Param's type is respected when its provided to your task.
Another way to access your param is via a task's ``context`` kwarg.

.. code-block::
:emphasize-lines: 1,2
def print_x(**context):
print(context["params"]["x"])
Expand All @@ -102,33 +120,17 @@ Another way to access your param is via a task's ``context`` kwarg.
python_callable=print_x,
)
Task-level Params
-----------------

You can also add Params to individual tasks.

.. code-block::
PythonOperator(
task_id="print_x",
params={"x": 10},
python_callable=print_it,
)
If there's already a dag param with that name, the task-level default will take precedence over the dag-level default.
If a user supplies their own value when the DAG was triggered, Airflow ignores all defaults and uses the user's value.

JSON Schema Validation
----------------------

:class:`~airflow.modules.param.Param` makes use of ``json-schema <https://json-schema.org/>``, so you can use the full json-schema specifications mentioned at https://json-schema.org/draft/2020-12/json-schema-validation.html to define ``Param`` objects.
:class:`~airflow.modules.param.Param` makes use of `JSON Schema <https://json-schema.org/>`_, so you can use the full JSON Schema specifications mentioned at https://json-schema.org/draft/2020-12/json-schema-validation.html to define ``Param`` objects.

.. code-block::
with DAG(
"my_dag",
params={
# a int with a default value
# an int with a default value
"int_param": Param(10, type="integer", minimum=0, maximum=20),
# a required param which can be of multiple types
Expand All @@ -147,7 +149,7 @@ JSON Schema Validation
maxLength=255,
),
},
) as my_dag:
):
.. note::
As of now, for security reasons, one can not use Param objects derived out of custom classes. We are
Expand Down

0 comments on commit fb9d06f

Please sign in to comment.